I’m getting really tired of “verify you’re not a bot” checks everywhere.
I understand these checks help prevent bots, but they seem to appear on so many websites now. Does anyone else find them frustrating?
The anti-bot checks are awful for usability, and the bots that keep spamming everything are even worse
It just means you’re difficult to track and fingerprint. So good job 👍
In the last 24 hours my small personal website got 8 legitimate visitors and 74,000 requests from bots, most of which were either malicious or aggressive ai crawlers
Wanna make that 9?
The internet is for bots
These days? Fuck no. I think it’s like 57% of all web traffic these days is just bots. This is absolutely needed.
Cloudflare is getting rich from all of these bots. We had turn to Cloudflare at work because bots were scraping our website and essentially DDoSing. Cloudflare worked perfectly and rate limiting bot scraping.
I wouldn’t be surprised if they make some of the bots they stop. It’d be like the people who bred snakes to get money for killing them in India.
You can add “Strengthened Cloudflare as the Internet’s single point of failure” to the long, growing list of awful things which have happened due to this AI boom.
I run a reticulum NomadNet site. I setup logging, since there are no ‘sessions’ and I don’t want to force everyone to identify, I just calcuate bot traffic based on time, order and frequency of loads, bots’s scraping are likely 80% of my traffic. And this is for a service that’s designed to support 200bps radio links.
Cloudflare can fuck right off!
Use anubis
I dunno, Anubis never gave me too much trouble, meanwhile that fucking “Gandalf” thing they have on some Nitter/Redlib instances keeps getting stuck on my device, and if you reload the page you get insta blocked 🫠
It’s proportional to the amount of rude AI crawling, from what I’ve heard.
The industry has put every reasonably accessible piece of human writing in their datasets, by now, and are scrounging for the remaining little bits. Old, obscure, obsolete paper books nobody ever reads are apparently a hot item. So, yeah, your latest blog post is probably worth some fraction of a penny, and that’s all it takes.
Im ok with them as long as theyre not demanding input by clicking everything that is a fire post so they can train their ai for free
They’ve made a large chunk of the internet inaccesible to me.
Are you a bot ?
No but definitely ain’t human !
Beep Bop Boop.
I was running a mediawiki instance for a while, and despite the very low human traffic and arguably useless nature of the site to anyone but me and a few online friends I was getting hammered by Amazon. I don’t think it got to the point of a denial of service but the traffic was excessive.
This was a problem before AI but has only gotten worse now that AI companies are scraping sites for training data.
CAPTCHAs and the like are often inaccessible to screen reader users, so the proliferation of CAPTCHAs has affected me greatly.
It is what it is I suppose.
Using a VPN? They trigger like whoA
Just a few moments ago I passed an anti-bot check with my VPN on.
Yeah you can pass the check but it’s everywhere. It sucks but I guess that’s the price to pay.
No, but only because I’ve helped run a website recently and you underestimate just how much wasted traffic is from scrapper bots. On our limited bandwidth, it was practically a DDOS attack all the time.
My solution reduced 95ish percent of traffic. Bots were hitting my tiny site like mad.
I went from thousands of requests per minute to 10, tops. Bot prevention is a must have.
If bot checks were removed then everyone would be complaining that it’s unusably slow.
It’s easy to block most bots. It’s impossible to block bots that target you specifically. I’m not blocking any bots on my websites, I like it better to optimize performance so that I can serve them all and have a lot of headroom
No, it’s not. Had request from a bot network, always changing ip’s, user agent and request url. There is no pattern there I can block.
I created a bot black hole. Specific recursive link that if the click it three times it blocks the IP. And the link is “click if ai” URL at the bottom. Works like a charm.
No; the traffic i get seems like it had stored my sitemap from ~10 years ago because the requested urls there are long gone, and now tries to guess other url’s with a similar pattern. It does not seem to follow links from the frontpage.
That’s fascinating. It almost sounds like they’re training a scraper. Which, really, is far and away the worst possible bot to have.
I have fail2ban take care if those. 404s and all that.
There surely is ja4 but I recognize it requires some setup
I wish Traefik exposed the ja4 fingerprint. There’s an open issue: https://github.com/traefik/traefik/issues/12421
What’s your solution?
Python script with fail2ban. Registers invalid requests then blocks. Works well.
https://radar.cloudflare.com/bots
Slowly but surely we’re approaching the point where bots make up the most internet traffic, and additionally because of LLMs more and more of the web’s content is slop, which is why the frontier ai companies are no longer just blanket training on all the internet.
Is there a way to trick the bots into thinking your site was made by an LLM so they won’t want to train on it?
The easiest way would likely be to add an ai written html comment attributing a real ai model and version to the content creation, with a comment that all of the info on the page was ai generated, anecdotal and hadn’t been researched. An ingestion model is then likely to ignore it or rate it low. It should be noted too that all the big ai companies now honor robots.txt and LLMs.txt, so that would more likely be defending against “neolabs” and international labs who ignore them.
I’ve just considered it a casualty of the privacy wars.
I don’t know if they’re deployed as a “weapon of annoyance” or anything, but I do notice that when I’m using a) a degoogled browser, b) an ad-blocking dns, c) mask my location, and/or d) block my IP. Suddenly (almost) every website makes me wait by throwing that at me.
I think it’s lowkey trying to annoy me to the point where I give up on having those privacy measures, but they underestimate my hatred of corporate fuckery.
I’d guess your client just looks more like a crawler that way and gets a JS challenge thrown its way
That depends. How well do they work?
It is that or shutting down your site.
Those fucking bots just keep coming, causing traffic no small or medium site can cope with. So yeah, deal with it.
