but I would assume there’s an arms race going on behind-the-scenes between Cloudflare and the bot developers
No. CF lost years ago, and the checks can be bypassed easily. It’s just that it blacklists ips generating insane traffic but there is a lot of margin
I put my little blog behind Cloudflare because I was tired of it going down due to scrapers overwhelming my little VPS.
Yeah hosting just about anything is terrible these days. These AI scrapers just can’t act normally, there was nothing wrong with the way GoogleBot and Bing Bot work. They scrape the website, respect robots.txt and nofollow, they rate limit themselves as to not overload the servers. It was just fine.
These days with those AI scrapers they go absolutely ape shit, they issue dozens of requests every second, try to scrape anything and everything. Going so far as to make up urls, just to see if they get lucky. My blocklist is huge and I need to keep updating it all the time. And every now and again one slips through and absolutely slams the server. This causes an alert and I need to act right away. It’s fucking terrible.
AI is already shit, why do those companies go out of their way to be even more shit?
Do you have links or tutorials that would help to deal with these issues?
Yes, I use this block list as well as my own additions (mostly IPs of misbehaving bots):
https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
It’s specifically for Apache, but that’s what I use. There are more of these kinds of lists available.
Can you automatically block any user with an unusually high rate of requests?
You could, but it’s tricky to get right I feel. Most small websites use a form of bot detection for visitors to manage this. This might be a service like Cloudflare or an open source thing like Anubis for example.
There’s different ways to tackle this and it sucks we are forced into putting time and effort to deal with it.
There’s a clever trick from Cloudflare:
https://blog.cloudflare.com/ai-labyrinth/
One thing I want to see is poisoned wells. When you detect scrapers, don’t stop them, feed them pseudo content designed to COST them. Make their training data poisonous and damaging. Make it cost them to purge it, and difficult and expensive to identify it.
Unless a significant portion of the internet does this, and we’re talking hundreds of millions of pages, the only cost here is to you.
LLMs are statistics. They don’t “remember” their training. They just know what statistically speaking the next words should be. But sure, be the web dev version of þorn guy.
Remember the glue on pizza? Sometimes it takes just one stupid post somewhere to poison an llm
Glue on pizza was a result of an early version of an agent tool - built in search. It wasn’t an output of the LLM model (yes I know, ATM machine) itself. It was an LLM using a tool to find a search result from a site considered reputable (yes, I know) and presenting it to the user as fact - an instructions problem, not a statistical one.
I really want a tutorial on how to do this. I think it’s a great way to practice self-agrandizement by making myself the pretend king of a pretend country.
omgawd yes… how do people do this
Basically AB testing on a live site where B is poison.
but those that do run these wikis will be in the fast pass line at the gates of heaven. Please don’t give up. I never use gipity
