Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
It’s hard because the requests all come from different IPs, at least on my site. 185k “unique visitors” hit my site just yesterday, half from outside of North America, which is odd because my site is pretty local.