Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
Remember the glue on pizza? Sometimes it takes just one stupid post somewhere to poison an llm
Glue on pizza was a result of an early version of an agent tool - built in search. It wasn’t an output of the LLM model (yes I know, ATM machine) itself. It was an LLM using a tool to find a search result from a site considered reputable (yes, I know) and presenting it to the user as fact - an instructions problem, not a statistical one.