It’s not that complicated - block all bot traffic by default, whitelist the bots you can verify are only indexing for search (which is actually beneficial traffic).
The thing that Google’s doing to piss off websites is that they use the same bots for indexing and AI trawling, so if you want to show up on their indexes you have to let them slam your website/server to train their AI.
Now websites are telling Google to fuck off because the cost of AI trawling isn’t worth being on their search index
Let’s see them try to opt out of AI training.
It’s not that complicated - block all bot traffic by default, whitelist the bots you can verify are only indexing for search (which is actually beneficial traffic).
The thing that Google’s doing to piss off websites is that they use the same bots for indexing and AI trawling, so if you want to show up on their indexes you have to let them slam your website/server to train their AI.
Now websites are telling Google to fuck off because the cost of AI trawling isn’t worth being on their search index
It’s not that easy to identify bot traffic. AI companies have been incredibly persistent and aggressive at pretending to be humans to scrape sites.
it’s not worth the cost because once google has your information, users won’t need to visit your website.
the question of "who is going to be creating all this high-quality data not only for no money but also for no recognition?‘’ goes unasked