Since July 1, I’ve been closely following Cloudflare’s battle against AI bots. Our CEO, Matthew Prince, recently shared that we have successfully blocked 416 billion AI bot requests for our customers during this time.
This insight sheds light on Google’s significant advantage in AI. They’re currently capable of viewing 3.2 times more web pages than OpenAI, underlining the challenge smaller AI companies face.
Why this matters. The flood of AI systems consuming vast amounts of web content is concerning, especially without a mechanism for publishers to counteract this. Our statistics at Cloudflare show just how aggressively these AI bots are searching for data.
The current scenario. Ever since we launched our pay-per-crawl initiative on July 1, our clients have been automatically blocking AI crawlers. At the recent WIRED Big Interview event, Prince highlighted that so far, 416 billion AI bot requests have been turned away.
Analyzing Cloudflare’s data reveals that Google sees:
- 3.2 times more webpages than OpenAI.
- 4.6 times more than Microsoft.
- 4.8 times more than Anthropic or Meta.
As Prince mentioned, Google enjoys “this incredibly privileged access.”
The bigger picture. As it stands, Google offers publishers a difficult choice: either block AI training and risk disappearing from Google Search or allow it and accept AI scraping.
- Prince said, “You can’t opt out of one without opting out of both, which is crazy. You shouldn’t get to use your monopoly of yesterday to secure a monopoly of tomorrow.”
The AI landscape. Prince believes:
- AI signifies a major shift in platforms that could reshape the web’s business model.
- Cloudflare aims to prevent market consolidation, ensuring the web remains open while assisting creators and businesses in adapting to this shift.
- Encouragingly, publishers that already block AI crawlers report positive results, Prince noted.
Looking ahead. As AI models pursue superior training data, the worth of “creative, original human thought” will climb, potentially leading to opportunities in paid licensing, Prince forecasted. Meanwhile, Cloudflare is advocating for AI giants, particularly Google, to distinguish between search and AI crawling.
- Prince asserted, “Google is the problem here. It is the company that is keeping us from going forward on the internet, and until we force them – or hopefully convince them – that they should play by the same rules as everyone else and split their crawlers up between search and AI, I think we’re going to have a hard time completely locking all the content down.”
The story. To explore further, check out WIRED’s article on Cloudflare Has Blocked 416 Billion AI Bot Requests Since July 1 (subscription required).
Inspired by this post on Search Engine Land.
Leave a Reply