In 2025, Googlebot once again led the charge in generating more web traffic than any other crawler, as revealed in a new report by Cloudflare. Google continued its tireless web crawling for both search indexing and AI training, proving its dominance over other search and AI bots.
According to the numbers from Cloudflare, Googlebot was responsible for more than 25% of all Verified Bot traffic. In fact, Googlebot alone accounted for 4.5% of all HTML request traffic, which is more than all other AI bots combined at 4.2%.
The surge in AI “user action” crawling, which is when bots simulate human behavior, saw an impressive 15x increase year over year. Despite the rise in AI crawlers, Googlebot still had a crawl volume that eclipsed every other AI bot, including those from OpenAI, Anthropic, and Meta.
In the world of AI crawlers, they were the most frequently disallowed in robots.txt files. Moreover, Anthropic notably had the highest crawl-to-refer ratio among major AI and search platforms, crawling much more content than it returned as traffic. Early in the year, this ratio hit ~500,000:1, before settling between ~25,000:1 and ~100,000:1 after May, as compared to OpenAI’s peak at ~3,700:1 in March and Perplexity’s lowest among major platforms.
Diverse search platforms exhibited different behaviors. Microsoft’s ratio oscillated between ~50:1 and ~70:1, with a notable weekly cycle. Google’s ratio climbed from just over ~3:1 to ~30:1 by April, dropped to ~3:1 by mid-July, then gradually increased again. Meanwhile, DuckDuckGo stayed below 1:1 until jumping to ~1.5:1 in mid-October.
Despite these changes, Google maintained its monopoly in search, delivering almost 90% of search engine referral traffic. Bing, Yandex, Baidu, and DuckDuckGo completed the top five, but their shares were significantly smaller.
Throughout the year, very little shift occurred; Google remained dominant as Yandex’s share dipped from 2.5% to 1.5%, and Baidu experienced a modest rise from 0.9% to 1.6%.
I found the full report quite insightful. If you’re interested in exploring it yourself, you can check out The 2025 Cloudflare Radar Year in Review for comprehensive insights on AI, post-quantum advancements, and notable DDoS attack trends.
Inspired by this post on Search Engine Land.
Leave a Reply