The Surge of AI Crawlers: Publishers Unite Against Bots as Platforms Fail to Deliver Traffic!

Googlebot once again generated more traffic than any other crawler in 2025, according to a new Cloudflare report. It outpaced every search and AI bot as Google continued crawling the web for search indexing and AI training. This impressive performance highlights the increasing reliance on Google’s algorithms by businesses. Content creators understand the importance of optimizing their online presence. Their goal is to attract organic search traffic. As the digital landscape evolves, Google continuously refines its crawling capabilities. It adapts to emerging technologies. These efforts ensure that it remains at the forefront of search engine innovation. It provides users with relevant and timely information. This helps in supporting the growth of the entire internet ecosystem.

By the numbers. Googlebot accounted for more than 25% of all Verified Bot traffic observed by Cloudflare.

• Googlebot alone generated 4.5% of all HTML request traffic – more than all other AI bots combined (4.2%).
• AI “user action” crawling surged more than 15x year over year. This shows a sharp rise in bots that simulate human behavior.
• Googlebot’s crawl volume dwarfed every other AI crawler, including OpenAI, Anthropic, and Meta.

AI crawling surges. AI crawlers were the most frequently fully disallowed user agents in robots.txt files.

• Anthropic had the highest crawl-to-refer ratio among major AI and search platforms. It crawled far more content than it sent back as traffic. The ratio peaked near ~500,000:1 early in the year, then settled between ~25,000:1 and ~100,000:1 after May. For comparison:
  • OpenAI spiked to ~3,700:1 in March.
  • Perplexity was the lowest among major AI platforms. It started below 100:1. It briefly jumped above 700:1 in late March during a PerplexityBot crawl spike. Then it stayed mostly below 400:1. From September onward, it remained under 200:1.

Search platforms looked very different:

• Microsoft hovered between ~50:1 and ~70:1 with a weekly cycle.
• Google rose from just over ~3:1 to ~30:1 by April, fell back to ~3:1 by mid-July, then gradually increased again.
• DuckDuckGo stayed below 1:1 for the first three quarters, then jumped to ~1.5:1 in mid-October and remained elevated.

Google still monopolizes search. Traditional search dominance barely changed.

• Google remained the top search engine by a wide margin, delivering nearly 90% of search engine referral traffic.
• Bing (3.1%), Yandex (2.0%), Baidu (1.4%), and DuckDuckGo (1.2%) rounded out the top five.
• Cloudflare saw minimal movement during the year.
  • Google stayed dominant throughout.
  • Yandex slipped from 2.5% in May to 1.5% in July.
  • Baidu rose from 0.9% in April to 1.6% in June.

The report. The 2025 Cloudflare Radar Year in Review: The rise of AI, post-quantum, and record-breaking DDoS attacks