Why Cloudflare’s AI block could safeguard the open internet

Cloudflare’s decision to help publishers clamp down on generative AI chatbots crawling their sites could reshape how AI models train — and help publishers protect their traffic and ad revenue.

The internet infrastructure company first introduced a one-click option to block AI crawlers last September, which more than a million customers activated. Now it’s taking it a step further by blocking AI crawlers by default — with the goal of giving publishers more leverage.

Why now? Because the data is stark.

Cloudflare, which manages roughly 20% of the internet’s traffic, saw that as AI bots scrape publisher content at scale, they send almost nothing back — ultimately undermining the referral dynamics that help keep publishers funded.

AI training crawlers operate similarly to search crawlers, but they exhibit vastly different crawl-to-referral ratios. In June 2025, according to Cloudflare data, OpenAI’s crawlers had a crawl-to-referral ratio of 1,700-to-1. Anthropic’s was a jaw-dropping 73,000-to-1. Google Search’s, by comparison, was 14-to-1.

“We began hearing from publishers that their traffic was decreasing. We initially thought it could be hackers, and as we began to look at the data, we realized this was a new challenge and the conditions were changing rapidly,” Will Allen, VP of product at Cloudflare, tells The Current.

“Part of our role in this is to help make the conditions for a new business model. It’s to help bring together the two ecosystems so we can have a more sustainable ecosystem for both AI companies and content creators, for the future of the internet as a whole.”

Marketers feel the shift

Cloudflare’s move comes as the open internet faces its biggest stress test since the rise of social media. For media buyers, it could be a seismic shift.

“GenAI is shifting discovery from traditional search and publisher sites to AI-generated summaries and recommendations,” says John Cucka, head of analytics at Kantar in Australia. “This could fragment consumer journeys and reduce direct engagement with original content. Brands need to think about how they show up in those environments — because the old rules of SEO and display won’t cut it.”

This fragmentation could also shrink the pool of brand-safe, quality environments marketers rely on for effective reach.

“Marketers could find themselves with fewer premium environments to advertise in. That’s a problem, not just for brand clarity and consistency, but for marketing effectiveness,” Cucka says. “It also reinforces the need for diversified media plans and smarter measurement.”

And if generative AI reduces direct traffic to publisher sites further, marketers’ favorite KPIs could lose relevance.

“We’re already seeing this play out. Less direct traffic means traditional metrics like CTRs [click-through rates] and site visits lose value,” Cucka says.

“Marketers need to lean into models that capture broader impact — like MMM, creative diagnostics and cross-channel attribution. It’s about measuring influence in fragmented environments, ensuring campaigns remain accountable and optimized.”

None of this is theoretical. A recent Reuters Institute report found that AI chatbots are increasingly being used as a source of news, especially among younger audiences — causing concern among publishers about further declines in website traffic.

The future of the open internet

For Allen, the stakes are clear: If the open internet is to survive, publishers need to control how AI models use their content.

“I think that to help prevent the internet from becoming a walled garden, it’s critical for a crawler to identify themselves and their intent, and for publishers to be armed with the power to decide how those crawlers can access their content,” he says.

“The current landscape seems to force content creators into a binary choice: either grant AI unimpeded access to all their creations or attempt to block access, usually done with things like paywalls. Such a scenario would impede the free flow of information and centralize content, undermining the very essence of an open internet.”

For now, Cloudflare’s mission has the support of audience measurement experts like Kantar.

“We support moves that protect publisher content and promote ethical data use,” Cucka says. “More than that, it’s a smart call — publishers deserve control over how their content is used, especially with AI scraping at scale.”