The battle between news publishers and AI bots is heating up

AI web scraping is getting a little out of hand.

A new report by TollBit released on Wednesday showed that scraping activity increased by 20% in Q4 2025 compared to Q3. National news was one of the most-scraped categories in Q4, a 55% jump.

The findings underscore why the IAB is intensifying its push against AI bots.

The organization is calling for a federal bill to protect online publishers and original content from AI scraping. This week, it released draft legislation called the AI Accountability for Publishers Act.

“If we keep the status quo where AI bots can leverage publisher content at will, the internet will become a shadow of itself,” IAB CEO David Cohen said during his keynote at IAB’s Annual Leadership Meeting (ALM) on Monday. “We don’t want to live in a world where, due to AI bots scraping publisher content without paying a dime, the publisher market disappears.”

IAB Tech Lab began work last year on a technical framework to help publishers receive compensation for content scraped by AI. But the legislative proposal suggests that frameworks may not be enough to rein in AI companies and third-party scrapers.

In part, the draft proposes that the controller or deployer of bots that exploit covered content without consent could face civil action in a U.S. district court. They could also face treble damages if they fail to disclose the nature and purpose of bots to the digital publication.

“Unless we act to protect original human-created content, the internet risks devolving into an echo chamber of recycled, low-quality information,” Cohen warned.

Is legislation feasible?

Toshit Panigrahi, CEO and co-founder of TollBit, said that forcing bots to be identified as such, rather than posing as humans, would be the best “sniff test” for regulation.

“That would be the smallest piece of legislation that everyone can get behind and that would protect the entirety of the internet in any jurisdiction,” he told The Current.

The TollBit report elaborates: “Regulators must ensure that bots are not allowed to mimic humans. Otherwise, website owners may be caught in an increasingly expensive and ineffective cat-and-mouse game of trying to detect web scrapers.”

That takes on even more urgency with the rise of agentic AI, where AI agents perform tasks on a person’s behalf.

“If we continue a trend of bots controlling the browser and looking human, advertisers will not trust impressions,” Panigrahi added.

Some were skeptical of IAB’s proposal, though. David Nyurenberg, senior vice president of digital at InterMedia Advertising, called the effort “materially late.”

“This mirrors the broader pattern we saw with Google’s monopoly accountability, where enforcement arrived only after irreversible structural damage had been done,” he said.

What publishers are already doing

To Nyurenberg’s point, damage from AI is well underway.

A recent report from the Reuters Institute analyzed traffic data from 2,756 news sites globally and found that Google search traffic fell 33% worldwide in 2025 and 38% in the U.S., largely due to Google’s AI Overviews that summarize answers and deprioritize links to news sources.

But some major news publishers have taken matters into their own hands.

People Inc. said at IAB ALM that it’s started blocking some bots on some of its sites by default, resulting in a 600% increase in the number of bots blocked. Recent research showed that 8 in 10 of the top news websites in the U.S. and U.K. are blocking at least one AI training crawler.

Some publishers like The Guardian and The New York Times have started to restrict the Internet Archive over AI concerns. The New York Times has also sued OpenAI and Perplexity over copyright violations.

“The archive serves an important societal role, but publishers are operating in an environment where their work is being extracted at scale with little consent or compensation,” Nyurenberg said. “Blocking bots, even broadly, is an understandable response to a system that has repeatedly failed to protect them.”

Other publishers, like Business Insider owner Axel Springer and The Wall Street Journal publisher News Corp, have struck content licensing deals with AI companies.

Still, TollBit’s research found that click-through rates for websites both with and without AI licensing deals dropped significantly in 2025.

So it’s damned if you do, damned if you don’t? Panigrahi noted that, even if a publisher is prioritized in search results, people will click fewer links the more comfortable they become with AI search. On the flip side, the more a publisher blocks bots, the more costly it becomes to secure content.

“The odds are stacked against content owners,” he said.

Still, Cohen believes that it’s not too late for legislation to at least hinder the threat posed by AI bots.

“The future isn’t written yet, and neither is the legislation that will govern it,” he said during his keynote.