Cloudflare positions itself as the payment intermediary for AI-powered web

AI is fundamentally reshaping how people discover content online. Services like Google's AI Overviews now summarize information directly on the search results page, eliminating the need for readers to visit publisher sites. This shift has siphoned away the traffic that once fueled web publishing economics.

Cloudflare unveiled a suite of new features this week designed to address this disruption, positioning itself as the financial intermediary between AI companies and content creators. The company is essentially betting that if AI agents become the dominant way people access information, it can capture the economic layer that manages content access, attribution, and compensation.

From blocking to negotiating#

The company's strategy has evolved considerably in just a year. Previously, Cloudflare's messaging focused on giving site owners the power to block AI crawlers entirely. But that defensive posture has given way to something more ambitious: building what the company calls "rails" for an "agentic economy."

The shift reflects a pragmatic acceptance that AI crawlers are here to stay. Rather than fighting that reality, Cloudflare wants to be the entity that handles the business arrangements between AI developers and publishers.

Transitioning from access fees to usage fees#

Under Cloudflare's earlier Pay Per Crawl model, publishers could charge AI companies simply for the act of fetching their pages—essentially a toll booth approach. The company is now advancing toward Pay Per Use, where publishers only receive payment when their content actually appears in an AI-generated answer.

This represents a more sophisticated economic model. Rather than paying for raw access, AI companies compensate based on value delivered. Cloudflare is already piloting this approach with services like Ceramic.ai and You.com. The challenge: building the attribution infrastructure needed to track which content contributed to which AI-generated response—a capability that doesn't yet exist at scale.

Forcing crawler transparency#

Cloudflare is also pushing back against a murky aspect of AI data collection: companies bundling multiple crawler types under a single bot. A single Google crawler might simultaneously fetch pages for search indexing, model training, and agentic tasks, making it impossible for publishers to consent to some uses while blocking others.