Cloudflare enforces payment rules for AI crawlers accessing publisher content

Cloudflare is taking steps to reshape how AI developers gain access to online content. Starting September 15, 2026, the platform will automatically block "mixed-use" crawlers—bots that combine search indexing with AI training and agent operations—from pages displaying advertisements unless site owners explicitly allow them.

This default shift will apply to new Cloudflare customers, newly created sites, and existing free-tier users, creating friction for AI companies that currently blend multiple crawling purposes into a single bot.

The publisher protection angle#

Website operators typically want their content visible to search engines and available through AI services, yet they're increasingly uncomfortable with seeing that intellectual property repurposed without compensation. Cloudflare's move acknowledges this tension by giving publishers stronger default protections.

The company took a pointed shot at Google, noting that the search giant enjoys roughly double the data access of other AI companies because its dominance makes it difficult for site owners to remain discoverable while opting out of AI training use. Google maintains that it offers Google Extended, a separate bot that sites can restrict while remaining fully visible in Search results—though Googlebot itself still powers AI features like AI Overviews.

Marketplace and pay-per-use models#

Cloudflare has been expanding its toolkit for content monetization. The company's Pay Per Crawl marketplace already lets publishers charge for scraping access, and it's now broadening into Pay Per Use arrangements. Under this model, publishers earn when their content generates actual value—such as appearing in AI search results or powering premium features—rather than simply being fetched once.

Initial partners include Ceramic.ai and You.com. Publishers participating in these programs receive payments when content appears in search results or is accessed as premium material.

Why this matters now#

Bot traffic has crossed a major threshold: non-human visitors now exceed human traffic on the internet—a milestone that arrived faster than expected. This shift has made Cloudflare and others more urgent about establishing sustainable rules. As CEO Matthew Prince put it, "now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."