Cloudflare’s new AI bot controls only fire on pages that show ads, only bind the crawlers that announce themselves, and leave Googlebot drinking from the pipe. Here is what the September 15 deadline actually changes.

On July 1, 2026, Cloudflare shipped a set of controls that let site owners sort AI crawlers into three buckets and decide what each one gets. On September 15, those controls acquire a default, and a large slice of the web starts blocking AI training and agent traffic without anyone touching a dashboard.

The new default only fires on pages Cloudflare’s classifier decides are showing ads. Not on your content or documentation, but on your ad inventory.

Almost everything else about the announcement follows from that.

What shipped on July 1st, 2026

Cloudflare split AI bots into three categories by what they do rather than who runs them.

• Search covers crawlers that collect or index your content to answer questions about it later.
• Agent covers real-time automated activity on a person’s behalf — chat fetch bots, browser-use agents, the thing that loads your pricing page thirty seconds after someone asks an assistant about you.
• Training covers crawlers taking content to train or fine-tune a model, and it explicitly sweeps in mixed-purpose crawlers.

Each category gets three settings: block on all pages, block on pages with ads, or allow. Nine states, available on the free plan and up, under Security Settings → Configure AI bot policies. The old binary “Block AI bots” toggle survives as a legacy option, which tells you what it always was — a switch that could not express the thing most site owners actually wanted, which was never “all of them” or “none of them.”

Then the deadline. On September 15, 2026, Training and Agent are blocked by default on ad-serving pages. Search stays allowed.

Who inherits that default matters more than the default itself. It applies to new Cloudflare customers, to new sites added by existing customers, and — the line that will surprise people — to all existing free customers. If you run a configured paid zone, nothing happens to you on the fifteenth. If you are on the free plan and have not thought about this, something does.

The carve-out doing all the work

Read the policy again with the qualifier in front of it: on pages with ads.

If you run a SaaS product site, an agency site, a B2B blog, a documentation portal, a portfolio, a local service business (anything without display advertising), the new default is a no-op. Nothing changes. The announcement that has been covered as a shift in how the web handles AI crawlers does not reach most commercial sites on the internet, because most commercial sites do not sell impressions.

What the default protects is one specific revenue model: pages that earn when a human loads them and an ad slot fills. That is a legitimate thing to protect. It is also not the same thing as protecting content, or authorship, or intellectual property, and the coverage has quietly let those collapse into each other.

There is a second-order problem. Cloudflare decides which of your pages qualify, using automated ad detection. So the scope of your bot policy is set by a classifier you do not control, running against markup you may not have written, on a template that changes when your CMS updates. A policy whose scope is inferred is a policy you cannot audit.

If you want a rule you can reason about, the block-on-all-pages setting is right there. Nobody is defaulted into it.

Googlebot drinks free

Mixed-purpose crawlers get the most restrictive rule that applies to them. Cloudflare is direct about this: if a crawler does Search and Training, and you block Training, the crawler is blocked. Search permission does not rescue it.

Applied to Googlebot, that sounds severe. Googlebot indexes for search and feeds Google’s generative surfaces through the same fetch. Under the strictest reading, blocking Training blocks Google entirely.

Except the default does not block Training for everyone everywhere. It blocks Training on ad pages, for new and free zones. And the site owners most likely to override that default within the hour are the ones who cannot afford to lose Google, which is to say all of them.

So the practical outcome inverts. Google, whose crawler is the least separable, faces the least pressure, because no publisher will take the risk. The pressure lands on OpenAI, Anthropic, Perplexity, Mistral, and the agent startups — the companies that did the responsible thing and shipped distinct, declared, individually addressable crawlers.

To explain it plainly, separating your crawlers makes you legible. Legible makes you blockable. The operators that ignored the request to identify themselves are, mechanically, the ones this policy touches least.

Google has pushed back on framings like this before, pointing at Google-Extended as a training opt-out that costs you nothing in Search. It is a fair point and it does not resolve the asymmetry. Google-Extended governs downstream use, not the fetch, and it exists because Google is large enough to publish its own vocabulary and have the industry adopt it. Nobody else’s opt-out gets honored by default the way Google’s crawler gets admitted by default.

A block that only stops the polite

Cloudflare sits in front of more than 20% of web domains. That is an enormous surface, and it is not the web.

This enforcement works on bots that announce themselves. The classification system, the BotBase directory of known operators, and the per-bot detection IDs — the whole apparatus is built on identity. A crawler sending a declared user-agent from a declared IP range can be sorted into Search, Agent, or Training and handled accordingly. A crawler rotating through residential proxies behind a Chrome user-agent and a headless browser cannot, not reliably, not at scale, and not without false positives that cost you real customers.

The population this reaches is, broadly, the population that already honored robots.txt.

Forwarded

Which makes the genuinely new piece of engineering the one that got the least coverage. Cloudflare added a transitive trust model that reads the HTTP Forwarded header to identify the original requester behind an intermediary. Agent traffic routed through a browser-automation service or a proxy layer can be attributed to the operator that actually initiated it, and policy applied to that operator rather than to the middleman carrying the request.

That is the part to watch. It is an attempt to make attribution survive a hop of indirection, and if it holds, it matters more than any default setting — because the entire agent economy is built on indirection.

The number everyone quotes keeps moving

The argument underneath all of this is the crawl-to-refer ratio: pages an AI crawler fetches from your site, divided by the human visits its platform sends back.

Cloudflare Radar’s figures for June 2026 put Anthropic near 4,580:1, OpenAI at 848:1, Perplexity at 186:1, and Google at 5:1. Those numbers were quoted everywhere, and on their face they are damning.

By July, Anthropic was at 2,237:1 and OpenAI at 217:1, with Mistral taking the top spot at 3,389:1. On the trailing 28 days to August 1: Anthropic 1,782, Perplexity 303, OpenAI 233, Microsoft 37, Google 4.79.

Anthropic’s ratio fell by roughly 60% in two months, not because the crawling slowed but because the platforms started sending traffic back.

A grievance metric that halves quarterly is not a stable foundation for a permanent default. It is a snapshot of an immature referral economy, cited as though it described a fixed property of AI companies.

There is a second problem with it, and it is the one that matters for most people reading this. The ratio is a fraction. If the denominator was already near zero — if your site never pulled meaningful traffic from search to begin with — then any crawl at all produces a catastrophic-looking number that tells you nothing about what you lost. The ratio measures the traffic you used to get, not the traffic you are owed.

The bandwidth argument is the good one

Underneath the compensation debate sits the claim I find hardest to argue with: more than half of AI crawl traffic re-fetches pages that have not changed.

That is not a licensing dispute. That is a bug. Conditional requests, If-Modified-Since, ETag, 304 Not Modified — HTTP has had the machinery to avoid this since 1997, and a crawler that re-downloads an unchanged page is burning someone else’s bandwidth because it could not be bothered to send one header.

Matthew Prince framed the urgency as: “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.” The waste is real, the volume is real, and a company moving that much traffic has standing to complain about it.

But the fix for wasteful re-crawling is caching discipline, and it does not require a payments marketplace. Bundling the two means the most verifiable complaint in the announcement ends up doing rhetorical work for the least verifiable one.

Concede the bandwidth. Keep arguing about the rest.

Pay Per Crawl was wrong, and Cloudflare just said so

Pay Per Crawl launched on a simple mechanism: charge AI companies for the fetch. The obvious criticism arrived almost immediately — a fetch is not value. A crawler that downloads your page, decides it is irrelevant, and never surfaces it in an answer has cost you almost nothing and paid you the same as one that quoted you in a million responses.

Pay Per Use is the correction. Publishers get compensated when the content does something: appears in an AI-generated result, or supplies premium material an agent needed. It launches with Ceramic.ai and You.com — Ceramic paying when publisher content shows up in its AI search results, You.com paying for access to individual pieces of premium content its agents request.

The redesign is better. It is also an admission, inside eighteen months, that the original charged for the wrong event.

And it clarifies the position Cloudflare now holds. The company writing the default that turns AI access into a negotiation is the company operating the marketplace where the negotiation gets priced. Cloudflare’s Jin-He Lee and Bryan Becker put the goal as content owners wanting “to protect their content” and deserving compensation “for the original content.” I do not doubt it. I would still note that an infrastructure provider fronting a fifth of the web’s domains is now setting content policy for all of them, and nobody voted.

Two launch partners most publishers have never invoiced is not a market yet. It might become one.

Four files to say one thing

If you want to state your position on AI access unambiguously, here is what that currently takes.

• robots.txt, for the crawlers that read it.
• Cloudflare’s Content Signals, now extended with a use parameter carrying three values — immediate (interact, do not store), reference (index, excerpt, link back, and the default), and full (summarize and reproduce).
• IETF AIPREF, where draft-ietf-aipref-vocab is a working-group document on the Proposed Standard track defining train-ai and search categories with y/n values, and draft-ietf-aipref-attach defines a Content-Usage header plus a matching robots.txt rule, carrying an August 2026 milestone to send a standards-track specification to the IESG.
• RSL, Really Simple Licensing, which handles the permission and compensation layer AIPREF deliberately leaves out, and which folds Cloudflare’s Content Signals vocabulary into its own.
• Then ai.txt and TDMRep, both live, both proposing their own file.

Six overlapping vocabularies for one sentence: do not train on this.

Cloudflare’s use parameter is a reasonable design. It is also a seventh dialect bolted onto a file format specified in 1994 that has never had a formal grammar and has always been advisory. Every additional channel for expressing a preference is one more thing a crawler can claim it did not parse.

The standards work will converge. AIPREF exists precisely to make that happen, and RSL is built to sit on top of it rather than compete with it. Until it does, “I clearly signaled my preferences” is a sentence with six possible meanings and no enforcement behind any of them.

What to Decide Before September 15

There are three things to think about.

• Training. Blocking it is defensible and cheap. There is no measurable traffic consequence, the models that already trained on your archive have already trained on it, and if you intend to be paid for future training this is the position that creates the leverage. Block it if you want to, but do not expect it to change anything visible in analytics.
• Agent. Agent traffic is a fetch triggered by a person, right now, who asked an assistant a question and is about to be shown an answer that either includes you or does not. It is not a model harvesting your archive at 3am. For a publisher selling impressions, that fetch is a lost pageview. For an agency, a SaaS company, a consultant, a local service business (anyone whose site exists to generate inquiries rather than ad revenue) that fetch is a prospect standing at the door, and blocking it is paying to be invisible. The default blocks Agent, and B2B sites usually should not.
• Search. Almost nobody should block this, and the most-restrictive-rule logic makes it worse than it looks. Blocking Search reaches Googlebot’s indexing half through the same door.

Then the administrative point. If you are on a free plan, or you spin up a new zone in September, the defaults arrive whether or not you read the announcement. Open Security Settings before the fifteenth. It takes a minute, and every setting is reversible.

The Bottom Line

The toggle is new. The leverage is not. Cloudflare has given site owners a more precise way to express a preference that still depends on the other party choosing to honor it, defaulted it in a way that protects ad inventory specifically, and built the marketplace where the resulting negotiation gets priced.

Block AI bots if the trade makes sense for what you sell. Just be clear that it is a setting and not a strategy, and that for most sites, the expensive mistake is not letting the crawlers in. It is blocking the one that showed up because somebody asked about you.