Cloudflare's New AI Crawler Rules Start Sept 15
Cloudflare sits in front of more than one in five domains on the web. Starting September 15, 2026, its new AI crawler rules will block AI training and AI agent bots by default — but only on pages that carry ads. That one carve-out is the whole story: Cloudflare isn't trying to stop AI from reading the web, it's trying to make sure someone pays when there's money already sitting on the page.
The short version
- Cloudflare's new AI crawler rules take effect September 15, 2026: Training and Agent bots get blocked by default on any new domain's ad-monetized pages, while Search bots stay allowed. (Cloudflare)
- The old single "Block AI bots" toggle is replaced by three separate classifications — Search, Agent, Training — tracked inside a new bot directory called BotBase, which covers 11 behavior types in total.
- Multi-purpose crawlers get judged on their worst behavior: Googlebot, Applebot, and Bingbot combine Search with Training, so a site that blocks Training now blocks all three outright, even on pages where Search was welcome.
- A new content-use signal in robots.txt —
immediate,reference, orfull— lets a site owner say not just whether a bot can crawl, but what it's allowed to keep and reshare afterward. - This is Cloudflare's second "Content Independence Day." The first, in July 2025, shipped a one-click "Block AI bots" toggle and a Pay-Per-Crawl marketplace — a blunter, single-purpose version of the same idea.
What Actually Changes on September 15, 2026?
For every new domain onboarding to Cloudflare, Training and Agent crawlers get blocked by default on pages that display ads. Search crawlers stay allowed by default. Existing customers keep their current settings unless they opt in, and can opt out of the new defaults any time before the date.
Cloudflare's logic: an ad is a signal that a site owner wanted a human to land there and see it, so Training and Agent bots — which don't drive that attention back — get treated as the exception rather than the default on those pages. Search, which still tends to send referral traffic, gets the benefit of the doubt.
Why Multi-Purpose Crawlers Take the Biggest Hit
Most crawlers on the web already fall cleanly into one bucket. Googlebot, Applebot, and Bingbot don't — they combine Search indexing with model training inside a single bot identity, which is exactly the arrangement Cloudflare is now designing against.
Starting September 15, a multi-purpose crawler is allowed or blocked according to all of its behaviors, not the friendliest one. A site owner who blocks Training through Cloudflare's new controls, or through the legacy "Block AI bots" setting, blocks Googlebot's Search crawling too — because Cloudflare enforces the most restrictive applicable rule. Cloudflare's stated reason is transparency: if a bot operator runs automation that serves three different purposes, Cloudflare wants that operator to run three separate, separately-classified bots instead of one bot wearing three hats.
The New Bot Taxonomy, In Plain Terms
Cloudflare's directory, BotBase, tracks eleven behavior types — Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, and Monitoring & Operations. Only three are newly configurable by every customer, including free-tier sites:
| Classification | What it does | Blocked by default on ad pages (new domains) |
|---|---|---|
| Search | Indexes a site to answer questions about it later | No |
| Agent | Acts on a person's behalf right now (ChatGPT-User, browser-use agents) | Yes |
| Training | Crawls content to train or fine-tune a model | Yes |
Layered on top of that is the content-use signal, which governs what a bot may do with what it took: immediate means interact but store nothing, reference (the new default) means index, excerpt, and link back, and full means summarize and reproduce. A bot caught reproducing content in full loses Cloudflare's "Verified" status — which, after September 15, no longer means automatically allowed. Verified now just means honestly labeled; the category still decides access.
What This Means If You Build on Web Data
None of this closes the web. It reclassifies it. A crawler that used to get in because it looked like a search bot now has to declare itself as Search, Agent, or Training — and live with whichever one gets it blocked. We covered the revenue side of this shift when Cloudflare launched Pay-Per-Use pricing for AI crawls; September 15 is the access-control half of the same policy, and the two are clearly built to work together.
For anyone acquiring data for an AI product, the practical read is simple: the rules for reaching a page are becoming more granular, not more permissive, and an acquisition layer that can adapt to per-site, per-purpose rules matters more than one that just fetches URLs. That's the exact problem ScrapeOps is built to sit in front of — turning "can I even reach this page, under what terms" into a solved step instead of a recurring outage. For the broader dataset behind that argument, see our research page.
Nothing here stops a determined crawler. What it does is make the default answer to "can I train on this page" a no, on the pages that pay a site's bills, unless someone actively says otherwise. That's a meaningfully different internet for anyone whose data pipeline assumed the old default of "ask forgiveness, not permission."
The bigger shift is procedural, not technical: Cloudflare now sits between every AI company and more than a fifth of the web's domains, deciding by default what each bot is allowed to say it's doing. Whoever controls that default — not the model, not the scraper — controls how much of the open web AI companies actually get to see.
Frequently Asked Questions
What is Cloudflare's Content Independence Day? Content Independence Day is Cloudflare's annual policy update on how AI bots interact with the sites it protects. The second edition, launched July 1, 2026, replaces a single "block AI bots" toggle with three separate categories — Search, Agent, and Training — that site owners can allow or block independently.
When do Cloudflare's new AI crawler rules take effect? The new defaults apply September 15, 2026, to every new domain onboarding to Cloudflare. On any page that shows ads, Training and Agent bots get blocked automatically while Search crawlers stay allowed. Existing customers can opt out of the change any time before that date.
Why would Google's crawler get blocked if it's also used for search? Cloudflare blocks multi-purpose crawlers based on all of their behaviors, not just one. Because Googlebot, Applebot, and Bingbot combine Search with Training, any site owner who blocks Training bots also blocks these crawlers entirely — even on pages where Search access was welcome.
What is the new content-use signal in robots.txt?
It's a fourth field Cloudflare added to the Content Signals standard, letting site owners specify how a bot may use what it takes: immediate (no storage), reference (index and link back, the new default), or full (summarize and reproduce). Bots caught reproducing content in full lose Verified status.
Abhishek Gupta is Co-Founder at Dekrypt Labs, building ScrapeOps — the data acquisition engine that turns any question into clean, deduplicated, comprehension-ready sources. dekryptlabs.com