Cloudflare's AI Agent Crawler Blocking Starts Sept 15

Cloudflare already serves more than 1 billion HTTP 402 Payment Required responses to AI crawlers every single day. On September 15, 2026, it adds a second lever: any bot it classifies as an "Agent" gets blocked by default on every page that runs ads — no configuration required, no opt-in needed.

If your product sends a browser out to fetch pages on someone's behalf, that AI agent crawler blocking policy is not background noise. It is the line between "reaches the page" and "gets a 402."

The short version

• From September 15, 2026, new Cloudflare customers, new sites on existing accounts, and all free-tier sites default to blocking "Agent" and "Training" crawlers on any page carrying ads — "Search" crawlers stay allowed.
• Cloudflare's AI Crawl Control already returns over 1 billion HTTP 402 Payment Required responses to AI crawlers per day, separate from the September default-block change.
• Cloudflare now sorts bots into three buckets: Search (indexes for later), Agent (acts in real time on a person's behalf, like ChatGPT-User), and Training (absorbs content into a model).
• ChatGPT-User-style agent requests grew 2,825% year-over-year, and PerplexityBot's crawl-to-refer ratio hit roughly 700:1 — the traffic pattern this policy is built to throttle.
• A crawler keeps "Verified Bot" status only if it honors the site's declared preference; Cloudflare's BotBase now reports on compliance, not just identity.

What Actually Changes on September 15, 2026?

Cloudflare's current default lets any crawler through unless the site owner blocks it. The new default flips that for two of three categories: Agent and Training crawlers are blocked on any page that shows ads, while Search crawlers remain allowed. It applies to new Cloudflare customers, new sites on existing accounts, and every site still on the free plan — not, yet, to existing paid sites that haven't changed anything (The Register).

Why Split Bots Into Three Buckets Instead of One?

Cloudflare's old "block all AI bots" toggle didn't distinguish a search index crawl from an agent completing a task from a training run hoovering up text for a model. The new taxonomy does: Search bots index and send referral traffic back, Agent bots act live on a person's behalf, Training bots permanently absorb content (Playwire).

The Agent bucket is the one that should worry anyone building automated browsing products. It doesn't distinguish "malicious scraper" from "legitimate assistant fetching a page a user asked for" — it just checks the declared bot category against the site's preference.

The Number Cloudflare Isn't Leading With

Buried under the policy headline is the volume it's already managing: Cloudflare's AI Crawl Control, generally available since August 2025 and consolidated across every plan tier during Agents Week in April 2026, now serves over 1 billion HTTP 402 Payment Required responses to AI crawlers a day across its network (leadgen-economy.com). That's the scale a September default-block sits on top of.

Cloudflare CEO Matthew Prince framed the shift plainly: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster" (The Register). The crawl-to-refer math backs him up — PerplexityBot's ratio topped 700:1 and OpenAI's hit as much as 3,700:1 at their March 2025 peak, meaning thousands of pages crawled for every visitor sent back.

What This Means If You Run a Browser Agent

None of this is a wall — it's a classification problem. A crawler that identifies honestly as "Agent" walks into the September 15 default block on any ad-supported page unless the site owner opts back in. A browser that behaves like the human it's acting for doesn't trip the same category logic in the first place. That's the gap we built DLBrowser to close: reaching pages reliably without depending on a site owner's bot-category settings going your way.

Cloudflare isn't wrong to build this — non-human traffic really has overtaken human traffic on parts of the web, and publishers were getting crawled thousands of times for every referral they got back. But the practical effect for anyone shipping an agent that browses is immediate: test your traffic against the Agent classification before September 15, not after. For a deeper look at how detection actually works underneath policy layers like this, see our research.

The interesting fight from here isn't "block or don't block." It's who gets to decide what an "Agent" looks like, and whether that definition can tell the difference between a scraper and a person's assistant fetching one page they asked for.

Frequently Asked Questions

What is Cloudflare's AI agent crawler blocking policy? Starting September 15, 2026, Cloudflare defaults to blocking crawlers it classifies as "Agent" or "Training" on any page that displays ads, while "Search" crawlers remain allowed by default, for new customers, new sites, and free-tier accounts.

How does Cloudflare tell an Agent bot from a Search bot? Cloudflare sorts declared crawler identity into three categories: Search bots index content for later answers, Agent bots act in real time on a person's behalf (like ChatGPT-User), and Training bots absorb content permanently into a model.

How many requests does Cloudflare's AI Crawl Control already block or charge for? Cloudflare's AI Crawl Control serves more than 1 billion HTTP 402 Payment Required responses to AI crawlers every day across its network, a separate mechanism from the September 15 default-block change.

Does this affect existing Cloudflare sites that haven't changed settings? Not immediately. The new defaults apply to new Cloudflare customers, new sites created by existing customers, and all free-tier sites — existing paid sites keep their current settings unless the owner changes them.

Abhishek Gupta is Co-Founder at Dekrypt Labs, building DLBrowser — a stealth browser runtime for real-world data collection. dekryptlabs.com