Cloudflare Pay Per Use: Why Half of AI Crawls Waste Data
Cloudflare just admitted its own AI crawlers were the problem. On July 1, 2026, CEO Matthew Prince announced that more than half of all AI crawler traffic hitting Cloudflare-protected sites is spent re-fetching pages that haven't changed since the last visit — and rebuilt the company's year-old payment model around that number.
The short version
- Cloudflare's Pay Per Use, announced July 1, 2026, replaces the July 2025 Pay Per Crawl model — publishers now get paid when their content shows up inside an AI answer, not just when a bot fetches the page.
- Cloudflare's own data shows over 50% of AI crawler traffic re-fetches pages with no new content since the prior crawl.
- Launch partners Ceramic.ai and You.com pay publishers when their content appears in AI search results, not per crawl event.
- The change lands six weeks before Cloudflare's separate September 15, 2026 default block on "Agent" and "Training" crawlers hitting ad-supported pages.
- Prince tied the move to bots recently overtaking humans as the majority of internet traffic — a shift he said wasn't expected until 2027.
What is Cloudflare's Pay Per Use model?
Pay Per Use ties publisher payment to citation, not access. Under the old Pay Per Crawl model, launched in July 2025, an AI company paid a fee every time its bot fetched a page — whether that fetch produced anything a model or agent actually used, or just added another line to a server log.
Under Pay Per Use, publishers are paid when their content actually shows up inside an AI-generated answer. Prince framed the shift as fixing a mismatch: a crawl is an attempt to get information, but a citation is proof the information had value. Cloudflare's bet is that pricing the second event instead of the first gives both sides a cleaner signal.
Why does half of AI crawl traffic count as waste?
Because most of it hits pages the crawler already has. Cloudflare's own measurement found that over 50% of AI crawler requests re-fetch content unchanged since the previous crawl — bandwidth and compute spent confirming nothing new happened, on both sides of the request.
That number cuts two ways. Publishers pay for the server capacity to serve a repeat request that generates no new value. AI companies pay for the compute to fetch, parse, and (often) re-embed a document that hasn't moved a byte since last time. Neither side gets anything out of the transaction except confirmation that the page is still there.
How does Pay Per Use actually work for publishers?
Cloudflare's first two partners show two flavors of the model. When a publisher opts in, they get paid when Ceramic.ai's AI search surfaces their content in a result, or when You.com's agents pull from a piece of premium content the publisher has gated behind Cloudflare's tooling. Cloudflare says other AI companies can build custom commercial terms on the same rails — the mechanism is pay-on-value, not pay-on-fetch, and the specifics of what counts as "value" are negotiated per partner rather than fixed by Cloudflare.
| Model | Trigger for payment | Launched |
|---|---|---|
| Pay Per Crawl | Bot fetches the page | July 2025 |
| Pay Per Use | Content appears in an AI answer or citation | July 2026 |
Where this sits next to Cloudflare's September crawler block
Pay Per Use is a separate lever from the default-blocking policy Cloudflare rolls out on September 15, 2026, which we covered in our earlier dispatch on Cloudflare's Agent and Training crawler defaults. That policy decides who gets through the door at all — Agent and Training crawlers get blocked by default on ad-supported pages, Search crawlers stay allowed. Pay Per Use decides what happens to the ones who do get through: whether their traffic produces anything a publisher gets paid for, or just adds to the re-fetch pile Cloudflare just quantified.
Cloudflare also used the announcement to needle Google directly, claiming the "world's largest search engine" has access to roughly 2x more of the open web than other AI companies, because Google Extended lets sites opt out of AI training without losing Search visibility — an asymmetry no other crawler enjoys (TechCrunch).
What this means for teams building AI data pipelines
For anyone running a crawl-and-embed pipeline into a RAG system or model, Cloudflare's number is an external audit of a problem most teams already suspect but rarely measure: how much of a recurring crawl is spent re-processing pages that didn't change. Every unchanged page that gets re-fetched, re-parsed, and re-embedded is compute spent maintaining a vector index at the same freshness it already had — not improving it.
The fix isn't a smarter model. It's better change detection before the fetch happens at all — hashing, conditional requests, and knowing what you already hold before you ask for it again. That's the exact failure mode a data acquisition layer has to solve: not re-fetching, re-parsing, or re-deduplicating a page that returned identical bytes yesterday. ScrapeOps exists to sit on the other side of that inefficiency — turning a question into sources that are already deduplicated against what you've seen before, not fetched blind and sorted out later.
For more on how web scraping and AI crawling are changing in 2026, see our dispatches archive and the underlying research behind our data-acquisition approach.
The 50%-plus re-fetch number is the part worth sitting with longer than the payment mechanics. It means half the infrastructure AI companies run to keep models and agents current produces zero new information, and that inefficiency stayed invisible until the company sitting in front of a huge share of the web's traffic decided to publish it (PPC Land). Whatever payment model wins out between publishers and AI companies, the waste in the pipe is a data-engineering problem first and a commercial one second.
Frequently Asked Questions
What is Cloudflare's Pay Per Use model? Pay Per Use, announced July 1, 2026, pays publishers when their content appears inside an AI-generated answer or citation, replacing the per-fetch payment structure of the 2025 Pay Per Crawl model.
How is Pay Per Use different from Pay Per Crawl? Pay Per Crawl charged AI companies every time a bot fetched a page, regardless of outcome. Pay Per Use only triggers payment when that content demonstrably creates value — showing up in an AI search result or answer.
Why did Cloudflare say half of AI crawls are wasted? Cloudflare's own traffic data showed over 50% of AI crawler requests re-fetch pages that haven't changed since the previous crawl, meaning the bandwidth and compute spent produced no new information for either side.
Does Pay Per Use replace Cloudflare's September 15 crawler blocking policy? No. They're separate policies: the September 15, 2026 default blocks "Agent" and "Training" crawlers from ad-supported pages entirely, while Pay Per Use governs payment for the crawlers still allowed through.
Abhishek Gupta is Co-Founder at Dekrypt Labs, building ScrapeOps — the data acquisition engine that turns any question into clean, deduplicated, comprehension-ready sources. dekryptlabs.com