74% of New Web Pages Now Contain AI-Generated Content
74.2% of newly created web pages now contain AI-generated content. That's not a guess — it's what Ahrefs found after running an in-house detector over 900,000 English-language pages its crawler discovered fresh. If you're building anything that learns from the open web, the pool you're drawing from is now mostly synthetic.
The short version
- Ahrefs' bot_or_not detector scanned 900,000 new English-language web pages and found 74.2% contained AI-generated content — only 25.8% were pure human writing. (Ahrefs)
- Just 2.5% of pages were "pure AI" with no human editing; the largest group, 71.7%, was a human-AI mix, split across light, moderate, and heavy AI involvement.
- 87% of content marketers surveyed in the same study said they use AI to create or help create content — the supply side explains the saturation.
- Separately, Cloudflare Radar data shows AI companies aren't reading this content carefully either: Anthropic's crawlers fetched 1,917 pages for every 1 referral sent back in July 2026, down from 3,386:1 in June but still the worst ratio among major labs. (Cloudflare Radar via TechnologyChecker)
- OpenAI's ratio improved to 251:1 and Google's sits at 4.7:1 — a nearly 500x gap between the most and least extractive crawler in the same dataset.
Why Is So Much New Web Content AI-Written?
Because the marginal cost of producing a page collapsed. Ahrefs' survey found 87% of content marketers already use AI tools somewhere in their writing process, and publishing a page now takes minutes, not days. Volume went up long before quality controls caught up.
What Did Ahrefs Actually Measure?
Ahrefs built a proprietary classifier, bot_or_not, and ran it against 900,000 pages its own crawler discovered as newly published in April 2025. Each page got scored on a spectrum rather than a binary label, because most AI-assisted writing isn't 100% machine or 100% human.
The breakdown is where it gets specific. 2.5% of pages were pure AI — no meaningful human editing. 25.8% were pure human. The remaining 71.7% were mixed, and within that group, 25.86% showed moderate AI use (11–40% of the text), 20.50% showed substantial AI use (41–70%), and 15.51% were AI-dominant (71–99%). Put plainly: three out of four new pages have a machine's fingerprints on them somewhere.
Are AI Companies Even Reading This Content Carefully?
Not particularly, and the crawl data backs that up. Cloudflare Radar's full-month July 2026 figures show Anthropic's crawlers hit 1,917 pages for every referral they sent back — an improvement on June's 3,386:1, but still an order of magnitude worse than OpenAI's 251:1 or Perplexity's 289:1. Google, still running search-first infrastructure, sits at 4.7:1.
| Crawler | Crawl-to-refer ratio (July 2026) | Change vs. June |
|---|---|---|
| Anthropic | 1,917:1 | Improved from 3,386:1 |
| Perplexity | 289:1 | — |
| OpenAI | 251:1 | Improved from 647:1 |
| 4.7:1 | Baseline |
A ratio that lopsided means most of what these crawlers touch never resurfaces anywhere a human can trace it back to the source. Combine that with three-quarters of new pages carrying AI-generated text, and you get a compounding problem: high-volume, low-precision crawling pointed at an increasingly synthetic web.
What Does This Mean for Anyone Building on Web Data?
Two trends are converging on the same conclusion. The pages available to scrape are more likely than not to contain AI-generated text, and the dominant crawlers pulling that data aren't filtering hard for originality or relevance — they're optimizing for coverage. For a RAG pipeline or a training run, that's the exact recipe for quietly compounding noise: synthetic content gets ingested, resurfaces in model output, gets republished, and gets scraped again next cycle.
This is the problem we built ScrapeOps around — treating "more pages crawled" as the wrong success metric, and deduplicating and filtering sources down to what's actually comprehension-ready before it ever reaches a pipeline. A single query should return hundreds of clean, distinct sources, not a haystack a human still has to sort by hand.
None of this means the open web stopped being useful. It means the sorting problem got harder, and the teams that win on RAG accuracy in 2026 will be the ones treating source curation as seriously as model selection. Read more analysis like this on Dekrypt Labs Dispatches and dig into the underlying data in our research section.
Frequently Asked Questions
What percentage of web pages contain AI-generated content in 2026? Ahrefs found 74.2% of 900,000 newly published English-language web pages contained some AI-generated content, based on an April 2025 crawl. Only 25.8% were entirely human-written; 2.5% were entirely AI-generated with no human editing.
What is a crawl-to-refer ratio? It measures how many pages a crawler fetches for every one referral (click-through) it sends back to the site it crawled. Cloudflare Radar tracks this per AI company; Anthropic's July 2026 ratio was 1,917:1, versus OpenAI's 251:1 and Google's 4.7:1.
Does AI-generated content hurt RAG pipeline accuracy? It can, if pipelines ingest it unfiltered. Synthetic text that gets scraped, cited by a model, republished, and scraped again compounds noise over time. Deduplication and source-quality filtering before ingestion reduce the risk more reliably than filtering after the fact.
Why do some AI crawlers have worse crawl-to-refer ratios than others? Search-purpose crawlers like Google's send users back to source pages and get credited with a referral. Training and answer-generation crawlers from labs like Anthropic and OpenAI mostly extract content into model responses, so far fewer crawls translate into a visit back to the original page.
Abhishek Gupta is Co-Founder at Dekrypt Labs, building ScrapeOps — the data acquisition engine that turns any question into clean, deduplicated, comprehension-ready sources. dekryptlabs.com