RAG Data Pipeline: Why Deduplication Decides Quality

A single 61-word sentence appeared more than 60,000 times in C4 — the dataset that trained T5 and shaped half the language models that followed. Nobody put it there on purpose. It was boilerplate, scraped from thousands of near-identical pages, and it slipped past every filter.

That is the quiet failure mode of every RAG data pipeline: not missing data, but the same data over and over, wearing slightly different URLs.

The short version

• Google researchers (Lee et al., 2022) found one 61-word sequence repeated over 60,000 times in C4, and that deduplicated models emit memorized training text roughly 10× less often.
• The RefinedWeb pipeline behind the Falcon models discarded close to 90% of raw Common Crawl through filtering and strict deduplication — and trained a stronger model on what remained.
• Hugging Face's FineWeb team processed 96 Common Crawl snapshots into 15 trillion tokens and found that deduplicating within each snapshot beat deduplicating globally across all of them.
• In retrieval systems the cost is direct: duplicate chunks crowd the top-k results, so the model sees the same passage three times and the second-best source not at all.
• Deduplication belongs early in the pipeline — after fetching, before chunking and embedding — or every downstream stage pays for the copies.

The most duplicated sentence on the internet

C4 was built from a single April 2019 Common Crawl snapshot and cleaned down to roughly 750 GB of text. It was considered a careful dataset. Yet when Google researchers ran exact substring matching across it, they found that 61-word sequence — a piece of insurance boilerplate — more than 60,000 times.

The consequences were measurable, not cosmetic. Models trained on the raw corpus regurgitated memorized training sequences about 10× more often than models trained on the deduplicated version. Duplicates also wasted compute: the deduplicated model reached the same accuracy in fewer training steps, because it stopped re-learning the same paragraph thousands of times.

The web has only gotten more repetitive since 2019. Syndicated articles, scraped-and-republished blogs, boilerplate cookie banners, and AI-generated filler mean any crawl today contains a higher fraction of near-copies than the one C4 was built from.

What does deduplication do in a RAG pipeline?

Deduplication removes exact and near-identical documents or chunks before they reach your index. In a RAG data pipeline, it prevents duplicate passages from crowding the top-k retrieval slots, stops the same fact from being embedded hundreds of times, and cuts storage and embedding cost — often by a large fraction of the corpus.

The retrieval failure is worth spelling out. Suppose your corpus holds one press release republished on twelve news aggregators. A query on that topic returns the same passage in positions one through five. Your context window is now full of one source pretending to be five, and the genuinely different document that would have improved the answer sits at rank eleven — unseen.

The model then does what models do: it treats repetition as consensus. Twelve copies of one claim look like twelve independent confirmations. That is how a single wrong press release becomes a confidently wrong answer.

Exact matching is not enough

The naive approach — hash every document, drop identical hashes — catches almost nothing that matters. The twelve aggregator copies of that press release each carry a different header, a different footer, a different "related articles" block. Every hash is unique. Every document is the same.

Production pipelines use near-duplicate detection instead, and MinHash with locality-sensitive hashing is the workhorse. It reduces each document to a compact signature and flags pairs whose estimated overlap crosses a threshold, without comparing every document to every other one.

The scale of what this removes surprises people. The RefinedWeb team, building training data for the Falcon models, discarded nearly 90% of raw Common Crawl through their filtering and deduplication stages — and the models trained on the survivors outperformed models trained on curated corpora. The trash was not the minority of the crawl. It was most of it.

FineWeb added the counterintuitive finding: more deduplication is not always better. When the team deduplicated globally across all 96 snapshots, benchmark performance dropped compared to deduplicating within each snapshot. Aggressive global dedup preferentially deleted well-distributed, high-quality text while leaving each snapshot's unique junk untouched. Dedup is a dial, not a switch.

Where deduplication sits in the pipeline

Order matters more than tooling. The pipeline that works is: fetch → normalize (strip boilerplate, canonicalize) → deduplicate → chunk → embed → index. Teams that dedup after chunking work at the wrong granularity — chunk boundaries shift between copies, so near-identical documents produce chunks different enough to evade matching.

Deduplicating at ingestion also compounds savings. Every duplicate you drop before embedding is an embedding you never compute, a vector you never store, and a retrieval slot you never poison. On a corpus where 30–50% of fetched pages are near-copies — normal for news, e-commerce, and anything syndicated — that is the single largest cost lever in the pipeline.

This is the unglamorous half of the "where's the data" problem. Getting sources is step one; getting distinct sources is the step that decides whether retrieval works. It's the reason we built deduplication into ScrapeOps as a core stage rather than a post-processing script — one query should return hundreds of sources, not hundreds of copies of twelve sources.

The open web makes this harder every quarter. As we covered in the closing-web dispatch, more of the accessible web is syndicated, mirrored, or machine-generated — which means the duplicate fraction of any fresh crawl keeps rising.

The corpus you don't index

The teams with the best retrieval quality are not the ones with the biggest indexes. They are the ones who deleted the most before indexing. RefinedWeb threw away nine-tenths of its input and won. The discipline is in what you refuse to store.

Audit your own corpus this week: MinHash it, cluster the near-duplicates, and look at the size of the largest cluster. If the answer embarrasses you, your retrieval quality was already paying for it.

More engineering notes on data pipelines live in our research and the dispatches archive.

Frequently Asked Questions

Why is deduplication important in a RAG data pipeline? Duplicate passages crowd the top-k retrieval slots, so the model sees one source repeated instead of several distinct ones. Deduplication frees those slots, cuts embedding and storage costs, and prevents repetition from being mistaken for consensus. Google research also showed deduplicated training data reduces memorized output roughly 10×.

What is the best method for near-duplicate detection at scale? MinHash with locality-sensitive hashing is the production standard. It converts each document into a compact signature and flags pairs above a similarity threshold without pairwise comparison. RefinedWeb and FineWeb both used MinHash-based fuzzy deduplication on Common Crawl-scale corpora — trillions of tokens.

Should I deduplicate before or after chunking documents? Before. Chunk boundaries shift between near-identical copies of a document, so duplicates evade chunk-level matching. Deduplicating whole documents after normalization — and before chunking, embedding, and indexing — catches copies reliably and saves the cost of embedding them at all.

Can too much deduplication hurt data quality? Yes. Hugging Face's FineWeb team found that deduplicating globally across 96 Common Crawl snapshots performed worse than deduplicating within each snapshot, because aggressive global dedup removed well-distributed high-quality text while keeping each snapshot's unique junk. Tune the threshold; don't maximize it.

Put ScrapeOps on your data problem → dekryptlabs.com

Abhishek Gupta is Co-Founder at Dekrypt Labs, building ScrapeOps — the data acquisition engine that turns any question into clean, deduplicated, comprehension-ready sources. dekryptlabs.com