RAG Data Freshness: Your Index Is Quietly Going Stale

GPT-4 launched in March 2023 knowing nothing after September 2021 — an 18-month blind spot on day one. Retrieval-augmented generation exists largely to close that gap.

Here is the uncomfortable part: RAG data freshness has the same failure mode as the model it patches. Your index only knows what the web looked like on the day you ingested it.

The short version

• GPT-4 shipped in March 2023 with a September 2021 knowledge cutoff. Retrieval was the industry's answer — but retrieval from a static index just moves the cutoff, it doesn't remove it.
• A RAG index is a snapshot. Pages updated, repriced, or deleted after ingest are invisible to it, and it will cite the old version with full confidence.
• Web change is heavily skewed: in Fetterly, Manasse, Najork and Wiener's 2003 study "A Large-Scale Study of the Evolution of Web Pages," which tracked roughly 150 million pages weekly for 11 weeks, about two-thirds showed no change at all — while a small fraction changed every single week.
• HTTP conditional requests (ETag / If-Modified-Since → 304 Not Modified, defined in RFC 7232) let a crawler check thousands of pages for change while downloading almost nothing.
• The right recrawl cadence is per-source, not global. Schedule by observed change rate, not by calendar.

Retrieval doesn't refresh itself

The original RAG paper (Lewis et al., 2020) made a clean pitch: keep knowledge in a non-parametric index so you can update the index instead of retraining the model. The architecture supports freshness. The operations rarely deliver it.

The standard failure looks like this. A team runs a one-shot ingest — docs, pricing pages, policy pages, a few hundred domains. The demo works. The pipeline ships. Nobody schedules the second crawl.

Six months later the index still answers from January. The pricing page changed twice. The API docs went from v2 to v3. A product line was discontinued and its pages deleted — but the chunks are still in the vector store, still landing in top-k.

Stale retrieval is worse than no retrieval. A bare model at least says "my information ends in 2021." A stale index returns last year's answer formatted as today's, with a citation attached. The citation is what makes it dangerous — it reads as verification.

How often does the web actually change?

Unevenly — and that skew is the whole game. When Microsoft researchers tracked ~150 million pages weekly for 11 weeks, roughly 65% never changed at all, while news, prices, and listings churned daily. So any single global recrawl interval is wrong for most of your corpus.

The skew cuts both ways. Recrawl everything weekly and two-thirds of your fetches retrieve bytes you already have. Recrawl monthly and your fast-moving sources — filings, prices, changelogs — are stale 29 days out of 30.

Scale makes the naive approach worse. English Wikipedia alone logs well over 100,000 edits a day. Common Crawl publishes a new multi-billion-page crawl every month or two — and even that is months old by the time downstream datasets are built on it. If your freshness strategy is "re-ingest a public dump periodically," your floor is their cadence, not yours.

This is also why data licensing went from afterthought to line item: Google's reported $60 million-a-year deal for Reddit data in February 2024 was priced on freshness and exclusivity, not volume. Fresh, structured, continuously updated data is the expensive kind.

Recrawl on a budget: 304s, sitemaps, change-rate tiers

The cheap trick most pipelines skip: conditional requests. Send If-Modified-Since or If-None-Match with your last fetch's timestamp or ETag, and an unchanged page answers 304 Not Modified — headers only, no body. You just verified freshness across a domain for a fraction of the bandwidth of one full crawl.

Second signal: sitemap lastmod. Google has said it uses lastmod when a site keeps it consistently accurate, and the same logic applies to your crawler — trust it per-domain, verify it occasionally, drop it for domains that lie.

Then tier your sources by observed behavior:

The tiers shouldn't be static. Track per-URL change history and adapt like TCP backoff: two consecutive 304s, double the interval; a real change, halve it. Within weeks the schedule converges on each source's actual rhythm — no manual tuning.

One more thing an "append-only" index never does: forget. A 404 or 410 on recrawl should evict that page's chunks from the vector store. Deleted pages that keep answering questions are the quietest data quality bug in production RAG — we covered its sibling, duplicate chunks crowding top-k, in our dispatch on deduplication.

This scheduling problem — thousands of sources, each with its own change rate, checked cheaply and re-ingested only on real change — is the unglamorous half of data acquisition, and it's exactly the layer ScrapeOps runs so pipeline teams don't rebuild it per project.

Freshness is operated, not shipped

Teams agonize over embedding models and chunk sizes — decisions made once — while retrieval quality decays a little every day through the decision nobody made: when to look again.

The index you built in January answers with January. That's not a bug in RAG; it's the default behavior of any snapshot. Fixing the model's cutoff with a snapshot of your own just relocates the staleness where dashboards don't show it.

Budget for the second crawl before you celebrate the first. More engineering notes on data pipelines live in our research and the dispatches archive.

Frequently Asked Questions

How do I keep a RAG index up to date? Schedule recrawls per source based on observed change rate, not a global interval. Use conditional HTTP requests so unchanged pages cost a 304 response instead of a download, re-embed only chunks that actually changed, and evict chunks when pages return 404 or 410 on recrawl.

Why does data freshness matter for RAG? A RAG index is a snapshot of the web at ingest time. Pages updated or deleted afterward are invisible, yet the index cites its old copy with full confidence. That makes stale retrieval worse than no retrieval — the answer looks verified while being outdated.

How often should I recrawl sources for a RAG pipeline? Match the source: news, prices, and filings need hourly-to-daily checks; product and API docs, daily conditional checks; evergreen reference, weekly to monthly. Then adapt per URL — double the interval after consecutive no-change responses, halve it when a change appears.

What is a 304 Not Modified response and why does it matter for crawling? It's the HTTP reply to a conditional request saying the page hasn't changed since your last fetch. The server sends headers only, no body. That lets a crawler verify freshness across thousands of URLs for a fraction of the bandwidth of re-downloading them.

Put ScrapeOps on your data problem → dekryptlabs.com

Abhishek Gupta is Co-Founder at Dekrypt Labs, building ScrapeOps — the data acquisition engine that turns any question into clean, deduplicated, comprehension-ready sources. dekryptlabs.com