Scoping RAG Features Around Data Freshness Requirements

A RAG pipeline that ships clean often looks worse three weeks later, and the model didn't cause it. The evals passed, the demo worked, and nobody touched the code. The data underneath it changed, and most teams never built anything to catch that.

A nightly crawl runs fine, no errors in the logs, everything green. But two distinct failures hide inside that green checkmark. New pages that showed up after the last crawl simply don't exist in the index, no retrieval trick brings them back. And pages that got edited since the last crawl still sit in the index under their old text, so the embeddings point at content that no longer matches reality. Neither failure throws an error. Neither one looks like a data problem from the outside.

Compare that to a normal ETL pipeline. A field gets renamed upstream, the job breaks, someone gets paged, and the fix typically lands within hours. AI pipelines don't work that way. They ingest the changed content without complaint, quality drifts down slowly over days or weeks, and by the time someone notices, the instinct is to blame the model or start prompt-tuning. The actual fix was never in the model. It was in how the data got refreshed, and how nobody was watching for the refresh to fail.

What data volatility means and why it varies by source

Stock prices move by the second. Annual reports change once a year, if that. Product documentation might get a paragraph edited every few weeks. None of these are the same freshness problem, and treating them like they are causes most stale-RAG complaints.

Volatility belongs to the source. A crawl schedule should be set by how fast the underlying content actually changes, not by whatever cadence is easiest to configure in a cron job. A lot of teams do the opposite: they pick nightly crawls because nightly is the default, then wonder why a pricing page that updates every four hours keeps handing the model outdated numbers.

Two things determine how volatile a source really is. First, change frequency: does this content move in seconds, hours, days, months, or basically never? Second, change detectability: does the source give any signal that something changed, like a sitemap with lastmod timestamps, an RSS feed, HTTP ETags, Last-Modified headers, or a CDC event stream, or does the pipeline have to re-fetch and diff the whole thing blind to find out?

And volatility isn't the same thing as importance. A lot of scoping conversations go sideways over that distinction. A source that changes every ten minutes but nobody ever queries is a waste of crawl budget sitting there burning compute. A source that barely changes but anchors half the answers a system gives is cheap to maintain and critical to get right. Scoping by volatility alone, without asking how much retrieval traffic a source actually pulls, means over-investing in freshness nobody needs and under-investing in freshness everybody depends on.

A practical three-tier volatility classification for RAG sources

Before any crawl schedule gets built, sources need to get sorted. This is a planning exercise, done once per source (and revisited when a source's behavior changes), performed ahead of time rather than decided at runtime.

Real-time, high-volatility sources need updates within seconds to hours. Live pricing feeds, breaking news, security vulnerability databases, stock data, event status pages. Get this stale and the answer isn't just a little outdated, it's wrong in a way that matters. The point is direct: in security contexts, a more recent attack vector is more urgent than an older one. In that category, freshness isn't a nice-to-have quality bar: it separates a correct answer from a dangerous one. Scheduled crawls, no matter how frequent, structurally can't keep up here. These sources need event-driven ingestion or a fetch that happens at query time.

Moderate-volatility sources need updates within hours to days. Product docs, competitor pages, regulatory guidance, SaaS changelogs, news sites. A few hours of lag won't break anything, but a few days will start eroding trust in the answers. This is where scheduled crawls actually work, as long as the cadence gets tuned to the source instead of set to a blanket default. Finance, retail, travel, and healthcare all lean heavily into this tier, since so much of what they publish updates on a daily or weekly rhythm.

Low-volatility, stable sources need updates within weeks, or never. Evergreen reference docs, academic papers, historical records, whitepapers, company founding documents. Staleness here rarely produces a wrong answer, it just occasionally misses a minor update. Infrequent full re-indexing, or a re-crawl triggered only when a change signal fires, is the right level of effort. Crawling this tier nightly is a waste, the freshness gain per crawl cycle is close to zero.

Crawl schedules and re-indexing triggers matched to each tier

The mechanism should follow the tier's change signal, not get copy-pasted across every source in the corpus.

For Tier 1, the crawl schedule mostly gets replaced. A streaming model built on Change Data Capture watches for document changes as they happen and re-embeds only the content that changed, so cost scales with how much actually changed rather than with the size of the whole corpus. Where CDC access isn't available, which is most public web pages, the pipeline fetches and cleans the page at query time and caches the result under a short, source-specific TTL. This is the live-web RAG pattern: discover candidate sources with a search API, fetch them fresh at the moment of the query, and ground the answer with citations back to that fetch. The freshness ceiling shifts from "whenever we last crawled" to "the current state of the page" because that's the only ceiling that actually matches a Tier 1 source's behavior.

For Tier 2, the crawl runs on a schedule, but a smart one. HTTP ETags and Last-Modified headers let the pipeline send a conditional GET and skip re-embedding anything that hasn't actually changed, which keeps compute costs proportional to real change rather than to crawl frequency. Sitemap crawling helps prioritize inside a crawl window: walk the lastmod timestamps in the sitemap and re-fetch the pages that moved recently first. And using incremental versus full reindexing isn't either-or here. Incremental updates catch day-to-day edits, while a periodic full re-index catches accumulated changes that incremental updates can miss. Running both at this tier is standard practice, not overkill.

For Tier 3, RSS feeds or webhooks work where a source offers them. Where they don't, a low-frequency scheduled crawl, weekly or monthly, is enough. Full re-indexing is fine here too, since the corpus at this tier tends to be small relative to Tier 1 and Tier 2, so the batch cost of doing it all at once stays low. One thing still applies across every tier, including this one: even slow-moving sources restructure their pages occasionally, and DOM diffing catches that shift before it quietly corrupts the index.

Freshness metadata and SLA enforcement as first-class pipeline components

A failure that looks identical to success in every log line: the ingestion job finished, exit code zero, nothing red anywhere. But the upstream source went quiet for four days because of some transient issue on their end, the crawl kept "succeeding" against stale content, and the index drifted without a single alert firing. Inference kept running, confidently, on data that had aged out of relevance.

Preventing that means giving every chunk in the index a small set of metadata fields it carries around: the source URL, the timestamp of when that content was last verified current (not when it was last ingested, since a re-crawl that confirms "nothing changed" is a different event from an ingestion that pulled new text), the volatility tier assigned to that source, and the freshness SLA threshold that applies to it.

None of that matters without alerting wired to it. The alert needs to fire when a source breaches its SLA threshold, before the stale data starts degrading answers, not after someone notices the output looks off. Most production AI stacks skip this entirely, and it's not hard to see why: in staging, nothing ever goes stale long enough to trip anything, so the work never gets prioritized until it's already costing someone in production.

At retrieval time, freshness metadata earns its keep in two ways. Hard filtering just drops any chunk older than its SLA threshold from the candidate set before ranking even starts. Soft down-weighting is gentler: it applies a recency prior that lowers the retrieval score for older chunks without removing them outright. On an NVD CVE corpus with mixed topics, a half-life recency prior lifted Latest@10 from 0.00 to 0.60, on a set where plain cosine similarity never once found the newest relevant item. That's the clearest evidence that similarity alone, with no notion of time, is the wrong tool whenever recency is part of correctness.

Schema drift monitoring as the freshness problem teams consistently miss

Content staleness and schema drift look similar from a distance but they're different problems entirely. A source can be perfectly current, updated an hour ago, and still poison the index because a change in the shape of the data produces that outcome as the content stays the same.

Picture a field getting renamed upstream. A JSON field called price becomes unit_price. The ingestion job doesn't throw an error, because there's nothing technically wrong with the request, it just quietly starts ingesting under the new field name (or drops the field, depending on how brittle the mapping is). The model was calibrated against the old shape. Nothing crashes. Quality just degrades, and it looks exactly like model drift to anyone watching the outputs, because a change in the schema causes that appearance while remaining invisible to them.

Conventional ETL doesn't have this problem, or at least not for long, because a schema break there appears almost immediately and gets fixed in hours. AI pipelines absorb the same kind of change silently, and detection can take weeks. That asymmetry, fast failure in ETL versus slow, invisible failure in AI ingestion, is the real engineering risk hiding in most freshness conversations, and it gets the least attention relative to how much damage it does.

Catching it takes different tooling depending on the source. Structured data sources need schema diffing on every ingestion run, with alerts on any field addition, removal, or type change that wasn't expected. Web-sourced data needs DOM structure diffing or XPath contract validation, something that flags a structural change on the source page before that change ever reaches the index. Third-party data feeds need explicit data contracts with version checks enforced at ingestion, so a vendor's silent schema bump doesn't slip through unnoticed.

What the crawl infrastructure layer needs to handle across all tiers

Everything above, the tiering, the crawl cadence, the freshness metadata, the schema monitoring, sits on top of one dependency: an extraction layer that actually delivers clean, structured, model-ready output. If that layer is weak, none of the freshness engineering above it matters.

A few things are non-negotiable regardless of which tier a source falls into. JavaScript rendering matters because a meaningful share of the modern web doesn't exist in the raw HTML response, appearing only after the page executes client-side code. Authentication and rate-limit handling, since source connectivity and access management is consistently one of the layers teams underbuild until it breaks in production. Output format has a concrete effect: raw HTML dumped into a pipeline means someone downstream has to write a parser, so the extraction layer needs to hand back clean Markdown or structured JSON directly, not another parsing job for the next stage to deal with.

For Tier 2 and Tier 3 sources especially, schema-driven extraction pays off twice. A developer defines a JSON schema up front, specifying exactly which fields matter, and that same schema doubles as a drift detector: the moment an expected field goes missing from the extracted output, the schema drift signal fires immediately, not three weeks later when someone notices the answers are off.

There's a maintenance dimension to this too. Traditional scraper-building splits roughly 20% building and 80% maintaining, as sources change layout and break selectors underneath. Extraction that can re-map its own logic when a page's structure shifts cuts down on that 80%, which matters directly for freshness: a scraper that's broken and waiting on a manual fix isn't hitting its crawl schedule no matter how well that schedule was designed.

And a lot of teams end up stitching together a crawler from one vendor, a parser from another, a structured-data extractor from a third, and a separate change-detection layer on top, each piece needing its own maintenance as sources evolve independently. That fragmentation is its own tax on freshness, since every seam between tools is a place where a source's change can slip through unnoticed.

Retrieval logic that respects freshness tiers at query time

None of the crawl engineering above matters if retrieval ignores it. A pipeline can nail the tiering, run the right crawl cadence for each source, tag every chunk with accurate freshness metadata, and still hand back a stale answer if the retriever just grabs whatever chunk scores highest on cosine similarity and ignores the timestamp sitting right next to it.

Freshness metadata has to actually get used at query time. That means retrieval logic checks the SLA threshold and volatility tier tagged on a chunk before it ranks that chunk, applying either the hard filter or the recency prior discussed earlier, tuned to how urgent freshness is for that particular query. A question about a CVE published this morning and a question about stable reference content are not the same retrieval problemd a question about a company's founding date are not the same retrieval problem, and a retriever that scores them identically on similarity alone is going to get one of them wrong.

Getting the crawl side right and leaving the retrieval side untouched is half a fix. The tiering work only pays off once retrieval is built to actually respect it.