Keeping a LlamaIndex or Pinecone RAG Index Fresh With Scheduled Recrawls

A web scraping pipeline for a live RAG index should crawl URL sets on a schedule, extract clean Markdown, check for changes, and update only the documents that changed. Keeping a LlamaIndex or Pinecone index fresh also requires explicit deletion of stale chunks and monitoring for failed updates. The practical default is incremental recrawling, backed by periodic full rebuilds, rather than paying to re-embed every page every night.

Why a RAG index backed by live web sources goes stale faster than most teams expect

A policy page changes last Tuesday, but the index still has the old wording. When someone asks about that policy, retrieval finds the old chunk because it still matches the question. The retrieval score can look perfectly healthy. The LLM then reasons over superseded content and delivers a confident answer without knowing that the source has changed. Nothing in that sequence requires a broken query or an obvious error.

Even if the page appeared in its training corpus, that copy is frozen at the training cutoff. The newer policy stays out of reach until fresh content gets to the retrieval layer. This makes source drift harder to diagnose than a failed request, because the system keeps returning plausible answers. Documentation, pricing pages, and support articles pose the same risk when they change but the stored copy stays the same.

The three update strategies and why full nightly re-index is the wrong default for most web-sourced RAG

Full re-indexing deletes the existing content and rebuilds from scratch. It is the simplest approach to reason about, and a clean rebuild removes stale chunk IDs and orphaned vectors. A nightly rebuild pays to process every document, including those that have not changed. For small, infrequently changing collections, that expense can still be a fair price for simpler operations.

Incremental updates instead detect changed pages and re-embed only those documents. Embedding work follows the change rate rather than the whole collection, which favors large corpora with frequent but limited edits. The savings come with an obligation. Change detection must be reliable, and chunk IDs must be tracked accurately, or the pipeline will quietly retain superseded content. Cost and update latency justify this complexity only when enough unchanged content can be skipped.

Versioned shadow indexing addresses a different constraint, namely continuity during replacement. A new index is built and validated while the application continues querying the live one. Cutover must be atomic so queries continue to use a fully built index. The old index stays around for a short time for rollback. This adds infrastructure cost, but it allows updates without interrupting queries and can support the periodic rebuilds that complement routine incremental work.

How to detect which pages have actually changed before deciding what to re-embed

SHA-256 hash diffing of normalized page content is the right starting point for most documentation and articles. It is fast, cheap, and deterministic. A matching hash means the embedding step can be skipped, while a changed hash sends the document toward an update. For a Markdown pipeline, hash the converted Markdown. This puts the change detector and the embedder on the same representation.

DOM element diffing narrows detection to a price field, status badge, or table row when surrounding markup changes often. Text and Markdown diffing compare the cleaned content itself, making them useful when developers need to see what changed in the representation being indexed. Visual diffing compares pixels or rendered pages. It is rarely the right gate for RAG updates, because it serves UI regression testing better than content freshness.

Semantic diffing asks an LLM whether a textual edit changes meaning. It costs more and adds latency, but provides a higher-signal filter where cosmetic edits would otherwise trigger repeated embedding work. Whitespace, punctuation, and boilerplate changes are examples of that noise. When detection runs separately from ingestion, a webhook can trigger re-scraping, re-indexing, and downstream notifications. Structured diffs give those stages a machine-readable handoff without requiring a person to interpret each edit.

Getting clean, LLM-ready Markdown from crawled pages as the input to the embedding pipeline

Raw HTML is a poor choice for the default embedding input. Its tags and page furniture consume tokens and dilute the content retrieval needs to find. Clean Markdown preserves headers, lists, emphasis, and code blocks while exposing their hierarchy without the surrounding markup. That structure helps the LLM make sense of the retrieved passage. Markdown is both the canonical intermediate representation and a handy export format.

The cleaning step also determines what counts as a change. When extraction removes ads, cookie banners, and navigation, hashing that Markdown keeps changes to those parts from counting as edits to the document itself. If irrelevant page elements remain in the output, they remain part of the comparison and the embedding input.

A pricing grid, specification list, or table may work better as schema-driven JSON extraction with defined fields. That approach avoids fragile CSS selectors tied to page layout and yields a directly indexable representation. The choice follows the content being collected. Markdown works well for structured prose, while field-based extraction works well for pages where the useful information is already in records.

Upserting changed documents into a LlamaIndex index without rebuilding from scratch

LlamaIndex provides refresh_ref_docs to insert new documents and update documents whose text or metadata has changed. It does not remove deleted source documents on its own. An external-source sync must also handle those deletions to finish the job of refreshing the pages that remain. Stable document IDs are equally important. Use the canonical URL, or a deterministic hash of it, so retries and recrawls update the same document.

The production pattern is a versioned ingestion transaction. First retrieve the page and validate that the content is non-empty, meets acceptance criteria, and matches the expected schema. Then calculate the Markdown hash and compare it with the stored hash for that URL. A match ends the embedding path. A mismatch starts transformation and indexing of the new chunks, but it does not yet justify discarding the previous version.

The new version must be retrievable before the old version is retired. Only after that check should the pipeline commit the new stored hash and retire the previous document version. If acquisition or indexing fails, retain the last accepted version and record the failure. The same pattern applies with or without Workflows. For TypeScript teams, LlamaIndex.TS was deprecated in March 2026 and its repository archived April 30, 2026, so new pipeline plans need Python or the LlamaCloud managed API.

Managing chunk IDs and stale vector cleanup in Pinecone

Pinecone updates become tricky when a document shrinks or its chunk boundaries change. Upserting its new chunks leaves surplus IDs from the previous version in place, allowing those old vectors to appear in retrieval. The fix is a sidecar record, a simple key-value store that maps each canonical URL or document ID to its current chunk IDs. On update, compare the old set with the new set and explicitly delete IDs that no longer belong.

Pinecone does not automatically expire stored vectors. Deletion must cover document replacement, user or tenant offboarding, and retention policies that require removing vectors past a defined age. For safer transitions, a soft-delete pattern marks vectors with a deleted metadata flag set to true and excludes them through query filters. A periodic hard-delete job then removes the flagged vectors. This keeps vectors out of queries before they are removed, which avoids races with in-flight queries.

Shadow indexing offers another transition pattern. Build updated chunks in a staging namespace or index, verify them, and atomically redirect queries while retaining the old namespace briefly for rollback. Dependency control needs attention too. Pinecone’s SDK upgrade notes state that the package rename from pinecone-client to pinecone began in v5.1.0, and upgrading under the old name does not work as of v6.0.0. Pin dependency versions explicitly so a scheduled ingestion job does not depend on a floating package upgrade.

Scheduling the recrawl pipeline and instrumenting it for production reliability

Crawl frequency should follow the source’s update cadence and the consequence of a stale answer. Documentation and knowledge base articles usually need daily crawls, while pricing, policy, and status pages need to be checked more often. News feeds and changelogs suit hourly crawls or event-triggered updates through webhooks. Static reference content can run weekly or on a known update signal. Tune detection thresholds with these schedules.

Monitoring starts with discovery count and accepted-page rate. These show how many URLs were checked and how many yielded content that passed validation. Change rate records the fraction with a hash mismatch, where a sudden spike or drop can signal trouble in the source or detector. Embedding lag measures the delay between a changed page’s update timestamp and the indexing of its new embedding. Together, these distinguish a quiet source from a pipeline that has stopped accepting updates.

Query-side metrics expose failures that successful crawl jobs can hide. Stale retrieval rate measures retrievals whose embeddings predate the latest detected source change and should stay near zero. Upsert failure rate and rejected-page reasons show where changes were found but failed to reach the index. Citation accuracy and grounded no-answer rate check whether retrieval that works as planned still leads to correct answers.

Each stage needs its own failure boundary. Crawl, detection, embedding, and upsert should have independent retry logic and dead-letter queues so one failed scrape does not stall the whole pipeline. Failures must also preserve the last accepted document intact. Repeated failures for the same URL should trigger alerts.

When to run a full re-index instead of relying on incremental updates alone

Orphaned chunks and missed deletions can accumulate when detection or cleanup logic fails, even if most updates succeed. A full re-index clears those leftovers and provides a clean audit baseline for testing the change detector. It also resets representations when the embedding model or chunking strategy changes. Those changes also affect pages that have stayed the same since the last crawl.

Hash comparisons, version tracking, sidecar records, and deletion logic all require maintenance, and that work is harder to justify when a full rebuild is inexpensive. A nightly full re-index can remain the simpler and more defensible choice for a small or infrequently changing collection.

For most production web-sourced systems, the recommendation is hybrid. Run incremental updates on a schedule matched to each content type, then rebuild the full index weekly or monthly depending on corpus size. The routine path limits repeated embedding work. Where query continuity is required, the shadow-index pattern provides the cutover method for that rebuild.

Putting it together: the end-to-end scheduled recrawl pipeline for a live RAG index

The assembled pipeline begins with a scheduler and configured URL sets. A crawl API supplies extracted page content, which becomes clean Markdown for validation and hash comparison against the stored value for each canonical URL. If the hash matches, update the last-checked timestamp and skip embedding. The crawl still happened, but no embedding work is needed. This is the gate that keeps routine processing proportional to actual source changes.

A changed page enters the versioned ingestion transaction. Validate the content, chunk and embed it, upsert the new chunks, and verify that the replacement is retrievable before committing the accepted version. Then update the sidecar record, remove orphaned old chunk IDs, and retire the previous document version. LlamaIndex handles the insert and update path through refresh_ref_docs, while explicit deletions remain part of the pipeline. Pinecone’s filtered soft-delete pattern can protect the transition before physical cleanup.

A failed upsert preserves the last accepted content. The pipeline records the reason, alerts on repeated failures, and tracks embedding lag, stale retrieval rate, upsert failures, and change rate by URL set. Weekly or monthly full rebuilds then clear accumulated leftovers.