Wiring a Web Crawl API Into a LangChain RAG Pipeline

A web scraping pipeline for LangChain RAG follows four stages, which are crawl, clean, chunk, and retrieve. The crawl API should deliver clean Markdown and source metadata before LangChain handles splitting, embedding, and retrieval. Reliable answers depend on preserving those handoffs and refreshing the index as the source pages change.

Why static RAG pipelines quietly degrade over time

A documentation page changes, but its old chunks stay in the index. Similarity search can still rank those chunks highly because their embeddings are based on what the page said before it changed. Nothing about a strong match proves that the answer is current. A chunk embedded six months ago may still fit the question but be wrong about the product. Pricing pages, product specs, and documentation are especially at risk because the material that helps ground an answer can also change often.

The failure is easy to miss in practice. A pipeline can keep retrieving passages that read well and closely match the question without showing that the source has been rewritten. Changing the prompt or retrieval model does not swap those passages for up-to-date ones. The fix belongs in the data pipeline, where crawling must become a recurring operation. That requirement shapes everything from the metadata captured during collection to the way stored chunks get replaced.

The four-stage pattern that structures every crawl-to-RAG integration

Crawl discovers and fetches pages from a target site or URL set, producing content with a source URL and crawl timestamp. Clean removes navigation, footers, ads, and other boilerplate, then converts the useful content to Markdown. Chunk splits that Markdown into passages suitable for embedding without discarding its structure or metadata. Retrieve embeds and stores those passages, then returns the most relevant ones at query time.

The tool boundary should be just as clear. The crawl API absorbs crawl and clean, so LangChain receives prepared documents. LangChain then handles chunking and the embedding, storage, and query operations grouped under retrieval. If HTML crosses that boundary instead, the application inherits cleaning work that the API was meant to remove. If metadata is lost, even clean passages lack the information that citations and freshness checks need.

What the crawl API must produce before LangChain sees anything

Clean Markdown is the output requirement. Raw HTML has scripts, styles, and navigation that use up tokens without helping answer the user's question. Passing that material downstream inflates embedding costs and introduces noise into the representation of each page. The API should remove boilerplate and perform Markdown conversion server-side. Otherwise, the LangChain layer needs a parser that the team must write and maintain.

Structure must survive that cleaning step as well. Headings and paragraph boundaries give the splitter useful places to divide a page, while source metadata keeps the resulting passages traceable. Each returned document needs a Markdown body that maps to page_content, a sourceURL, and a crawl timestamp. The URL supports citation, while the timestamp supports freshness checks. Neither can be recovered from an embedding alone.

For full-site indexing, the API also needs multi-page crawl mode. A documentation root should return a separate Markdown document for each page found beneath it that is within the configured scope. Fetching a single URL is a different capability because it leaves link discovery to the application.

Wiring the crawl API into LangChain: the crawl and load step

Start by choosing the collection mode that fits the job. Crawl mode discovers and fetches pages under a root, up to a configurable page limit, which suits full-site indexing. Single-URL mode fetches one chosen page and works for an agent that needs targeted, on-demand retrieval. The choice must be made before anything is loaded into LangChain. It determines whether the API is collecting a corpus or answering a narrow request for live content.

Next, set the scope of that collection. Use the page limit to cap the crawl and URL inclusion or exclusion patterns to skip paths the assistant does not need, such as auth pages or changelog archives. Decide whether the response should have screenshots or structured JSON as well as Markdown. Set those output choices in the API configuration and leave them out of the chunking logic. The text for RAG should still be clean when it arrives.

Loading is then a direct mapping into LangChain's Document object. Put the Markdown body in page_content and preserve the source URL and crawl timestamp in metadata at this first handoff to support citations. Once documents have been split and stored, missing provenance becomes a problem across the whole index.

Chunking Markdown for embedding without destroying its structure

Splitting at arbitrary character boundaries can detach a heading from the explanation it introduces or leave a passage starting mid-sentence. That fragment may embed as topically ambiguous, matching questions it should not answer and missing questions it should. Markdown-aware splitting first uses the document's structure, then falls back to smaller boundaries where needed. Cleaning makes this possible, but it does not replace the need for a suitable splitter.

Use RecursiveCharacterTextSplitter with a custom separator order that starts with \n# , then \n## and \n### , followed by \n\n and \n. This gives heading boundaries priority over paragraph and line breaks. A chunk size of 1,000 characters is a practical starting point for documentation content, with enough overlap to preserve concepts that cross a boundary. The goal is a passage that retains enough surrounding text to make sense when retrieved on its own.

For example, a poor split leaves a heading in one chunk and starts the next halfway through its explanation. A better split keeps the heading with the passage it names. Preserve attribution at the same time by calling split_documents(), which inherits metadata from each parent document. split_text() does not propagate that metadata. Confirm that every resulting chunk still has its sourceURL and timestamp, or add them explicitly before embedding.

Embedding, storing, and retrieving with LangChain's vector store layer

Once the chunks are ready, embed their text with OpenAIEmbeddings or another compatible embedding model through LangChain's vector store integration. Each chunk becomes a dense vector representing its semantic content. The store uses those vectors to find passages related to a query. This is where retrieval becomes searchable storage. It cannot compensate for passages that were noisy, incomplete, or stripped of their source information upstream.

Choose storage according to corpus size and persistence needs. FAISS suits in-memory prototypes and small corpora, while Chroma provides local persistence. Pinecone and Qdrant offer managed, production-scale options integrated with LangChain.

The full connection runs from crawl API output to Document objects, then through split_documents() into embedding and vector storage. A retriever supplies relevant chunks to an LLM call through a retrieval chain, such as one built with create_retrieval_chain. Include each passage's source URL and crawl timestamp in the prompt so that citations are possible and the application can flag material older than its freshness threshold. Start similarity search at k=4, evaluate answer coherence, and adjust. Retrieving extra passages by default can fill the prompt with marginal matches.

Keeping the index fresh: when and how to re-crawl

A refresh plan should follow how the source changes. For a small corpus, a scheduled full re-crawl is the simplest pattern. Run the crawl against the entire site daily or weekly, depending on content velocity, then clear and rebuild the vector store. This keeps the collection process straightforward because each run replaces the indexed snapshot. Its cost becomes less attractive as the corpus grows and much of the content stays stable.

Targeted re-crawling narrows the work to known high-churn pages. Keep a list of pricing pages, release notes, or API reference pages and refresh them more often while leaving content that has not changed alone. Change-triggered re-crawling goes further by using website change detection to start a crawl only when a page changes. That avoids unnecessary re-indexing but adds monitoring infrastructure. The choice is between a simple schedule, a maintained priority list, and a monitored trigger.

For larger corpora, use update-in-place operations keyed by source URL where the vector store supports them. Freshness filtering provides a separate query-time safeguard. Before retrieval, optionally exclude chunks whose crawl timestamps fall outside a configurable threshold. This prevents those older chunks from reaching an answer, but does not fetch their replacements.

Extending the pipeline to agentic and structured extraction use cases

The same crawl API can serve live requests as well as scheduled indexing. Wrap it as a LangChain tool so an agent can choose a page to fetch when a question needs current web evidence. In JavaScript, DynamicStructuredTool can use a Zod schema to define the input, which can include a URL string. The agent chooses when to invoke the tool. The API still owns the work of fetching and cleaning the page.

A two-tool research pattern separates discovery from extraction. A search_web tool returns candidate URLs, while an extract_content tool fetches clean Markdown from a selected URL. The agent makes those calls in order. This changes when collection happens without changing the quality required at the crawl-clean boundary. On-demand content still needs to be usable before it reaches the model.

For a pricing or features page, pass a JSON schema to the API's extract endpoint requesting a name string, a pricing array, and a features array. A structured record replaces the free-form Markdown used for prose chunks. That works for competitive intelligence, brand data, and other downstream uses that need the data in fields. The shared principle is to make the API return the form its next consumer needs.

Common wiring mistakes and how to avoid them

Discarding metadata during loading leaves chunks without usable attribution. Each Document needs explicit mappings for sourceURL and the crawl timestamp, with confirmation that both survive splitting. Without both, citations and freshness filtering fail even when the text that comes back looks useful.

Using a character splitter without Markdown-aware separators weakens passage coherence. Headings can become detached from the content they explain, leaving semantically ambiguous fragments. Use RecursiveCharacterTextSplitter with separators that go by heading level first, then by paragraph and line boundaries.

Setting k high as a precaution can reduce answer coherence. Marginally relevant chunks compete with useful evidence in the prompt. Start at k=4. Evaluate the answers and adjust from there.

Treating the index as immutable lets source changes accumulate unnoticed. Implement a refresh pattern, starting with a simple scheduled re-crawl where that fits the corpus.

Crawling without URL filters can mix unrelated content types. A documentation site may include a blog, a changelog, and an API reference, but the assistant may only need one of them. Use inclusion patterns to collect the content the RAG system is meant to serve.