Converting Raw HTML to Clean Markdown for LLM Input at Scale

HTML was built for browsers, not for language models. Every tag tells a rendering engine where to draw a box, which font to load, or which script to run. None of that carries meaning for a model trying to answer a question about the words on the page. When an LLM has to read raw HTML, it has to wade through markup, inline styles, navigation menus, and tracking scripts before it reaches a single sentence of actual content.

That wading has a cost. The model's attention has to split between structural noise and the meaning it's supposed to reason over, which makes inference slower and more expensive while making the answers worse. Think about how much of a typical web page is actually article text versus how much is cookie banners, footers, sidebar ads, and nav links. On most sites, that chrome accounts for a large share of the raw token count, leaving only a small fraction as substance.

Markdown solves this by keeping only what carries meaning: headings, lists, links, code blocks. It drops the presentation layer, producing a format a model can parse directly, with far less overhead standing between it and the content it needs to reason about.

The three-layer pipeline clean conversion requires

Stripping HTML tags out of a page sounds like a one-line script. In practice, getting from an arbitrary URL to clean, LLM-ready Markdown takes three distinct layers, and a mistake in any one of them carries forward into everything downstream.

The first layer is JavaScript rendering. Most modern websites build their content dynamically in the browser, so a plain HTTP request only grabs the initial HTML shell, often before any of the real content has loaded. Tools like Playwright solve this by automating an actual browser: they run the page's JavaScript and capture the fully rendered DOM before any extraction starts. Skip this step, and dynamic pages, like product listings, news feeds, or documentation built as a single-page app, hand back HTML that's nearly empty, because the content never finishes loading before the request completes.

The second layer is boilerplate extraction. Once the DOM is fully rendered, the pipeline needs to separate the actual content from navigation menus, ads, cookie banners, scripts, and footers. This is a content-detection problem: the pipeline must identify which text is meaningful, a harder task than simply removing tags. On one site that <div> structure might hold the main article, but on another it might hold a sidebar ad. Libraries like trafilatura handle this with heuristics and content-scoring to find the main body of a page. Get this step wrong, and boilerplate rides straight into the Markdown output, where it appears in retrieval results.

The third layer is structure-preserving conversion. The isolated content has to become Markdown that keeps the document's real structure, heading hierarchy, lists, code blocks, tables, without ballooning into verbose or over-converted syntax. The choice of converter library at this layer affects fidelity, token efficiency, and how messy edge cases like nested elements or malformed HTML are handled.

Converter choice at Layer 3 and its effect on token efficiency and pipeline throughput

Picking a converter isn't a detail to wave away. Different libraries handle output verbosity, token efficiency, streaming, and raw speed in different ways, and at high document volumes those differences add up fast.

On the Python side, several libraries serve different parts of this job. markdownify runs on BeautifulSoup4 and handles custom tags well, with active maintenance behind it, but it runs slower than the alternatives and isn't built for high-throughput pipelines. html-to-markdown supports full HTML5, ships with type hints, handles tables well, comes with a CLI tool, and sees active development, though it needs Python 3.10 or newer. trafilatura stands out specifically for content-extraction pipelines: it's fast, strips boilerplate well, pulls metadata, and ships a CLI tool, effectively covering both Layer 2 and Layer 3 by combining extraction and conversion in one pass. html2md offers async support and runs fast, fitting naturally into async pipeline architectures, and also needs Python 3.10 or newer. domscribe supports full HTML5 and handles tables well, though its development activity has slowed.

The JavaScript ecosystem has its own landscape. mdream, introduced in May 2025 by Harlan Zw, takes the opposite approach: a zero-dependency JavaScript library, around 10 to 19 kilobytes gzipped, built specifically for LLM consumption. It processes 1.8MB of HTML in roughly 60 milliseconds, about three times faster than Turndown on typical workloads. It produces what its creator calls Minimal GitHub Flavored Markdown, cutting token output by roughly half compared to conventional converters, through smarter formatting decisions. It also offers a streaming API, so a model can start processing the first chunks of Markdown before the rest of the document finishes converting, removing a serialization bottleneck that matters a lot in real-time pipelines.

Pandoc remains the high-fidelity option for batch work from the command line, supporting both GitHub Flavored Markdown and CommonMark as output dialects. It has a steeper learning curve than the others, but it fits large offline batch conversion jobs well.

The pattern that matters for anyone architecting these pipelines: speed and token efficiency reinforce each other. A converter that produces bloated output and can't stream creates a double cost, more tokens for the model to chew through downstream, plus a synchronous wait before any of it can be used in a live agent workflow.

Specialized small models as a replacement for general-purpose LLMs in the conversion step

Running a frontier LLM to convert HTML into Markdown works, but it's the wrong tool for high-volume pipelines. A general-purpose model has the raw capability to do the conversion, but its cost profile doesn't fit a task that needs to run millions of times a day. Task-specialized small models can hit similar quality at a fraction of the compute.

ReaderLM-v2 is one example: built on a small Qwen2.5-Instruction model and trained specifically for HTML-to-Markdown and HTML-to-JSON extraction. It works for this task, but on consumer hardware it converts notably slower than rule-based converters like the ones covered above.

A 2026 research paper on a crawling system used a small, task-specialized model as its backbone for a specific reason: general-purpose LLMs show strong zero-shot generalization on HTML conversion, but their compute cost makes them impractical to run across a large-scale crawling pipeline. The research goal was to distill that general capability into a small, task-specialized model that stays efficient at scale.

General models still have a place in this decision. Prompt-based extraction from a frontier model skips schema maintenance and fine-tuning entirely, which matters for low-volume work or content so varied that a specialized model would need constant retraining to keep up. Specialized small models win on throughput and cost once volume gets large. General models win on flexibility and lower maintenance overhead when volume is low or the content is too heterogeneous to standardize. The right choice depends on the workload, not on which model is more capable in the abstract.

How boilerplate removal quality directly determines RAG retrieval accuracy

A RAG system can only be as accurate as the extraction layer feeding it. If boilerplate survives into the Markdown that gets indexed, it contaminates the vector embeddings built from that content, and irrelevant chunks appear in retrieval results.

The mechanism is straightforward. RAG retrieves by embedding similarity, so if a chunk mixes navigation text in with actual article content, it produces an embedding that doesn't match either kind of query reliably. If a user asks a question about the article, the system might hand back a chunk half-full of menu links.

Freshness adds a second failure mode. A RAG system grounded in week-old content will give wrong answers on anything fast-moving, government policy changes, job listings, pricing. A model's training data can't substitute for that, because the training snapshot is frozen at a fixed cutoff and doesn't update on its own.

Clean Markdown solves a separate but related problem: consistent chunking across sources that all structure their pages differently. That consistency matters most for RAG pipelines pulling from many different domains, each with its own layout conventions. A URL-to-Markdown conversion step acts as a standardization layer in this setup, absorbing the web's structural chaos and handing back a predictable document format that the ingestion pipeline can chunk and embed the same way every time.

Skip proper boilerplate removal, and vector databases accumulate what amounts to ghost chunks, embeddings built from cookie banners, footer text, and nav menus, and retrieval precision degrades as the index grows.

Structured JSON extraction via schema

Clean Markdown is the right target when the job is retrieval or summarization. For AI agents that need to act on extracted data, filling a database, kicking off a workflow, comparing records against each other, prose Markdown isn't enough. What those agents need is structured JSON, built against a schema you actually define.

Structured data extraction pulls specific data points out of a page and organizes them into a predefined format, JSON or CSV, so you can feed them directly into analytical tools or machine learning models with no further parsing step in between.

The schema itself turns out to be its own failure point. Amazon researchers behind the PARSE system (arXiv:2510.08623) found that most current approaches treat JSON schemas as static contracts written for human developers to read. If a schema contains anything ambiguous or incomplete, that assumption leads to weaker extraction performance, more hallucinations, and agents that behave unpredictably. A schema written for a person to understand isn't automatically a schema a model can follow reliably.

PARSE optimizes schemas specifically for LLM consumption, rewriting the contract so a model can follow it more precisely. That approach produced up to a 64.7% improvement in extraction accuracy on the Structured Web Data Extraction (SWDE) benchmark, and extraction errors dropped sharply within the very first retry once the schema had been optimized this way.

The lesson for anyone building these pipelines: schema quality carries as much weight as extraction quality. If a team puts real engineering effort into clean Markdown conversion but ships a vague or ambiguous schema, hallucinations still occur at the extraction step. No amount of clean input fixes a schema the model can't interpret correctly.

The build-vs-buy decision at scale: self-hosted pipelines versus API-first infrastructure

Building the full pipeline in-house, rendering, extraction, conversion, and schema enforcement, means taking on a maintenance surface that grows with the structural diversity of the web itself, not with the complexity of the product being built on top of it.

The self-hosted version of this stack usually looks like: Playwright for rendering, trafilatura or a custom extractor for stripping boilerplate, a converter library for the Markdown output, and then custom logic layered on top for anti-bot handling, rate limiting, crash recovery, and schema enforcement. Each of those pieces carries its own dependency chain and its own way of breaking.

Open-source tools like Crawl4AI cut down that assembly cost considerably. It produces clean Markdown tuned for RAG use, with keyword-based content filtering built in, supports LLM-powered structured extraction directly, and handles full-site crawling with link following, parallel crawling, and crash recovery already wired in.

Self-hosting still makes sense if a team has unusual privacy requirements, extraction logic that needs heavy customization, or the engineering capacity to keep the whole stack running as the web keeps changing underneath it. For everyone else, you pay the real price of building this from scratch in the maintenance cost of assembling and babysitting three separate layers, each with its own edge cases.