Defining the Minimum Web Data Contract for an AI Feature

47Billion's 7Seers platform ran into the problem in its sharpest form: a teacher asked for a slide deck on a live policy topic, and no amount of prompt engineering could make up for the fact that the model's training data stopped months earlier. The pipeline didn't throw an error. It returned content that was structurally valid and completely empty, the right shape with nothing inside it, and the failure only became visible once the LLM tried to turn that nothing into slides and produced a deck that technically had titles, bullets, and formatting, but said nothing at all.

Across AI features built on live web data, teams spend their first weeks arguing over which model to use, when the actual ceiling on what the feature can do is the content pipeline feeding that model. A bad model produces a bad answer you can catch in testing. Empty JSON, a hallucinated summary, a cached price that no longer matches reality, none of these announce themselves. They look fine until someone downstream tries to act on them.

That asymmetry changes the first question a team should ask when building an AI feature. Whether the pipeline gave the model anything real to reason over in the first place matters more than whether the model is good enough.

Answering that question means specifying, before a line of code gets written, what the feature needs from its data layer: what shape the data has to arrive in, how old it can be, what fields it has to contain, and how it gets from the source to the feature. The contract has four clauses, each answering a distinct question.

What a minimum web data contract is

A minimum web data contract is the explicit, written-down specification of what a feature requires from its data layer, covering format, freshness, structure, and delivery, and nothing beyond that.

The word "minimum" carries the weight here, and it cuts in both directions. Specify too much, and the contract locks the feature to implementation details that are guaranteed to change: a specific CSS selector, an exact page layout, a particular vendor's schema. Those details shift constantly on the live web, and a contract built around them breaks every time they do. Specify too little, and the feature's real assumptions stay implicit. The team only discovers them once something fails in production rather than while the feature is still being designed.

The discipline is to find the floor: the smallest set of requirements the feature genuinely cannot function without. Everything above that floor is a build decision, not a contract term. A team can choose a particular scraping vendor, a particular caching layer, a particular database.

The contract has four clauses, each answering a distinct question.

Format asks what shape the data has to arrive in for the model to use it without a separate parsing step. Freshness asks how old the data can get before the feature starts giving wrong answers. Structure asks which fields the feature actually requires, defined precisely enough that incoming data can be checked against the definition. Delivery asks how the data reaches the feature at all, whether that's an on-demand fetch, a scheduled crawl, an event trigger, or a protocol-native tool call.

The format clause: why clean Markdown or structured JSON is a design decision, not a default

The format a model receives web content in shapes three things at once: how many tokens the request costs, how well the model retrieves the right information, and whether the model's reasoning holds together at all. Most teams never actually decide this. They take whatever shape their scraper happens to hand back and build on top of it.

Raw HTML is actively harmful to feed an LLM. A page's HTML is full of navigation menus, footer links, ad markup, and script tags that have nothing to do with the content a user actually wants. When a model produces an incoherent or hallucinated answer from a pipeline like that, the model usually isn't the weak link. The input it got was incoherent, and the output matched.

The format decision comes down to two real cases. The first is LLM-ready Markdown, either the full page or a content-only variant, and it fits RAG pipelines, research summarization, or any feature that needs the model to reason in prose over a page. Production guidance on this point is consistent: clean Markdown built for RAG beats raw HTML cluttered with navigation and boilerplate, and a "main content only" variant cuts token consumption further still. The second case is structured JSON, and it fits features that need to validate, compare, or act on specific fields, things like prices, availability, brand attributes, or entity metadata, where a prose summary is the wrong shape for the job entirely.

That choice locks in two downstream consequences. The first is cost. LLM inference cost scales with how much text goes into the request, so a content-only Markdown version of a page costs less per call than the full page, and considerably less than raw HTML. For any feature paying per token on inference, that's a direct lever on the bill, not a rounding error.

The second consequence is schema compatibility, and it's the one that catches teams off guard. A feature built to expect JSON fields cannot gracefully fall back to a paragraph of prose when the pipeline hands it something unexpected. That means the format clause has to get settled before the structure clause does, because the two constrain each other: the shape of the output determines what kind of schema can even sit on top of it. Some data layers make this an explicit parameter rather than an afterthought, a JSON endpoint that returns structured data directly, and a separate endpoint supporting multi-page research with source URLs attached to the response. Format, treated this way, is a parameter the feature sets explicitly.

The freshness clause: matching data age to the feature's tolerance for being wrong

Every AI feature has a point at which its data gets too old to be trusted, and that threshold has to be written down explicitly, or the feature will degrade without anyone noticing.

The failure here doesn't look like an error message. It looks like a confident, well-formatted, wrong answer. 47Billion's team found that freshness matters at the scale of days, not months: a RAG system grounded on a government policy page that's a week old will answer questions about that policy incorrectly, and the model's own training data can't fill the gap, because whatever version of that policy the model learned during training is frozen in place. An agent checking a live pricing page works the same way. If it pulls a cached result from three days earlier, it's acting on a price that may no longer exist.

Freshness is a spectrum between "live" and "cached," and different kinds of features sit at different points on it. An agent making real-time decisions, checking a price, checking availability, checking a live status page, needs data that's seconds to minutes old, since any caching layer at all can invalidate the answer. A RAG system retrieving from relatively stable reference material, documentation, policy pages, brand guidelines, can tolerate data that's days to weeks old, and re-crawling when the source changes is usually enough. A scheduled intelligence feature, like competitive monitoring or a content index, sets its own tolerance by the interval it monitors on, and the contract needs to name that interval specifically rather than settle for a vague promise to stay "up to date."

Freshness, treated as a contract term, becomes a runtime parameter: a single knob that runs from "must be live" at one end to "yesterday is fine" at the other, and the feature's design determines where that knob sits.

Pulling fresh data on every request improves accuracy but costs consistency, because a page's content can change between one request and the next, which makes the feature's outputs harder to reproduce and harder to debug. The freshness clause should specify whether the feature needs versioned snapshots, source timestamps on the data it retrieves, or a controlled refresh interval instead of an unbounded live fetch. Reproducibility, in other words, is itself a contract term that has to get decided up front, not a problem to patch in after the fact.

The structure clause: defining what fields the feature requires before choosing how to extract them

Teams that pick an extraction method before they've written down the fields they actually need tend to end up with a pipeline that works beautifully on the demo page and falls apart on the next site it touches.

The traditional trap runs through CSS selectors. A selector-based parser encodes structure as a location on the page, a specific div, a specific class name. When a site redesigns, that selector either points at nothing or, worse, silently points at the wrong element and returns data that looks plausible but isn't what it claims to be. Trawl's 2026 analysis puts the timeline on this bluntly: an hour spent crafting the perfect XPath query, a pipeline that works fine, and then the target site pushes a redesign on Tuesday and the pipeline is dead by Wednesday morning. Separate research cited by Tendem, drawing on findings from McGill University, found that AI extraction methods maintained high accuracy even when page structures changed, a distinct line from Tendem's own claim, attributed to Kadoa 2026, that setup time for this kind of extraction dropped from weeks to hours.

Schema-driven extraction flips the order of operations. The developer writes a JSON schema naming the fields the feature needs, their types, and which ones are required versus optional, before any extraction code gets touched. That schema, along with a natural-language instruction, goes to the extraction layer, which runs the page's content through an LLM and returns structured JSON no matter how the underlying HTML happens to be organized. A redesign on the source page stops being a breaking change, because the extractor is reading for meaning, not for a particular spot on the page.

Two implementation patterns handle this trade-off differently. The first is prompt-mode extraction, where fields get described in plain English and the LLM returns matching JSON for each page, one inference call at a time. It's the most flexible option and fits multi-domain scraping, where page structure varies widely from one source to the next, and some AI extraction APIs run exactly this way: no selectors, just a description of the fields and a JSON response back. The second pattern sits in the middle: use an LLM once to generate selectors for a known, stable domain, then run ordinary selector-based extraction at scale from there, since not every extraction job actually needs a full LLM call on every single page.

Whichever pattern a team picks, the structure clause in the contract has to name three things precisely. First, the exact fields required: typed fields like name, price, currency, availability. Second, which of those fields are required and which are optional, because a missing optional field is a minor gap while a missing required field is a pipeline failure. Third, the validation mechanism the extraction layer has to satisfy before any data enters the pipeline at all, whether that's a Pydantic schema, a Zod schema, or a plain JSON Schema object. Some data layers ship a version of this already built for common page types, an endpoint that classifies a URL and returns a typed product record straight away, which is what a pre-built schema for a known page type looks like in practice. A schema needs a validation step attached to it to function as a contract term; without one it is just a suggestion, and silent failures slip through suggestions.

The delivery clause: matching how data reaches the feature to what the feature does

A feature can get format, freshness, and structure exactly right and still fail, if the mechanism carrying the data to the feature doesn't match what the feature is actually trying to do. An agent that needs an answer right now stalls out waiting on a batch job that runs once an hour. A scheduled pipeline indexing content on a weekly cycle misses the one change, out of thousands, that actually mattered to the user.

Four delivery patterns cover most of what a feature needs, and each one assumes something different about how the feature operates.

On-demand fetch has the feature request data the moment it needs it, with a live result returned inside the same request cycle. This is what an AI agent making a real-time decision requires: a price check, a competitor's current status, a piece of content that has to be pulled fresh in the instant it's asked for.

Scheduled crawl has the pipeline fetch and index content on a set interval, with the feature querying whatever was last indexed rather than fetching live. This fits RAG systems built on reference content that doesn't move much: documentation, knowledge bases, brand guidelines. The contract needs to name the interval directly, since that interval is what defines the maximum staleness the feature can tolerate before its answers start drifting.

Event-driven delivery triggers a re-crawl when the source content itself changes, rather than running on a fixed clock. This suits features where staleness is unacceptable but a constant polling schedule would waste calls on content that rarely moves, catching the moment something actually shifts instead of checking on a timer regardless of whether anything did.

Each of these patterns implies a different pipeline shape underneath it, different latency guarantees, different costs, different failure modes when something upstream goes down. Choosing the wrong one doesn't get fixed by a better model or cleaner extraction. The delivery clause is where the contract meets the actual architecture the feature runs on, and getting it wrong creates a staleness or latency problem that no amount of good work in the other three clauses can undo.