Deterministic by Design
The data pipeline runs on an architecture that took months to converge on. There's already a name for this shape: a DEGP — a Deterministically Encapsulated Generative Pipeline. The developer didn't coin the term; he converged on the pattern independently and found it already had a name. The label matters less than what it enforces: the AI handles discovery. The deterministic tail handles everything else. And between them sits a barrier that refuses to let anything unverified through.
The problem with AI-generated data
AI agents are good at open-ended discovery. Give one a manufacturer website with product specs spread across five pages, PDF downloads, and an accordion menu, and it will find the data. It will navigate. It will extract. The problem is not accuracy — it is discipline. Left unconstrained, an agent stops following its skill steps. It wastes time veering off into tangents. It brute-forces workarounds instead of following the correct path. Given enough rope, it will spend 20 minutes re-scraping a page it already captured because it lost track of what step it was on.
Traditional data pipelines do not wander. They also cannot handle the entropy of the open web. Every brand structures their product information differently. Some use PDFs. Some use JavaScript-rendered tables. Some require clicking through a product selector that loads data via XHR calls. Rules-based scrapers break the moment a site changes its markup. AI agents do not break. They just need to be contained.
The two approaches solve different halves of the problem. One handles structure but not entropy. The other handles entropy but not discipline. Neither alone works at scale for data that has regulatory consequences.
The architecture
The pipeline splits into two halves separated by a barrier.
The top half, the onboarding head, handles discovery. An AI agent — model-agnostic, any capable runtime works — follows skill contracts to find product data across unstructured sources. It maps domains, identifies data locations, and packages raw evidence. This is where the entropy lives. The agent can operate freely here because its output never touches the database directly.
Between the two halves sits a de-isolation barrier. Everything the agent produces must pass schema validation and gate contracts before crossing. If evidence is missing, if a fact does not verify against its source, if the data shape is wrong — the item goes to held, not to production. No incomplete extraction is ever applied live. The barrier does not judge correctness. It judges completeness. It catches missing evidence, malformed output, and schema violations. It catches the exact failure modes an undisciplined agent produces when it veers off course.
The bottom half, the deterministic tail, does the heavy lifting. Once a domain is mapped and evidence is packaged, extraction, promotion, and classification run purely on rules. No model calls. No API keys needed for steady-state operation. A Python orchestration engine replays packaged facts against the database and object storage only. Same input always produces same output.
This is not a framework. It does not need LangGraph or a state machine library. The database handles state persistence. The barrier handles trust. The tail handles scale.
Why keyless matters
The deterministic tail calls no model. This is not a cost optimization — it is an architecture decision. Every fact is re-derived from its source document, not recalled from a model. The tail reads the actual nutrition panel that was captured during discovery and extracts the number deterministically. There is no memory to drift. No training cutoff to worry about. No provider API change that silently alters output quality.
The economic consequence is that enrichment costs are predictable. Per-record pricing is a business decision, not a function of token consumption. The pipeline's margins do not evaporate when a provider raises rates.
Where it runs
The pipeline powers K12 Products AI. It ingests product data from public and manufacturer sources, enriches it with nutrition facts and compliance information, and serves it through a search platform that school nutrition professionals use to build compliant menus. The pipeline is the product. The enriched data is what customers query. The architecture is how it stays trustworthy without per-record AI oversight.
The barrier is the feature. Every fact in the database crossed a gate. Every gate failure produced a held, not a guess. When a school nutrition director looks up a product's sodium content, the number came from a document that was captured, verified, and re-extracted deterministically. The AI was never in the path between the source document and the database cell.
What this means
The architecture pattern is teachable. The domain-specific extractors, the reviewed source ledger, and the accumulated authority mapping are not. Publishing the pattern does not give away the farm. It establishes the standard. A competitor could read this and build their own DEGP — but they would still need to write the extractors, map the domains, and accumulate the source trust that makes the pipeline produce reliable data.
DEGP is an existing pattern, not something this work coined. What the developer is adding is a worked example: how it holds up in production on data with regulatory consequences. The architecture was always there in how he worked, and putting the established name to it just makes this version easier to discuss, review, and improve alongside everyone else's. The next person who builds a data pipeline that uses AI for discovery but trusts rules for extraction will be building a DEGP whether they call it that or not.
The standard is the moat.