Structured Output and the Validity Ladder for AI
Valid JSON from an AI system is not proof the answer is true. The validity ladder separates format, type, domain and evidence checks before anything acts.
Yash Sharma5 min readContext, retrieval and memory
Research area 06: Structured-output, schema and constrained-generation engineering
In brief
- Valid JSON only proves a parser accepted the shape, not that the content is true.
- The validity ladder checks parse, type, domain and evidence before anything acts.
- Skipping the top two rungs lets a well typed, ungrounded value trigger a downstream action.
Structured-output, schema and constrained-generation engineering forces an AI system's output into a shape another system can consume, then proves that shape carries the truth it claims to. A document that parses cleanly and one that is correct are different achievements, and 2026 research keeps finding teams that shipped the first while assuming they had bought the second.
What it is
Six mechanisms sit on a spectrum from a plain request to a hard guarantee, and none substitutes for the others: prompt-only formatting, a provider's schema mode, grammar-constrained decoding that masks invalid tokens so the final string belongs to an allowed language, a typed intermediate representation, parsing and validation after the fact, and a bounded repair loop, though repair is a new attempt, not an evidence-preserving correction.
Constraining format is a different task from validating content, and passing one says nothing about the other. A decoder's advertised schema support has to be tested construct by construct, since recursion, references, Unicode escapes and cross-field rules are not equally well supported across implementations. An empty field, a zero and a genuinely unknown value are three different facts, and a system that cannot represent unknown will eventually fabricate one to fill the gap. Format compliance is a floor, not a ceiling.
AI systems increasingly generate structured artifacts that other systems act on directly, with no person reading the record in between, making the parseable-to-true gap an operational risk. Reddy and coauthors, arXiv, June 2026, tested a draft-conditioned approach: on a one-billion-parameter model against GSM8K, the strict valid-and-correct score rose from 15.2 to 39.0 percent, bought with a second inference pass. Li and coauthors, August 2026, measured grammar-compilation cost before generation begins: a JSON grammar took 28 to 53 seconds and 3 to 6 gibibytes, a SQL grammar up to roughly 4,770 seconds and 255 gibibytes. Zhang and coauthors, also August 2026, ran six models through 1,500 algorithmic JSON tasks and 600 synthetic table tasks each: structural placement and value correctness fail at different rates.
The architecture
The named framework here is the validity ladder. An output has to climb four separate rungs, parse validity, type validity, domain validity and evidence support, and clearing a lower rung never implies the higher ones are cleared too.
Parse validity asks only whether the output is well formed. Type validity checks whether each field carries its declared type and whether constructs such as numeric ranges are enforced by the decoder, not merely advertised. Domain validity checks the business's own rules, such as whether a start date precedes the event it follows: a record can clear the first two rungs while its fields' relationship to each other is wrong. Evidence support asks whether the value is traceable to the source, not just well typed.
Grammar-constrained decoding masks invalid next tokens at each step and renormalizes the remaining probabilities locally, a different operation from conditioning the whole sequence on the constraint from the outset. In one worked comparison, two next tokens were both locally valid, yet one led to a completion that succeeds nine times out of ten and the other only once in ten: local masking treats both as even odds since it looks only one step ahead. Syntactic reliability can climb while semantic quality falls.
One checkpoint sits above all four rungs: authorization, confirming this caller may trigger the request, independent of whichever vendor produced the output. A failed rung can trigger a bounded repair loop, but only one that preserves the original output, the exact error and the repaired result side by side.
Ownership of the top two rungs sits with the business unit that knows the rules and the source of truth, never with the vendor or engineering alone, since a vendor's schema mode can only promise the lower two. Fund a small, deterministic schema plus independent validation as the default path, and reserve grammar-constrained decoding for workloads where distribution fidelity or complex syntax justifies the logits access and compiled grammar it costs. Measure raw parse rate, post-repair parse rate and semantic correctness as three separate numbers, never folded into one figure, since a compiled grammar pays for itself only past a volume threshold. Keep the raw output and validation errors alongside the final record.
Two decisions belong to an executive: which schema constructs are in scope for a release, since an unapproved construct that works today can silently fail tomorrow, and whether a workflow may execute directly from validated output without a person in the loop, the point at which the ladder's guarantee gets spent.
What it is worth
Money shows up in two places: the cost of a defect that reaches a downstream system unnoticed, and the cost of the checking process itself, since the added inference pass and the multi-gigabyte grammar compilation measured for SQL are real, charged per request or amortized across reuse. Risk concentrates at the authorization checkpoint, since a rule a decoder never modeled cannot be caught by it, however clean the output looks. Measure raw parse rate, post-repair parse rate, semantic correctness and cost per validated request, before and after adoption, at a fixed budget, not your best pass out of several attempts.
Draft-conditioned projection raised a strict correctness score on a small model, a real result. It does not support any universal claim that constrained decoding beats independent validation at equal cost, or that valid syntax can be trusted as semantic truth.
Questions leaders ask
- Does valid JSON from an AI system mean the answer is correct?
- No. Parse validity only means a parser accepted the shape of the output. A 2026 study of six models found that where a value sits in the structure and what the value actually is are different kinds of failure, and a benchmark that repairs the output before scoring it measures a different question than raw correctness.
- Should we adopt grammar-constrained decoding across every system that produces structured data?
- Only where the schema you need and the runtime you operate actually match what the decoder supports, construct by construct. It requires logits access, a compiled grammar and sometimes an extra model, so weigh that operational cost against simply validating output after generation, which works for most low-risk extraction.
- What does forcing a strict output format actually cost us?
- More than the mask itself. One 2026 study measured grammar preprocessing at 28 to 53 seconds and 3 to 6 gibibytes of memory for JSON, and up to roughly 4,770 seconds and 255 gibibytes for SQL, before any generation begins. A fast masking step does not make the whole request cheap if compiling the grammar dominates the bill.
- Should an AI agent repair or retry its own structured output automatically?
- Only inside a bounded loop that keeps the original output, the exact validation error and the repaired result side by side. A repair must never invent the evidence it was missing to close a gap. If repeated repairs keep changing the actual content rather than the formatting, stop and route it to a person instead.
- What should a required field contain when the AI system does not actually know the answer?
- Never a fabricated value used only to satisfy the schema. Required means the field must exist, not that a truthful value is known, so the field should carry a known value with its evidence, an explicit unknown, a not-applicable marker or a refusal rather than a silent guess or an empty string standing in for missing data.
