Structured outputs in production LLMs

The extraction pipeline returned valid JSON. The invoice parser accepted it. Finance reconciled against a vendor name that never appeared on the document — the model invented a plausible string inside a perfectly typed object. Structured outputs in production LLMs fail in two layers: syntax passes while semantics drift, and teams discover the gap only when downstream systems treat parsed JSON as ground truth. JSON mode and schema-guided generation reduce formatting errors. They do not replace validation, observability, or the eval discipline that defines what "correct extraction" means before traffic arrives.

Large language models emit text. Production systems need objects — typed fields, enumerated values, nested records, arrays with cardinality constraints. The gap between token generation and application contracts is where most AI integration incidents originate. Treating the model as a serializer without a validation boundary is the same category of mistake as trusting client-side form validation for payment amounts.

JSON mode vs schema contracts: what each layer actually guarantees

Provider features fall on a spectrum from loose to strict:

JSON mode solves the trailing-comma and markdown-fence problem. It does not solve the model filling confidence: 0.99 on a guess or mapping "status": "approved" when the source text says rejected. Schema contracts at the API layer narrow the token space — required properties, type constraints, enum membership — which materially improves field presence and type correctness. Schema contracts still allow wrong strings in string-typed fields.

Valid JSON is a transport concern. Correct extraction is a product concern. Only validation and evals separate them.

Production architecture treats provider schema support as the first gate, not the last. The durable contract lives in application code:

import { z } from "zod";
 
const InvoiceExtraction = z.object({
  vendor_name: z.string().min(1).max(200),
  invoice_number: z.string().regex(/^[A-Z0-9-]{4,32}$/),
  total_cents: z.number().int().positive(),
  line_items: z.array(
    z.object({
      description: z.string(),
      quantity: z.number().positive(),
      unit_price_cents: z.number().int().nonnegative(),
    })
  ).min(1),
  currency: z.enum(["USD", "EUR", "GBP"]),
});
 
type InvoiceExtraction = z.infer<typeof InvoiceExtraction>;

Parse failures, type mismatches, and business rule violations surface here — with error messages suitable for repair loops and logging suitable for dashboards.

Where schema contracts belong in the request path

A typical production flow:

User input → prompt assembly → LLM call (schema mode) → raw JSON parse
  → schema validation → business rules → persist / downstream API
       ↓ fail                              ↓ fail
   repair loop (bounded)              human review queue

Prompt assembly includes examples, field definitions, and explicit null-handling ("use null when absent, never invent"). LLM call uses the strictest supported mode — response schema, tool schema, or grammar constraint — matched to provider capability. Validation runs server-side regardless of provider promises; providers change behavior across model versions.

Business rules sit above schema validation: total_cents must equal sum of line items within rounding tolerance; invoice_number must not duplicate an existing record unless flagged as revision. Schema validates shape; business rules validate meaning.

Teams debating fine-tuning as last resort for extraction often have fixable prompt and schema gaps first. Fine-tuning on format-stable tasks can help at scale — but only after eval infrastructure proves the remaining failure mode is model behavior, not missing validation or weak prompts.

Repair loops: bounded recovery when the model misses the contract

First-pass schema compliance in production rarely reaches 100%. Repair loops retry with structured feedback:

MAX_REPAIR_ATTEMPTS = 2
 
def extract_invoice(document_text: str) -> InvoiceExtraction:
    messages = build_initial_messages(document_text)
    for attempt in range(MAX_REPAIR_ATTEMPTS + 1):
        raw = call_llm(messages, response_schema=INVOICE_SCHEMA)
        try:
            parsed = json.loads(raw)
            validated = InvoiceExtraction.model_validate(parsed)
            validate_business_rules(validated, document_text)
            return validated
        except (json.JSONDecodeError, ValidationError, BusinessRuleError) as exc:
            if attempt >= MAX_REPAIR_ATTEMPTS:
                raise ExtractionFailed(str(exc)) from exc
            messages.append({"role": "assistant", "content": raw})
            messages.append({
                "role": "user",
                "content": f"Validation failed: {format_errors(exc)}. "
                           f"Return corrected JSON matching the schema. "
                           f"Do not invent fields not supported by the source.",
            })
    raise ExtractionFailed("unreachable")

Effective repair loops share properties:

Bounded attempts. Three tries with feedback beats infinite retry loops that burn latency budgets and amplify cost on pathological inputs.

Error messages the model can act on. "Validation failed at line_items[2].quantity: expected positive number, got 0" beats "invalid schema".

Assistant turn preservation. Including the failed output in context gives the model a diff target; omitting it forces full regeneration from scratch.

Escalation path. After bounded failure, route to human review, fallback heuristic, or degraded single-field extraction — never silent pass-through of partial objects.

Metrics per attempt. Track first-pass success rate, repair success rate, and terminal failure rate separately. A rising repair success rate with flat first-pass rate signals prompt or schema drift, not model degradation.

Repair loops add latency. For synchronous user-facing paths, cap total timeout including repairs. For async pipelines, repairs are acceptable if queue depth is monitored.

Streaming, partial JSON, and parser hardening

Streaming structured output improves perceived latency but complicates parsing. Token streams may deliver incomplete JSON until the final chunk. Strategies:

Buffer-until-complete. Simplest: accumulate tokens, parse once at stream end. Loses incremental UX but avoids partial-parse bugs.

Incremental parsers. JSON repair libraries and schema-aware stream parsers emit partial objects as fields complete — useful for UI progress indicators. Require fuzz testing on chunked boundaries.

Never execute on partial state. Downstream side effects — database writes, payment initiation — must wait for validated complete objects. Partial preview in UI is fine; partial persistence is not.

Even with JSON mode, defensive stripping of markdown fences handles provider regressions without taking down the pipeline.

Evals for structured output: shape is necessary, semantics is sufficient

Structured output quality requires eval dimensions beyond JSON.parse success:

The AI product evals framing applies directly: product owners define what correct extraction means per field — when null is acceptable, when inference is allowed, when human review is mandatory. Engineering implements validators; product owns the labeled set and pass thresholds.

Regression testing for structured outputs runs on every prompt change, model upgrade, and schema revision. A model upgrade that improves prose quality while degrading enum adherence is a common silent regression — only an eval set with enum-specific cases catches it before deploy.

Observability and failure taxonomy in production

Log structured output failures with taxonomy, not generic 500s:

extraction_outcome: success | schema_fail | business_rule_fail | repair_success | terminal_fail
schema_violation_path: $.line_items[0].quantity
repair_attempt: 1
model_version: gpt-4.x-2025-04-14
latency_ms: 1840

Dashboards slice by model version, document type, and violation path. Alert on terminal failure rate and repair-rate increases. Sample failed payloads into a review queue for eval set augmentation. Store raw model output alongside validated objects for audit replay.

How should teams ship structured LLM outputs?

Implementation decisions that survive model upgrades and traffic growth.

When is JSON mode enough?

JSON mode suffices for low-stakes internal tooling where a human always reviews output, schemas are flat, and retry-on-parse-failure is manual. It is insufficient for automated downstream writes, financial fields, or high-volume pipelines without validation.

Should validation run client-side or server-side?

Always server-side. Client-side schema checks improve UX for interactive flows; the authoritative boundary is the server before persistence. Never trust browser-validated LLM output for mutations.

How many repair attempts are reasonable?

Two repairs after the initial attempt (three total generations) is a common production default for extraction tasks. Tune against latency SLO and cost per document. Log terminal failures for eval expansion.

Does streaming require a different validation strategy?

Yes. Separate display streaming from commit validation. Commit only after complete-object validation passes. Document the UX behavior when stream completes but validation fails — show error, offer retry, never partial save.

A common argument runs the other way

The opposing view holds that strict schemas and repair loops add unnecessary complexity — frontier models with JSON mode are reliable enough to parse once and proceed.

Reliability is task- and tail-dependent. Median-case JSON from current frontier models looks excellent in demos. Production traffic includes scanned PDFs, ambiguous tables, multilingual headers, and adversarial inputs. Schema pass rate on a 50-example demo set does not predict p99 behavior on ten thousand daily invoices. Repair loops and validation cost milliseconds and cents; hallucinated vendor IDs cost reconciliation hours and audit risk. The complexity is proportional to the cost of being wrong — which is why automated pipelines never skip the validation layer even when the model "usually works."

Key takeaways

• JSON mode guarantees parseable syntax, not correct fields or business meaning.
• Schema contracts at the API layer plus server-side validation form the durable production boundary.
• Repair loops with bounded attempts and actionable error feedback recover format failures without infinite retry.
• Business rules above schema validation catch semantically valid but factually wrong extractions.
• Eval sets must score field-level accuracy and hallucination — not just schema pass rate.
• Stream for UX; validate and commit only on complete, validated objects.
• Observability taxonomy (schema path, repair attempt, model version) turns failures into eval improvements.

Conclusion

Structured outputs turn language models into components downstream systems can consume — but only when syntax guarantees are layered with validation, bounded repair, and eval-defined correctness. JSON mode and provider schema features reduce the first-pass failure rate; they do not eliminate the obligation to reject bad objects before they reach the database.

The production checklist: response schema enabled, server-side validator defined, repair loop bounded, business rules documented, eval set covers shape and semantics, raw output logged for audit. Missing items map to incident categories — invented fields, silent type coercion, unbounded retry storms — that structured output architecture exists to prevent.