Structured Output: Getting LLMs to Return JSON You Can Actually Trust

Any system that hands an LLM's output to another piece of software — which is most agent architectures — lives or dies on whether that output reliably parses. "Return valid JSON" as an instruction gets you close, and close is exactly the failure mode that shows up in production three weeks after launch, on the one input shape nobody tested.

Use schema enforcement, not instructions

Every major provider now supports constrained decoding against a JSON schema — the model is architecturally prevented from producing output that violates the schema, rather than merely asked to. If your provider or model supports it, use it as the default, not a fallback. It converts an entire category of production bug into something that cannot happen.

When you cannot enforce the schema

• Validate every response against the schema before it touches downstream code — treat a parse failure as an expected case to handle, not an exception to hope never fires.
• Retry with the validation error fed back to the model — "your last response failed because X" recovers far more often than a blind retry.
• Set a hard retry ceiling with a defined fallback (a safe default, a human handoff) — an infinite retry loop is a production outage waiting to happen.
• Log every validation failure with the raw output — these are the highest-signal examples for your next eval-set update.

The parse-and-hope approach — try/catch around a JSON.parse and pray — is the single most common root cause we find when debugging a client's "the AI feature randomly breaks" ticket. It is rarely random. It is an unhandled shape the model was always going to produce eventually, given enough traffic.

Keep the schema boring

Deeply nested, highly conditional schemas are harder for a model to fill correctly, even with enforcement. Flatten where you can, make fields optional rather than conditionally required, and resist the urge to encode complex business logic into the schema shape itself — that logic belongs in the code that consumes the output, where it is testable in the ordinary way.