Structured outputs: getting reliable JSON out of a language model

The first version of every LLM integration parses prose. You ask for "the three main risks, one per line", split on newlines, and it works until the day the model writes an introductory sentence first. Structured outputs exist to end that loop: you hand the model a JSON Schema, the decoder is constrained so it cannot emit anything that violates it, and you get back a payload with the fields you asked for. That guarantee is real but narrower than it sounds, and the difference is where the bugs live.

What the guarantee actually covers

Constrained decoding guarantees shape. The response will parse as JSON, it will have your keys, the types will match, and enum fields will contain one of your listed values. That is a large improvement over prompt-and-hope.

It guarantees nothing about content. A field typed as a string will contain a string; whether that string is accurate, or invented to satisfy the schema, is unchanged from before. In fact a strict schema makes invention slightly more likely, because a required field must be filled with something, and the model has no way to say "not present in the source". This is the single most common failure in production: a required invoice_number in a document that has none.

Design around it. Allow null where absence is possible, add an explicit found: boolean or a "unknown" enum member, and never make a field required unless the answer genuinely exists in every input.

Designing a schema a model can fill

Schemas written for validation and schemas written for generation look different. A few rules that hold across providers:

• Flat beats nested. Every level of nesting is a place for the model to lose track. Three shallow objects extracted in three calls beat one deeply nested tree.
• Enums beat free strings. If a field has five legitimate values, list them. You remove an entire class of normalisation code and the constrained decoder enforces it for you.
• Field names are instructions. due_date_iso8601 produces better output than date with the format explained in a separate prompt. The names are in the context; use them.
• Use description on fields. Most providers pass schema descriptions to the model. One clause per ambiguous field is cheaper and more reliable than a paragraph of prose instructions above.
• Set additionalProperties: false. Several providers require it for strict mode anyway, and it stops the model helpfully inventing an extra key your parser silently ignores.
• Prefer arrays of flat objects to maps. Dynamic keys — an object whose property names are unknown in advance — are poorly supported by strict modes and awkward to type on the receiving end.

Providers also restrict which JSON Schema keywords they support in strict mode: oneOf, pattern constraints, numeric bounds and recursion are commonly limited or ignored. Assume the schema constrains structure only, and enforce your business rules yourself after parsing.

Validate anyway

Even with a strict mode, validate every response before it touches your database. Strict mode may be disabled by a config change, a fallback path may hit a model that does not support it, a network retry may return an error object, and the response may be truncated by a token limit mid-object — which produces invalid JSON no schema mode prevents.

When a payload does fail, the useful question is which field and why. The JSON Schema validator takes the document and the schema and reports one line per problem, with the JSON Pointer path, the keyword that failed and the value actually found. That is exactly the form you want for two purposes: fixing your schema during development, and feeding back into a retry — a failed validation report pasted into a follow-up message is a remarkably effective repair prompt, since the model can see precisely what it got wrong.

Cap the retries at one or two. A model that fails a schema twice is usually being asked for something the input does not contain, and the third attempt will invent it.

From a sample payload to code

Most schemas start life as a sample response somebody pasted into a chat. Turning that into typed, validated code by hand is tedious and error-prone, so do not.

Paste the sample into JSON to Zod schema and you get a Zod schema plus the inferred TypeScript type. It infers per field rather than blanket-typing everything as a string: values that look like emails, URLs, UUIDs or ISO dates get .email(), .url(), .uuid() and .datetime() instead of a bare z.string(), arrays carry their element type, and a field whose type varies across occurrences becomes a union rather than silently picking the first one it saw. That last behaviour is worth feeding deliberately: paste three or four real responses concatenated, not one, so optional and polymorphic fields show up as unions instead of being locked to whatever the first sample happened to contain.

If you only need types and not runtime validation — because the payload comes from a source you already trust — JSON to TypeScript produces the interfaces directly, with nested objects broken out as named sub-interfaces rather than inlined anonymous blobs.

Keep the direction of truth straight: one schema definition, with everything else generated from it. A Zod schema and a JSON Schema maintained separately will disagree within a month, and the disagreement will surface as a production parse error rather than a test failure.

Reading what came back

Model responses arrive minified on one line, which makes eyeballing them miserable. The JSON formatter indents and sorts them into something readable, and its validation catches the truncation case immediately — a payload that ends mid-string is the signature of a hit token limit, and the fix is a higher output cap or a smaller schema, not a prompt change.

When you are comparing what two models produced from the same input, or checking whether a prompt change altered the output in ways you did not intend, diff the formatted payloads rather than reading both. Structural differences that are invisible in a wall of JSON — a field that quietly became a string of digits instead of a number — stand out immediately in a diff.

Versioning

Schemas change, and the model output changes with them. Put a version field in the payload from the first day, so stored records carry the shape they were written under. Retro-fitting that after you have a million rows of untagged JSON is the kind of migration nobody schedules time for.

All of these tools run locally in your browser, which matters more than usual for this workflow: the sample payloads you are turning into schemas are usually real API responses with real customer data in them, and they should not be uploaded to a formatting service to be prettified.