Structured Outputs and JSON Schema

Interview answer (say this first). A language model emits free text, so anything that consumes its output has to parse it, and parsing breaks on invalid JSON, wrong types, and invented values. JSON Schema is a standard description of the exact shape you want. Structured outputs make the model conform to that schema, usually by constraining decoding so invalid tokens are impossible, and you still validate the result on your side.

Why this exists

An LLM produces a sequence of tokens. Nothing in that process knows about JSON. When you ask for JSON, you are asking the model to imitate the appearance of JSON, and appearance is not a guarantee.

Here is what actually comes back in practice:

Sure! Here is the JSON you asked for:

{ "action": "search", "query": "lora" }

Let me know if you need anything else.

The JSON is correct, but it is wrapped in prose and a markdown fence. A naive json.loads() fails. Now consider an agent that must decide the next tool to call. A parse failure stops the whole chain, and the failure may be rare enough to look like a flaky bug rather than a design flaw.

Even when the JSON parses, other failures remain:

• Wrong type. "confidence": "high" where you expected a number.
• Missing field. The model omits steps because the prompt was ambiguous.
• Invented value. "action": "delete_everything", which is not one of your actions.
• Wrong nesting. A flat object where you needed a list of objects.
• Truncation. The output hits the token limit mid-object, so the JSON is unfinished.
• Refusal. The model declines and returns prose entirely.

String prompts and retry loops reduce these failures but never remove them, because they rely on the model choosing the right format. Structured outputs fix the root cause: instead of asking for a shape and checking afterward, they constrain generation so that only valid output is possible.

Note:

The one-sentence purpose. JSON Schema says exactly what shape is valid, and structured output makes the model incapable of producing anything else.

Start from zero

Two distinctions are worth pinning down now:

• JSON mode vs schema mode. JSON mode promises syntactically valid JSON but not any particular fields. Schema mode promises a specific shape. Schema mode is what you want for machine consumption.
• Generation-time vs post-hoc. Constrained decoding makes bad output impossible during generation. Validation checks the finished text afterward. Good systems do both, because the schema can be right while the content is wrong.

The core idea

Imagine a road with a fence on both sides. Without the fence, a driver can drift anywhere; you can only check afterward where they ended up. With the fence, the car physically cannot leave the road. That is constrained decoding: the invalid paths are removed while driving, not punished afterward.

A JSON Schema is the fence plan. The provider compiles it into a grammar: a state machine describing which token sequences are valid. At each generation step the engine looks at the current state, computes the set of tokens that keep the output valid, and forbids every other token. Then it samples among the allowed ones as usual.

flowchart LR
    A["Pydantic model<br/>or JSON Schema"] --> B["compile to grammar<br/>state machine"]
    B --> C["at each step,<br/>mask invalid tokens"]
    C --> D["sample next token<br/>from allowed set"]
    D --> E{"grammar<br/>complete?"}
    E -->|no| C
    E -->|yes| F["valid JSON string"]
    F --> G["validate again<br/>with Pydantic"]
    G -->|invalid| H["retry with error<br/>in the prompt"]
    G -->|valid| I["typed object"]

The important consequence: with true constrained decoding, the output is syntactically guaranteed. It is not guaranteed to be semantically correct. A schema can force action to be one of three strings, but it cannot make the model choose the right one. Structured output removes parsing failures, not reasoning failures.

How it works

1. You define the schema. Usually you write a Pydantic model and call model_json_schema(). This keeps the schema and the validator from diverging.
2. The provider compiles the schema into a grammar. Objects, arrays, enums, and required fields become states and transitions. Some schemas compile cleanly; very complex ones can be rejected or slow decoding.
3. The model computes logits for the next token. Nothing has changed yet.
4. The engine masks invalid tokens. Given the grammar state, it sets the logits of all tokens that would break the schema to negative infinity. This is the heart of constrained decoding.
5. Sampling happens among the allowed tokens only. Temperature, top-k, and top-p still apply, but only within the legal set. This is why structured output can still be varied.
6. The loop repeats until the grammar reaches an accepting state. When a complete valid value has been emitted, the engine permits the end-of-sequence token.
7. You parse and validate. With Pydantic, model_validate_json() gives a typed object and raises ValidationError with the exact field path if anything is off.
8. On failure, you retry with feedback. Include the validation error in the next prompt. This handles the cases grammar cannot: refusals, truncation, and content that is wrong even though it is well-formed.

There are two quieter failure paths to design for:

• Refusal. The model may decline before the grammar completes. Refusals are not schema violations, so you must detect them separately.
• Truncation. If max_tokens is too small, generation stops mid-object. The partial string is invalid, and no amount of grammar helps. Budget tokens for the largest valid document.

The syntax you will use

Write a schema by hand. This is the smallest useful schema: one enum field.

{
  "type": "object",
  "properties": {
    "action": { "type": "string", "enum": ["search", "summarize", "finish"] }
  },
  "required": ["action"],
  "additionalProperties": false
}

Generate the schema from Pydantic. One source of truth for the schema and the validator.

from typing import Literal
from pydantic import BaseModel, Field

class Step(BaseModel):
    action: Literal["search", "summarize", "finish"]
    query: str | None = None
    confidence: float = Field(ge=0.0, le=1.0, default=1.0)

class Plan(BaseModel):
    steps: list[Step] = Field(min_length=1, max_length=5)

schema = Plan.model_json_schema()

Pydantic emits the nested Step under $defs and refers to it with $ref:

"steps": {
  "items": { "$ref": "#/$defs/Step" },
  "maxItems": 5,
  "minItems": 1,
  "type": "array"
}

Validate any JSON against the schema with jsonschema. This is how you check a provider’s output against the same contract.

import jsonschema

jsonschema.validate({"steps": [{"action": "search"}]}, schema)   # passes

Parse into typed objects with Pydantic. This is the call that belongs in production code.

from pydantic import ValidationError

try:
    plan = Plan.model_validate_json(raw_text)
except ValidationError as e:
    for err in e.errors():
        print(err["loc"], err["type"])   # ('steps', 0, 'action') literal_error

Forbid extra fields. By default Pydantic allows unknown keys and its schema omits additionalProperties. Set extra="forbid" to tighten both.

from pydantic import ConfigDict

class Strict(BaseModel):
    model_config = ConfigDict(extra="forbid")
    a: int

# schema now contains "additionalProperties": false

OpenAI: request a schema. The strict flag asks the provider to enforce it.

from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="<a-model-with-structured-outputs>",
    messages=[{"role": "user", "content": "Plan how to answer: what is LoRA?"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "plan", "strict": True, "schema": schema},
    },
)
raw = response.choices[0].message.content

OpenAI: parse directly into Pydantic. The SDK builds the schema for you and returns a parsed object.

parsed = client.beta.chat.completions.parse(
    model="<a-model-with-structured-outputs>",
    messages=[{"role": "user", "content": "Plan how to answer: what is LoRA?"}],
    response_format=Plan,
)
plan = parsed.choices[0].message.parsed

Anthropic: force a single tool call. Tool input schemas are JSON Schema, so tool_choice is a structured-output mechanism.

resp = client.messages.create(
    model="<a-claude-model>",
    max_tokens=1024,
    tools=[{"name": "emit_plan", "description": "Return the plan.",
            "input_schema": schema}],
    tool_choice={"type": "tool", "name": "emit_plan"},
    messages=[{"role": "user", "content": "Plan how to answer: what is LoRA?"}],
)
payload = resp.content[0].input     # already a dict, validated by the shape

Gemini: pass the schema in the config.

from google import genai
from google.genai import types

gemini = genai.Client()
resp = gemini.models.generate_content(
    model="<a-gemini-model>",
    contents="Plan how to answer: what is LoRA?",
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=Plan,
    ),
)

Libraries that constrain locally. outlines, xgrammar, and llguidance compile schemas to token masks for open models, so you get the same guarantee without a hosted API. instructor wraps providers and adds validation plus retries.

# Illustrative: local constrained decoding with an open model.
import outlines

# outlines 1.x: build a Generator from a model and an output type.
generator = outlines.Generator(outlines.from_transformers(model, tokenizer), Plan)
plan = generator("Plan how to answer: what is LoRA?")

A minimal retry loop. Feed the validation error back rather than repeating the same prompt.

feedback = None
for attempt in range(3):
    raw = call_model(prompt, feedback)
    try:
        plan = Plan.model_validate_json(raw)
        break
    except ValidationError as e:
        feedback = str(e.errors())

Examples: simple to real

Example 1 — a Pydantic model produces a real schema. Here is the actual output for the Plan model, shortened:

{
  "$defs": {
    "Step": {
      "properties": {
        "action": { "enum": ["search", "summarize", "finish"], "type": "string" },
        "query": { "anyOf": [{"type": "string"}, {"type": "null"}], "default": null },
        "confidence": { "default": 1.0, "maximum": 1.0, "minimum": 0.0, "type": "number" }
      },
      "required": ["action"],
      "type": "object"
    }
  },
  "properties": { "steps": { "items": {"$ref": "#/$defs/Step"}, "minItems": 1, "maxItems": 5, "type": "array" } },
  "required": ["steps"],
  "type": "object"
}

Notice three things. Literal became an enum. str | None became an anyOf with null, which is how JSON Schema expresses nullability. Nested models moved into $defs and are referenced with $ref.

Example 2 — the same schema rejects bad data. jsonschema.validate gives a precise reason for each failure.

good instance: VALID
invented action: INVALID -> 'deploy' is not one of ['search', 'summarize', 'finish']
missing steps: INVALID -> 'steps' is a required property
empty steps: INVALID -> [] should be non-empty
confidence too high: INVALID -> 2.0 is greater than the maximum of 1.0

This is the value of a schema: four different classes of bug, each caught with a specific message instead of a vague parse error.

Example 3 — Pydantic reports the exact path. When validation fails, loc tells you which nested field was wrong.

Plan.model_validate_json('{"steps": [{"action": "deploy"}]}')
# ValidationError: loc=('steps', 0, 'action'), type='literal_error'

For nested lists the path is ('steps', 0, 'action'): field, index, field. Log the path; it is far more useful than the message alone.

Example 4 — extra fields are allowed by default. This surprise catches many teams. Pydantic’s default schema has no additionalProperties key, so unknown fields pass JSON Schema validation, though Pydantic drops them unless configured otherwise.

loose schema additionalProperties: absent
strict schema additionalProperties: False
strict extra field -> extra_forbidden

Turn on extra="forbid" when a stray field indicates an upstream bug you want surfaced.

Example 5 — the provider request shape is just a dict. Building it is static; the schema travels inside.

response_format = {
    "type": "json_schema",
    "json_schema": {"name": "plan", "strict": True, "schema": schema},
}
# response_format["type"] -> "json_schema", strict -> True

Example 6 — a retry loop recovers from a bad first attempt. The model’s first answer uses an invented action; the second, with the error in the prompt, is valid.

retry attempts: 2
recovered plan: search lora

Retries fix schema violations, but each retry costs tokens and latency. If a model fails often, fix the schema or the prompt rather than raising the retry limit.

In production

• Design schemas the model can satisfy. Deeply nested schemas and huge enums increase decoding cost and failure rates. Flatten where possible and keep enums short.
• Cap max_tokens above the largest valid output. Truncation produces invalid JSON regardless of the schema, and it is a common cause of sudden parse failures in production.
• Handle refusals separately from schema errors. A refusal is valid text that is not JSON. Detect it, log it, and decide whether to retry, fall back, or escalate.
• Validate on your side even with strict mode. Providers vary in what they enforce, and schemas can be relaxed by the provider. Your validator is the guarantee you control.
• Never trust valid output as correct output. A schema can prove the shape and nothing about the content. Validate business rules afterward, especially before side effects.
• Keep one source of truth. Derive the schema from the same Pydantic model you validate with. Hand-written duplicates drift and produce confusing failures.
• Watch schema keyword support. Providers support different subsets, and some reject $ref, anyOf, or advanced keywords. Test your schema against the provider early.
• Set additionalProperties deliberately. For your own pipelines, forbid it to catch upstream drift. For third-party data that may add fields, allowing it is more forward-compatible.
• Log the raw output on failure. Validation errors are much easier to debug when you have the exact bytes the model produced, including any refusal or truncation.
• Bound the retry loop. Two or three attempts, then a fallback path. Unbounded retries turn a bad schema into a latency incident.
• Mind streaming. Partial JSON is not parseable. If you stream, either parse incrementally with a tolerant parser or wait for the complete object before validating.
• Version your schemas. When the contract changes, old callers break. Treat the schema like an API: version it and migrate.

Interview questions

1. Why is free-text output a problem for software?

Answer. Software needs predictable structure, and free text has none. Even when a model is asked for JSON it may wrap it in prose, use the wrong type, omit a field, invent an enum value, or get truncated. Every consumer then needs fragile parsing and a retry path, and failures surface far from their cause. Structured outputs replace hopeful parsing with an enforced contract.

Follow-up: “Does a schema fix all of that?” No. It fixes syntax and shape. Refusals, truncation, and semantically wrong content still need handling.

Trap. Assuming a good prompt guarantees format. Prompts are probabilistic; schemas are constraints.

2. What is JSON Schema?

Answer. It is a standard JSON document that describes which JSON documents are valid. Keywords declare types, required fields, allowed values, numeric bounds, array lengths, and nesting. Tools, editors, and LLM providers all understand it, which is why it is the shared language for structured output.

Follow-up: “How does Pydantic relate to it?” Pydantic reads your type annotations and generates a JSON Schema with model_json_schema(). The same model then validates the model’s output, so the schema and the check cannot drift apart.

Trap. Thinking JSON Schema validates data types at runtime. It describes validity; you still need a validator like jsonschema or Pydantic to enforce it.

3. What is constrained decoding, and how does it work?

Answer. It restricts generation at each token step so the result must match a grammar compiled from the schema. At every step, tokens that would violate the grammar are masked to negative infinity, so sampling can only choose a legal token. The output is then syntactically guaranteed.

Follow-up: “Can the model still be wrong?” Yes. The grammar controls form, not meaning. The model can pick a valid but incorrect value. Constrained decoding eliminates parse errors, not reasoning errors.

Trap. Saying constrained decoding makes the model “more accurate”. It makes the output well-formed; accuracy is a separate problem.

4. What is the difference between JSON mode and JSON Schema mode?

Answer. JSON mode guarantees syntactically valid JSON but does not control which fields appear, so you can still get the wrong shape. JSON Schema mode enforces a specific structure, including required fields, types, and enums. Schema mode is what machine consumers should use.

Follow-up: “When would JSON mode be enough?” When the content is genuinely freeform, such as an arbitrary settings object a human will inspect. For pipelines that branch on field values, use schema mode.

Trap. Treating {"type": "json_object"} as a schema. It is only a promise about syntax.

5. How do you handle a validation failure?

Answer. Retry with the validation error included in the prompt, so the model can see exactly which field was wrong, and cap the number of attempts. If it keeps failing, fall back to a safer path, such as a simpler schema, a different model, or a human review. Never pass unvalidated output into a tool with side effects.

Follow-up: “Why include the error rather than just retrying?” Repeating the identical prompt tends to reproduce the same mistake. The error narrows the correction and raises the success rate.

Trap. Retrying forever. Each attempt costs tokens and latency, and a persistent failure means the schema or prompt is wrong.

6. What are the common failure modes of structured output?

Answer. Refusals, where the model returns prose; truncation, where max_tokens cuts the JSON short; schemas the provider cannot compile; invented values inside a valid shape; and unknown extra fields when additionalProperties is not set. Each needs its own handling.

Follow-up: “Which is hardest to detect?” Semantically wrong but well-formed output. The schema passes and nothing alerts you, so correctness requires your own business-logic checks or an evaluation harness.

Trap. Assuming a successful parse means success. A parsed object can still be nonsense.

7. How does Pydantic fit into this workflow?

Answer. Pydantic plays both roles. model_json_schema() produces the schema you send to the provider, and model_validate_json() checks and parses the response into typed objects. Using one model for both means the contract and the validator are always in sync, and failures come back with a precise field path.

Follow-up: “What extra protection does Pydantic add over the schema?” It can enforce rules the provider’s schema support may not, and it catches cases where the provider’s enforcement is weaker than promised. It is your line of defence.

Trap. Reusing the response model as an input model. Keep the schemas for what you send and what you return separate, as with any API.

8. How do structured outputs and tool calling relate?

Answer. They share the same machinery. A tool is described by a JSON Schema of its arguments, and a tool call is a structured object containing the tool name and arguments. Forcing a single tool call is a common way to get structured output. The difference is purpose: structured output returns data to your program, while tool calling asks the model to trigger an action.

Follow-up: “How does that affect safety?” Because a tool call can have side effects, validation must happen before execution, and risky tools should require confirmation or a sandbox. Valid structure is not permission to act.

Trap. Confusing a tool schema with a response schema. They are both JSON Schema, but one describes an action’s inputs and the other describes data you want back.

Remember this

• Free text is not a contract. Ask for a schema, or you own the parsing bugs.
• JSON Schema defines the shape; Pydantic generates it and validates the result. One source of truth.
• Constrained decoding masks invalid tokens, so syntax is guaranteed while correctness is not.
• Valid is not correct. Schema checks shape; your business rules check meaning.
• Handle refusals, truncation, and bounded retries explicitly, because grammar cannot cover them.