Structured Output Prompting: How to Force JSON and Exact Formats from Any AI

You've written this prompt before: "Respond ONLY with valid JSON. No markdown. No commentary. Use exactly these keys..." The model returns JSON wrapped in a code fence. Or valid JSON with an extra notes field you didn't ask for. Or it stops mid-object because you hit max_tokens.

Prompting for format was necessary in 2023. In 2026 it's the fallback, not the strategy. Every major API can constrain generation at the token level so invalid structure literally cannot be emitted. If you're still regex-parsing model output in production, you're doing unpaid QA on the model.

Three levels of format control

Level 1 — Prompt-only: describe the JSON in prose. Works 80–90% on easy schemas. Fails on enums, nested objects, and any model having a creative Tuesday.

Level 2 — JSON mode: API flag that forces syntactically valid JSON. No schema guarantee. The model can still pick wrong keys, omit required fields, or nest things differently than you imagined.

Level 3 — Structured output (schema-constrained decoding): you pass JSON Schema (or Pydantic/Zod). The runtime masks invalid tokens during generation. Output matches schema by construction, not by hope.

Production default in 2026: Level 3 where supported. Level 2 as legacy fallback. Level 1 for quick UI experiments only.

How constrained decoding actually works (short version)

Autoregressive models pick one token at a time. Structured output compiles your schema into a state machine. At each step, tokens that would break the schema get probability zero. The model can't open a string where a number is required. Can't close an object early if required fields remain.

This is not post-hoc validation with retries (though retries exist as backup on some providers). It's prevention at generation time. Latency overhead is small after schema compilation — often 50–200ms first call, cached after.

OpenAI: Structured Outputs vs JSON mode

OpenAI distinguishes them clearly:

• JSON mode (json_object): valid JSON syntax only. No schema adherence.
• Structured Outputs (json_schema with strict: true): must match your schema.

Use Structured Outputs on supported models (gpt-4o and later, current GPT-5.x family). Pass schema via response_format or the Responses API text.format block.

OpenAI strict mode quirks:

• All object properties typically need to be in required
• additionalProperties: false at every object level in strict mode — forget it and the API rejects the request
• Subset of JSON Schema features — no $ref in strict mode, limited anyOf

Python SDK: .parse() with Pydantic models. TypeScript: Zod integration via helpers or manual schema export.

Delete the three-paragraph "ONLY JSON" lecture from your prompt. Keep field descriptions that clarify semantics the schema can't express.

Anthropic (Claude): native structured output and tool use

Claude shipped native structured output via output_config.format with JSON Schema (GA early 2026). Constrained decoding applies similarly to OpenAI.

Older pattern still works: define a tool with your schema, force tool_choice, parse tool_use block. Many codebases still use this. Native structured output is cleaner for pure extraction.

Claude gotcha: the SDK may transform schemas — constraints like minLength, maxLength, numeric min/max sometimes move into field descriptions as text hints rather than hard decoder rules. The SDK validates after generation and may retry. Budget for retry latency.

Claude also excels at XML structured output in prompts when you're not on the API schema path — <result><field>...</field></result>. Useful for chat UI workflows. Production pipelines still prefer API enforcement.

Google Gemini: response_schema

Gemini uses response_mime_type: application/json plus optional response_json_schema (or Pydantic model in Python SDK).

Without schema: JSON mode — valid syntax, arbitrary shape.

With schema: constrained decoding. Gemini respects propertyOrdering in schema — useful if you want reasoning fields before answer fields (or the reverse, depending on task).

Gemini is often looser on additionalProperties and required enforcement than OpenAI strict mode. Treat provider guarantee as necessary but not sufficient — validate in app code.

What belongs in the schema vs the prompt

Schema handles: keys, types, enums, nesting, required fields, array items.

Prompt handles: semantics, grounding rules, what to put IN those fields, business logic the schema can't express.

Good split:

Prompt: "Extract invoice data from the text below. Use null for missing fields. Amounts in cents."

Schema: { vendor: string, date: string, total_cents: integer, line_items: [...] }

Bad split: repeating the entire schema in prose AND in JSON Schema. Redundant tokens, lost-in-the-middle risk, contradictions when you update one and not the other.

Prompt patterns that still help with structured output

Even with schema enforcement, prompt quality affects field content:

• Grounding: "Populate fields only from provided context. Use null if absent."
• Enum semantics: describe what severity: critical means vs high in the prompt; schema enforces the enum exists, not that the model picks right
• Reasoning placement: some teams put a reasoning string field first in schema (Gemini propertyOrdering) for audit; others keep output minimal and log chain separately
• Null vs omit: decide policy; JSON Schema required + nullable types vs optional keys — be explicit

Structured output guarantees shape. You still own correctness.

Validation after the API (non-negotiable)

Always parse through Pydantic (Python) or Zod (TypeScript) even when the provider guarantees schema compliance.

Why:

• Cross-provider portability breaks in subtle ways (extra keys, enum drift)
• Business rules exceed JSON Schema ("total must equal sum of line items")
• Semantic nonsense fits schema perfectly (confidence: 0.99 on a hallucinated fact)

Pipeline: API structured output → Pydantic/Zod parse → business validation → downstream.

Retries: on validation failure, retry with error message injected ("field X failed: expected integer cents, got float dollars"). Cap retries at 2.

Common failure modes in 2026

Using JSON mode thinking it's schema mode — green CI, wrong keys in prod.

Copying OpenAI strict schema to Claude without testing — silent extra keys or failed requests.

Max tokens too low — truncated JSON that validates only because the parser never runs (use structured output + adequate completion budget).

Huge schemas — 40-field nested objects slow compilation and confuse the model's content choices even if shape is valid.

Reasoning field after answer field when model decides before filling reasoning — order matters for content quality even when schema validates.

Regex extraction from markdown fences — fragile; delete when API supports structured output.

Non-JSON structured formats

JSON is default for APIs. Alternatives when appropriate:

• XML tags (Claude workflows, human-readable logs)
• YAML (config generation — validate carefully, YAML injection is real)
• CSV rows (tabular extraction with fixed column schema)
• Markdown tables (human-facing reports, not machine pipelines)

For machine consumption, JSON with schema enforcement wins. For human+model collaboration in chat, tagged XML sections often parse more reliably than JSON mode without schema.

Fallback router pattern (multi-provider)

If you route between OpenAI, Claude, and Gemini:

• Maintain a provider capability matrix (strict schema subset per vendor)
• Test the same schema against each provider in CI
• Document dialect differences (additionalProperties, enum handling, retry behavior)
• Never assume "JSON mode" means the same thing across providers — 2026 modes are schema-aware but not portable byte-for-byte

One schema definition, provider-specific adapters, unified validation layer.

When prompt-only format still makes sense

• Quick chat UI experiments without API access to structured output flags
• Models or hosts that don't support constrained decoding yet
• Free-form creative output where structure is advisory
• PromptMake /text plain-text generation for human paste into chat tools

PromptMake Pro offers JSON and XML output formats for developer workflows — useful when you're generating prompts to paste into API code, not replacing API-level schema enforcement on the inference call itself.

Example: extraction prompt + schema (conceptual)

Prompt:

Extract meeting action items from the transcript. Each item needs owner and due date if stated. Use null when unknown.

Schema (abbreviated):

{ "items": [{ "task": string, "owner": string | null, "due_date": string | null }] }

API: structured output enabled. No "ONLY JSON" paragraph.

Post-parse: reject if items empty and transcript contained action language (business rule).

Frequently asked questions

Is JSON mode enough for production?

No if you need reliable field names and types. JSON mode guarantees syntax. Structured output guarantees schema.

Can I force JSON from ChatGPT web UI?

Limited. Custom GPTs and instructions help but don't offer API-level constrained decoding. Copy-paste workflows stay prompt-heavy.

OpenAI strict vs Anthropic structured output — which is stricter?

OpenAI strict mode is rigid at request validation time. Claude may allow looser generation with post-hoc validation/retries. Test both.

Should reasoning models use structured output?

Yes for final answers. Keep reasoning in hidden thinking channel when available; expose only schema fields you need in visible output.

What about streaming structured output?

Supported on major providers with partial JSON parsing libraries. Useful for UX; validate complete object before acting.

Does few-shot help structured output?

Sometimes for field semantics, but schema enforcement reduces format need for examples. See our few-shot vs zero-shot guide.

Related articles

Prompt engineering best practices 2026 — Rule 3: structured output over JSON pleading.

Few-shot vs zero-shot — one-shot for format before reaching for API schema.

Positive framing — "respond with schema X" beats "don't include markdown."

Bottom line

Stop negotiating with the model about JSON. Pass a schema. Let constrained decoding do what prompts never did reliably.

Keep prompts for meaning. Keep schemas for shape. Keep Pydantic/Zod for truth. The "respond ONLY with valid JSON" era is legacy — act like it.