LLM Structured Output in Production: JSON Mode, Function Calling, and Schema-Validated Response Patterns

LLMs are genuinely useful for extracting, classifying, and transforming data. The problem is that their default output format is natural language, and your downstream systems want typed objects, not prose. If you have ever had a pipeline fail at 2am because an LLM returned "Sure! Here is your JSON: ..." instead of a bare JSON object, you already understand the problem.

This article covers how to reliably get structured data out of LLMs in production: the mechanisms providers offer, how to validate what comes back, how to recover from malformed output, and how to think about the tradeoffs.

Why Unstructured Output Breaks Systems

The failure modes are predictable once you have seen them a few times:

• The model wraps JSON in a markdown code fence (```json ... ```), so JSON.parse throws
• The model adds a conversational preamble or postamble around the JSON
• Field names drift from the schema (camelCase vs snake_case, or the model invents synonyms)
• Optional fields come back as the string "null" instead of actual null
• Numbers come back as strings ("42" instead of 42)
• The model truncates output mid-JSON when hitting context limits

Every one of these is a real incident waiting to happen. The fix is not better prompting alone. Prompting helps at the margin, but production systems need structural guarantees.

Provider JSON Modes

OpenAI

OpenAI offers two mechanisms. The older response_format: { type: "json_object" } tells the model to return valid JSON, but does not constrain the shape. You still need to validate the schema yourself.

The newer Structured Outputs feature (available on gpt-4o and later models) accepts a JSON Schema and guarantees the response matches it exactly. The model uses constrained decoding at the token level, so invalid structure is impossible by construction.

import OpenAI from "openai";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";

const client = new OpenAI();

const ExtractionSchema = z.object({
  company: z.string(),
  sentiment: z.enum(["positive", "negative", "neutral"]),
  topics: z.array(z.string()),
  confidence: z.number().min(0).max(1),
});

type Extraction = z.infer<typeof ExtractionSchema>;

async function extractFromReview(review: string): Promise<Extraction> {
  const response = await client.chat.completions.create({
    model: "gpt-4o-2024-08-06",
    messages: [
      {
        role: "system",
        content: "Extract structured information from the user review.",
      },
      { role: "user", content: review },
    ],
    response_format: {
      type: "json_schema",
      json_schema: {
        name: "extraction",
        strict: true,
        schema: zodToJsonSchema(ExtractionSchema),
      },
    },
  });

  const raw = response.choices[0].message.content;
  if (!raw) throw new Error("Empty response from model");

  // With strict: true, this parse will not throw for structural reasons,
  // but Zod validation is still useful for type narrowing and runtime safety.
  return ExtractionSchema.parse(JSON.parse(raw));
}

The strict: true flag is important. Without it, OpenAI falls back to best-effort JSON generation, which reintroduces the failure modes from above.

Anthropic

Anthropic’s Claude does not offer constrained decoding as of early 2026, but tool use (their term for function calling) is the recommended pattern for structured output. You define a tool with an input schema, and the model returns a tool_use content block rather than a text response.

import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";

const client = new Anthropic();

const extractionTool = {
  name: "extract_review_data",
  description: "Extract structured data from a customer review",
  input_schema: {
    type: "object" as const,
    properties: {
      company: { type: "string" },
      sentiment: { type: "string", enum: ["positive", "negative", "neutral"] },
      topics: { type: "array", items: { type: "string" } },
      confidence: { type: "number", minimum: 0, maximum: 1 },
    },
    required: ["company", "sentiment", "topics", "confidence"],
  },
};

const ExtractionSchema = z.object({
  company: z.string(),
  sentiment: z.enum(["positive", "negative", "neutral"]),
  topics: z.array(z.string()),
  confidence: z.number().min(0).max(1),
});

async function extractFromReview(review: string) {
  const response = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 1024,
    tools: [extractionTool],
    tool_choice: { type: "tool", name: "extract_review_data" },
    messages: [{ role: "user", content: review }],
  });

  const toolUseBlock = response.content.find((b) => b.type === "tool_use");
  if (!toolUseBlock || toolUseBlock.type !== "tool_use") {
    throw new Error("Model did not invoke the expected tool");
  }

  // tool_use.input is already a parsed object, not a JSON string
  return ExtractionSchema.parse(toolUseBlock.input);
}

The key difference from OpenAI’s approach: tool_use.input is already a parsed JavaScript object. You never touch JSON.parse manually. This eliminates the code-fence and preamble problem entirely.

Forcing tool use via tool_choice: { type: "tool", name: "..." } ensures the model always invokes your tool rather than potentially responding in text.

Open-Source Models

For self-hosted or open-source models (Llama, Mistral, Qwen), Outlines and llama.cpp’s grammar sampling provide constrained generation. If you are using vLLM, it supports guided decoding with JSON Schema natively.

// vLLM OpenAI-compatible API with guided decoding
const response = await fetch("http://localhost:8000/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "meta-llama/Llama-3.1-8B-Instruct",
    messages: [{ role: "user", content: review }],
    guided_json: zodToJsonSchema(ExtractionSchema),
  }),
});

vLLM’s guided_json parameter accepts a JSON Schema and applies constrained decoding, giving you the same guarantees as OpenAI’s Structured Outputs for locally hosted models.

Schema Validation with Zod and TypeBox

Regardless of whether you use constrained generation, you should always run schema validation on what comes back. Constrained generation guarantees structure, not semantics. A confidence score of 1.5 is structurally valid JSON but semantically wrong.

Zod is the most common choice in TypeScript because it integrates well with the rest of the ecosystem and produces readable error messages:

import { z } from "zod";

const InvoiceSchema = z.object({
  invoiceNumber: z.string().regex(/^INV-\d{6}$/),
  amount: z.number().positive(),
  currency: z.enum(["USD", "EUR", "GBP"]),
  lineItems: z
    .array(
      z.object({
        description: z.string().min(1),
        quantity: z.number().int().positive(),
        unitPrice: z.number().positive(),
      })
    )
    .min(1),
  dueDate: z.string().datetime(),
});

function parseInvoiceResponse(raw: string) {
  const parsed = JSON.parse(raw); // can still throw for malformed JSON
  const result = InvoiceSchema.safeParse(parsed);

  if (!result.success) {
    // result.error.issues gives you structured error info per field
    throw new ValidationError("Invoice extraction failed", result.error.issues);
  }

  return result.data; // fully typed as z.infer<typeof InvoiceSchema>
}

TypeBox is a faster alternative that generates JSON Schema directly from type definitions, which is useful when you need to pass the schema back to the model:

import { Type, Static } from "@sinclair/typebox";
import { Value } from "@sinclair/typebox/value";

const InvoiceSchema = Type.Object({
  invoiceNumber: Type.String({ pattern: "^INV-\\d{6}$" }),
  amount: Type.Number({ exclusiveMinimum: 0 }),
  currency: Type.Union([
    Type.Literal("USD"),
    Type.Literal("EUR"),
    Type.Literal("GBP"),
  ]),
});

type Invoice = Static<typeof InvoiceSchema>;

// TypeBox schema IS a JSON Schema object, pass it directly to the API
const schema = InvoiceSchema; // no conversion needed

function validate(data: unknown): Invoice {
  if (!Value.Check(InvoiceSchema, data)) {
    const errors = [...Value.Errors(InvoiceSchema, data)];
    throw new Error(errors.map((e) => `${e.path}: ${e.message}`).join(", "));
  }
  return data;
}

Retry Strategies for Malformed Output

Even with constrained generation, you need a retry strategy. Constrained generation can produce structurally valid but semantically nonsensical output. Post-processing validation catches this. When it fails, you have three options:

Option 1: Blind retry. Call the model again with the same prompt. Works for transient issues but wastes tokens if the schema or prompt is fundamentally incompatible.

Option 2: Feedback retry. Include the validation errors in the retry prompt. This works well when the model understands what it got wrong.

Option 3: Repair attempt. Ask the model to fix its own output. Useful for partially valid responses.

import { ZodError, ZodSchema } from "zod";

interface RetryOptions {
  maxAttempts: number;
  onRetry?: (attempt: number, error: ZodError) => void;
}

async function withStructuredRetry<T>(
  callModel: (context?: string) => Promise<string>,
  schema: ZodSchema<T>,
  options: RetryOptions = { maxAttempts: 3 }
): Promise<T> {
  let lastError: ZodError | null = null;

  for (let attempt = 1; attempt <= options.maxAttempts; attempt++) {
    let raw: string;

    if (attempt === 1) {
      raw = await callModel();
    } else {
      // Pass validation errors back to the model
      const errorContext = lastError
        ? `Previous attempt failed validation. Errors: ${JSON.stringify(
            lastError.issues.map((i) => ({
              path: i.path.join("."),
              message: i.message,
            }))
          )}. Please fix these issues and return valid JSON.`
        : undefined;
      raw = await callModel(errorContext);
    }

    try {
      const parsed = JSON.parse(raw);
      const result = schema.safeParse(parsed);

      if (result.success) return result.data;

      lastError = result.error;
      options.onRetry?.(attempt, result.error);
    } catch (e) {
      // JSON.parse failure, try again
    }
  }

  throw new Error(
    `Structured output failed after ${options.maxAttempts} attempts`
  );
}

Cap retries at 2-3. More than that usually means the prompt or schema has a fundamental mismatch. Log every retry with the validation errors, they are a useful signal for debugging prompt quality.

Streaming Structured Responses

Streaming is tricky with structured output because you cannot validate a partial JSON object. The practical approaches:

Approach 1: Buffer until complete, then validate. Simplest. Works for most use cases. The user sees no output until the full response arrives.

Approach 2: Stream to a UI wrapper, validate on complete. Stream tokens to the frontend for a typing effect, but hold back business logic until the full object is validated.

Approach 3: Partial object streaming. Some providers stream structured output as partial objects you can read field by field. OpenAI’s streaming with response_format: json_schema streams tokens that can be assembled incrementally.

import OpenAI from "openai";

const client = new OpenAI();

async function streamExtraction(text: string): Promise<void> {
  const stream = await client.chat.completions.create({
    model: "gpt-4o-2024-08-06",
    stream: true,
    messages: [{ role: "user", content: text }],
    response_format: {
      type: "json_schema",
      json_schema: {
        name: "extraction",
        strict: true,
        schema: {
          type: "object",
          properties: {
            summary: { type: "string" },
            tags: { type: "array", items: { type: "string" } },
          },
          required: ["summary", "tags"],
        },
      },
    },
  });

  let buffer = "";

  for await (const chunk of stream) {
    const delta = chunk.choices[0]?.delta?.content ?? "";
    buffer += delta;

    // Stream partial content to UI while buffering for validation
    process.stdout.write(delta);
  }

  // Validate the complete response
  const result = JSON.parse(buffer);
  console.log("\nValidated result:", result);
}

For production pipelines where latency matters more than streaming UX, buffer and validate. Streaming structured output adds complexity that rarely pays off unless you have a real user-facing reason for it.

Approach Comparison

Approach	Schema Guarantee	Latency Impact	Provider Support	Complexity
Prompt-only	None	None	All	Low
JSON mode (json_object)	Valid JSON only	Minimal	OpenAI, some OSS	Low
Structured Outputs (strict)	Full schema	Low	OpenAI gpt-4o+	Medium
Function/tool calling	Full schema	Low	OpenAI, Anthropic	Medium
Constrained decoding (vLLM)	Full schema	Low	Self-hosted	High
Post-processing validation	Semantic only	Validation overhead	All	Low
Retry with feedback	Improves over attempts	High (retries)	All	Medium

A few observations from using these in production:

• Prompt-only works fine for simple, high-tolerance use cases. If you are extracting a yes/no classification and the model returns "Yes" vs true, you can handle that in a line of code.
• Structured Outputs (OpenAI) and tool use (Anthropic) are the right defaults for anything that goes into a database or feeds downstream logic.
• Constrained decoding on self-hosted models is worth the setup cost if you are running high volume and cannot afford retry token overhead.
• Always layer schema validation on top, even with constrained generation. You are validating semantics, not just structure.

Production Considerations

Log raw responses before parsing. When a parse fails, you want the raw model output in your logs. Do not let the exception swallow it.

Track schema validation failure rates. A sudden spike in validation failures usually means a model update changed output format, or a prompt change introduced ambiguity. Treat it like a service degradation.

Version your schemas alongside your prompts. A schema change that drops a required field or adds a new enum value is a breaking change. Coordinate schema and prompt versions in deployment.

Handle refusal content blocks. OpenAI’s Structured Outputs can return a refusal block instead of a tool call when the model declines to answer. Anthropic’s tool use can return a text block when tool_choice is not forced. Check for these explicitly rather than assuming the expected block type will always be present.

// OpenAI refusal handling
const message = response.choices[0].message;
if (message.refusal) {
  throw new ModelRefusalError(message.refusal);
}

// Anthropic: check stop_reason
if (response.stop_reason !== "tool_use") {
  const textBlock = response.content.find((b) => b.type === "text");
  throw new UnexpectedResponseError(textBlock?.text ?? "Unknown response type");
}

Use TypeScript generics to keep extraction functions fully typed. A generic extract<T>(prompt, schema) function that returns T is worth writing once and reusing everywhere.

Constrained Generation vs. Post-Processing Validation

These are not alternatives, they are complementary. Constrained generation prevents structural failures. Post-processing validation catches semantic failures. The question is where to invest your complexity budget.

If you control the model (self-hosted), constrained decoding is worth it. You get hard guarantees at inference time with minimal latency cost.

If you are using a cloud provider, use their structured output feature when available. For providers without it (or when using older models), aggressive prompting plus validation plus retry is a reasonable substitute.

The one pattern to avoid: relying solely on prompting with regex extraction as a fallback. It works until the model changes its output style slightly, and then it fails silently in ways that are annoying to debug.

Pick the strictest guarantee your infrastructure supports, then add schema validation on top. The combination handles the cases neither approach covers alone.