Structured LLM Output in Production: JSON Mode, Function Calling, and Constrained Decoding

Most teams discover the problem the same way: an LLM that behaves perfectly in testing starts returning malformed JSON in production. The response has a trailing comma, or the model wraps the JSON in a markdown code fence, or it includes a one-sentence explanation before the opening brace. Your JSON.parse call throws. Your error rate goes up. Your users see failures.

“Just tell the model to return valid JSON” is not a production strategy. At scale, LLMs produce malformed output often enough that you need a real architectural answer.

There are three main approaches: JSON mode, function calling (also called tool use), and constrained decoding. Each has different guarantees, different failure modes, and different operational costs. This article covers how each works, when each fails, and how to build a validation and fallback chain that holds up in production.

Why Raw Parsing Fails

Before covering the solutions, it is worth being precise about why naive parsing fails. The causes fall into a few categories.

Formatting noise: Models add explanations, code fences, apologies, or preamble before the JSON. A response that starts with “Here is the requested JSON:” followed by a code block fails a direct JSON.parse call even though the data is valid.

Structural errors: Trailing commas, unquoted keys, single quotes instead of double quotes, missing closing brackets. These are common in models that were not specifically fine-tuned for JSON output.

Schema drift: The JSON is syntactically valid but structurally wrong. The model returns a string where you expected a number, omits a required field, or invents fields that are not in your schema.

Truncation: Long responses get cut off mid-JSON when the generation hits the model’s token limit. You get structurally incomplete output with no error signal from the model itself.

Each failure mode needs a different fix. JSON mode helps with the first two. Function calling helps with all four. Constrained decoding is the most robust for the first two but has its own costs.

Approach 1: JSON Mode

JSON mode is a model configuration option that constrains the model’s output to syntactically valid JSON. It is available in most major model APIs and requires no schema definition up front.

import OpenAI from "openai";

const client = new OpenAI();

interface ExtractedEvent {
  name: string;
  date: string;
  location: string;
  attendees: number;
}

async function extractEvent(text: string): Promise<ExtractedEvent> {
  const response = await client.chat.completions.create({
    model: "gpt-4o",
    response_format: { type: "json_object" },
    messages: [
      {
        role: "system",
        content:
          "Extract event details from the user's text. Return a JSON object with fields: name (string), date (ISO 8601 string), location (string), attendees (number).",
      },
      {
        role: "user",
        content: text,
      },
    ],
  });

  const raw = response.choices[0].message.content;
  if (!raw) throw new Error("Empty response from model");

  // JSON.parse is safe here because json_object mode guarantees valid JSON
  const parsed = JSON.parse(raw) as ExtractedEvent;
  return parsed;
}

JSON mode guarantees syntactic validity. It does not guarantee schema compliance. The model might return { "event_name": "..." } instead of { "name": "..." }, omit the attendees field entirely, or return a top-level array when you expected an object.

This is the gap that catches teams after they switch to JSON mode and declare victory. The syntax errors go away. The schema violations remain.

When to use JSON mode: Simple extraction tasks where you control the prompt tightly and the schema is shallow. Fast to set up, no schema definition overhead. Add Zod validation on top and you have a workable setup for many use cases.

When it fails: Complex or deeply nested schemas where the model invents its own structure. Long inputs where the model starts drifting from the prompt instructions. Any case where field presence or type correctness is a hard requirement.

Approach 2: Function Calling and Tool Use

Function calling (the mechanism that underpins tool use in agent frameworks) lets you pass a JSON Schema to the model and ask it to return a structured call to a named function. The model’s output is constrained to fit the schema you provide.

This is currently the most practical approach for production structured output in most systems. It handles schema compliance better than JSON mode and works with the same APIs you already use for tool use.

import OpenAI from "openai";
import { z } from "zod";

const client = new OpenAI();

// Define your schema as both a Zod validator and a JSON Schema for the API
const EventSchema = z.object({
  name: z.string(),
  date: z.string().describe("ISO 8601 date string"),
  location: z.string(),
  attendees: z.number().int().positive(),
  is_virtual: z.boolean(),
});

type ExtractedEvent = z.infer<typeof EventSchema>;

async function extractEventWithFunctionCalling(
  text: string
): Promise<ExtractedEvent> {
  const response = await client.chat.completions.create({
    model: "gpt-4o",
    tools: [
      {
        type: "function",
        function: {
          name: "extract_event",
          description: "Extract structured event details from the provided text",
          parameters: {
            type: "object",
            properties: {
              name: { type: "string", description: "Name of the event" },
              date: {
                type: "string",
                description: "Event date in ISO 8601 format",
              },
              location: { type: "string", description: "Event location" },
              attendees: {
                type: "integer",
                description: "Expected number of attendees",
              },
              is_virtual: {
                type: "boolean",
                description: "Whether the event is virtual",
              },
            },
            required: ["name", "date", "location", "attendees", "is_virtual"],
          },
        },
      },
    ],
    tool_choice: { type: "function", function: { name: "extract_event" } },
    messages: [
      {
        role: "user",
        content: `Extract event details from this text: ${text}`,
      },
    ],
  });

  const toolCall = response.choices[0].message.tool_calls?.[0];
  if (!toolCall || toolCall.type !== "function") {
    throw new Error("Model did not return a tool call");
  }

  const rawArgs = JSON.parse(toolCall.function.arguments);

  // Validate against your Zod schema
  const result = EventSchema.safeParse(rawArgs);
  if (!result.success) {
    throw new Error(
      `Schema validation failed: ${result.error.issues.map((i) => i.message).join(", ")}`
    );
  }

  return result.data;
}

The key pattern here: define the schema once (as Zod), derive types from it, and maintain a parallel JSON Schema definition for the API call. Some libraries (like zod-to-json-schema) can automate this conversion, though you should review the output for complex schemas.

When to use function calling: This is the default choice for production structured output. It handles required fields, type constraints, and nested objects better than JSON mode. The tool_choice parameter forces the model to call a specific function, removing the ambiguity about what shape the response takes.

When it fails: Very large schemas with many nested levels can cause the model to miss fields or invent values. The model can still return semantically wrong data (a date string that does not parse as ISO 8601 even though it is a valid string). Streaming is more complex because you cannot validate until the full arguments are received.

Approach 3: Constrained Decoding

Constrained decoding works at the token generation level. A grammar or schema is compiled into a set of constraints that are applied during sampling: at each step, tokens that would violate the schema are assigned zero probability. The model can only generate output that is structurally valid by construction.

Libraries like llama.cpp (via its grammar feature), outlines, and guidance implement this. Some inference providers expose it as a configuration option.

// Example using a local inference server that supports JSON Schema constraints
// (e.g., llama.cpp server, vllm with guided decoding, or similar)

const schema = {
  type: "object",
  properties: {
    name: { type: "string" },
    date: { type: "string", format: "date" },
    location: { type: "string" },
    attendees: { type: "integer", minimum: 1 },
    is_virtual: { type: "boolean" },
  },
  required: ["name", "date", "location", "attendees", "is_virtual"],
  additionalProperties: false,
};

async function extractWithConstrainedDecoding(
  text: string
): Promise<Record<string, unknown>> {
  const response = await fetch("http://localhost:8080/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "your-model",
      messages: [
        {
          role: "user",
          content: `Extract event details: ${text}`,
        },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "event",
          schema,
          strict: true,
        },
      },
    }),
  });

  const data = await response.json();
  return JSON.parse(data.choices[0].message.content);
}

Constrained decoding gives you the strongest structural guarantees: the output literally cannot be syntactically or structurally invalid. But it has real costs. Compiling a grammar adds latency, especially for complex schemas. Very constrained schemas can hurt output quality because the model is forced into token sequences it would not have chosen, which can cause it to produce valid-structure but wrong-semantics output. And it requires running your own inference infrastructure or a provider that explicitly supports it.

When to use constrained decoding: When you are running your own inference, have strict schema requirements, and have validated that output quality is acceptable under the constraints. Good for high-volume batch extraction where structural validity is non-negotiable and you have tuned the schema and prompt together.

When it fails: Complex schemas with many enum values or deeply nested structures can degrade generation quality noticeably. Not available from all hosted providers. Adds operational overhead if you are adopting it specifically.

Tradeoffs Comparison

Criterion	JSON Mode	Function Calling	Constrained Decoding
Syntactic validity	Guaranteed	Guaranteed	Guaranteed
Schema compliance	Not guaranteed	High (not perfect)	Guaranteed
Required fields	Not enforced	Enforced by schema	Enforced by grammar
Semantic correctness	Not guaranteed	Not guaranteed	Not guaranteed
Setup complexity	Low	Medium	High
Latency overhead	Minimal	Minimal	Low to medium
Streaming support	Full	Partial	Partial
Hosted API availability	Wide	Wide	Limited
Schema evolution cost	Low	Medium	Medium to high

The one thing none of these approaches guarantees: semantic correctness. A field can be structurally valid and type-correct but factually wrong. A date can parse correctly but reference the wrong year. An address can be a valid string but be a hallucinated location. Structural validation is necessary but not sufficient.

Validation Strategy

Regardless of which approach you use, build validation in layers.

import { z } from "zod";

// Layer 1: Type validation (is it the shape we expect?)
const EventSchema = z.object({
  name: z.string().min(1),
  date: z.string().datetime(),
  location: z.string().min(1),
  attendees: z.number().int().positive(),
  is_virtual: z.boolean(),
});

// Layer 2: Business rule validation (does it make sense?)
function validateEventSemantics(event: z.infer<typeof EventSchema>): string[] {
  const errors: string[] = [];
  const date = new Date(event.date);

  if (isNaN(date.getTime())) {
    errors.push(`Invalid date value: ${event.date}`);
  }

  if (event.attendees > 100_000) {
    errors.push(`Suspiciously large attendee count: ${event.attendees}`);
  }

  if (event.is_virtual && event.location !== "online" && event.location !== "") {
    // Not an error, but flag for review
    errors.push(
      `Virtual event with physical location set: ${event.location} -- may need review`
    );
  }

  return errors;
}

async function extractAndValidate(text: string) {
  const raw = await extractEventWithFunctionCalling(text);

  // Layer 1
  const parseResult = EventSchema.safeParse(raw);
  if (!parseResult.success) {
    return { success: false as const, error: "schema_violation", details: parseResult.error };
  }

  // Layer 2
  const semanticErrors = validateEventSemantics(parseResult.data);
  if (semanticErrors.length > 0) {
    return {
      success: false as const,
      error: "semantic_validation",
      details: semanticErrors,
    };
  }

  return { success: true as const, data: parseResult.data };
}

Fallback Chains

No single approach succeeds 100% of the time. Build a fallback chain that degrades gracefully.

type ExtractionResult<T> =
  | { success: true; data: T; method: string }
  | { success: false; error: string; method: string };

async function extractWithFallback<T>(
  text: string,
  schema: z.ZodType<T>
): Promise<ExtractionResult<T>> {
  // Attempt 1: Function calling
  try {
    const result = await extractEventWithFunctionCalling(text);
    const parsed = schema.safeParse(result);
    if (parsed.success) {
      return { success: true, data: parsed.data, method: "function_calling" };
    }
  } catch (err) {
    console.warn("Function calling attempt failed:", err);
  }

  // Attempt 2: JSON mode with more explicit prompt
  try {
    const result = await extractWithJsonModeStrict(text, schema);
    const parsed = schema.safeParse(result);
    if (parsed.success) {
      return { success: true, data: parsed.data, method: "json_mode_strict" };
    }
  } catch (err) {
    console.warn("JSON mode attempt failed:", err);
  }

  // Attempt 3: Ask the model to fix its own output
  try {
    const corrected = await askModelToCorrect(text, schema);
    const parsed = schema.safeParse(corrected);
    if (parsed.success) {
      return { success: true, data: parsed.data, method: "self_correction" };
    }
  } catch (err) {
    console.warn("Self-correction attempt failed:", err);
  }

  return {
    success: false,
    error: "All extraction attempts failed",
    method: "none",
  };
}

async function askModelToCorrect<T>(
  originalText: string,
  schema: z.ZodType<T>
): Promise<unknown> {
  // Send the schema description and ask the model to retry with a very explicit prompt
  const response = await client.chat.completions.create({
    model: "gpt-4o",
    response_format: { type: "json_object" },
    messages: [
      {
        role: "system",
        content: `You must return a JSON object exactly matching this structure. No extra fields, no missing fields:\n${JSON.stringify((schema as z.ZodObject<z.ZodRawShape>).shape, null, 2)}`,
      },
      {
        role: "user",
        content: `Extract from: ${originalText}`,
      },
    ],
  });

  return JSON.parse(response.choices[0].message.content ?? "{}");
}

The self-correction step works surprisingly well for schema violations caused by field naming differences. It is not a cure-all, but it meaningfully reduces the failure rate without significant latency overhead when it triggers rarely.

Production Considerations

Log every raw response before parsing. When your extraction fails in production, you need to see exactly what the model returned. Parsed-and-discarded responses make debugging nearly impossible. Store the raw content with a correlation ID you can match to your structured output.

Track validation failure rates by method. JSON mode failures and function calling failures have different root causes. Aggregate error types separately so you know whether to tune your prompt, change your approach, or flag specific input patterns.

Instrument your fallback chain. If your fallback to JSON mode is triggering 15% of the time, that is a signal your function calling prompt needs work. If self-correction is triggering 5% of the time, that is worth investigating before it becomes 20%.

Schema versioning matters. When you evolve your schema, existing cached or queued inputs will hit the new validation. Use explicit version identifiers in your extraction pipeline so you can correlate failures with schema changes.

Timeout and circuit break the whole chain. Three retry attempts with self-correction can easily take 10-15 seconds under load. Set a hard timeout on the full extraction pipeline and surface a structured error rather than blocking indefinitely. Downstream systems should handle extraction failures as a known case, not an exception.

Test with adversarial inputs. LLMs handle adversarial and ambiguous text differently from clean test examples. Build a test fixture with inputs that include: missing information the schema requires, conflicting information, very long text, and text in multiple languages if relevant. These expose schema compliance gaps that clean examples hide.

Choosing an Approach

Start with function calling. It has the best combination of schema compliance, wide provider support, and operational simplicity. Add Zod validation on top. Build a fallback to JSON mode with a stricter prompt for the small fraction of calls that fail.

Add constrained decoding if: you are running your own inference infrastructure, you have high-volume batch workloads, and you have confirmed that schema constraints do not degrade output quality for your specific data.

Use JSON mode alone only for shallow schemas where field presence is not critical, or as a fallback layer in a multi-attempt chain.

The failure mode that matters most is silent schema violation: output that passes JSON.parse but fails your application logic later. Function calling with explicit Zod validation closes that gap. Everything else is optimization.

Structured output from LLMs is a solvable problem. The solution is not trusting the model, it is building a pipeline that validates, falls back, and surfaces failures with enough signal to fix them.