LLM Observability in Production: Tracing Chains, Measuring Latency, and Debugging Agent Failures

Your LLM application works in development. A few weeks after launch, users start reporting that answers are worse. A support ticket says the agent looped for 30 seconds before timing out. Your cost dashboard shows token usage spiked 4x overnight. You have no idea why any of this happened because you have no traces, no structured logs, and no baseline to compare against.

This is the most common production failure mode for LLM applications: not a crash, not a hard error, but a silent degradation you cannot diagnose because you never built the instrumentation to see it.

Standard application observability does not transfer cleanly to LLM systems. A span that tracks “database query, 12ms” is easy to interpret. A span that tracks “LLM call, 3400ms, 1800 tokens, output quality unknown” requires a different layer of tooling and discipline. The unit of failure is different. The failure modes are different. And the remediation path depends on data you likely are not collecting yet.

This article covers what to instrument, how to structure the data, and how to debug the failure modes that actually show up in production.

What Changes When Your App Has an LLM

In a traditional service, observability answers three questions: is it up, is it slow, and what went wrong. With LLM applications, those questions are still relevant but incomplete. You also need to answer: did the output make sense, did the model follow the prompt correctly, and did the cost match expectations.

The key structural differences:

• Chains and loops. A single user request may trigger 5-20 LLM calls, tool invocations, and retrieval steps. Each step can fail independently or produce output that causes downstream steps to fail silently.
• Non-determinism. The same input can produce different outputs. A regression might not be a code change. It might be a model update, a context window difference, or a retrieved document that changed.
• Token budgets. Cost is a function of token count, not request count. A single misconfigured prompt can spend 50x what you budgeted.
• Latency structure. Most of the latency is inside the model, not your code. Time-to-first-token (TTFT) and streaming throughput matter differently than total response time.

The instrumentation layer has to capture all of these, not just the surface-level timing.

Tracing Multi-Step Chains

The right mental model for LLM tracing is the same as distributed tracing: a root span per user request, with child spans for each operation. The difference is that the operations include LLM calls, tool executions, retrieval steps, and decision branches.

Here is a baseline tracer implementation in TypeScript that creates structured spans with the data you actually need:

import { randomUUID } from "crypto";

interface SpanData {
  spanId: string;
  parentSpanId?: string;
  traceId: string;
  name: string;
  startTime: number;
  endTime?: number;
  attributes: Record<string, unknown>;
  status: "ok" | "error" | "timeout";
  error?: string;
}

class LLMTracer {
  private spans: SpanData[] = [];
  private traceId: string;

  constructor(traceId?: string) {
    this.traceId = traceId ?? randomUUID();
  }

  startSpan(name: string, parentSpanId?: string, attributes: Record<string, unknown> = {}): SpanData {
    const span: SpanData = {
      spanId: randomUUID(),
      parentSpanId,
      traceId: this.traceId,
      name,
      startTime: Date.now(),
      attributes,
      status: "ok",
    };
    this.spans.push(span);
    return span;
  }

  endSpan(span: SpanData, attributes: Record<string, unknown> = {}, error?: Error): void {
    span.endTime = Date.now();
    span.attributes = { ...span.attributes, ...attributes };
    if (error) {
      span.status = "error";
      span.error = error.message;
    }
  }

  getTrace(): SpanData[] {
    return this.spans;
  }

  flush(): void {
    // Ship to your observability backend here
    console.log(JSON.stringify({ traceId: this.traceId, spans: this.spans }));
  }
}

Now apply this to a real agent loop. The span names and attributes are where the value lives:

import OpenAI from "openai";

const openai = new OpenAI();

interface LLMCallResult {
  content: string;
  promptTokens: number;
  completionTokens: number;
  model: string;
  latencyMs: number;
}

async function instrumentedLLMCall(
  tracer: LLMTracer,
  parentSpanId: string,
  messages: OpenAI.Chat.ChatCompletionMessageParam[],
  options: { model: string; temperature?: number }
): Promise<LLMCallResult> {
  const span = tracer.startSpan("llm.call", parentSpanId, {
    "llm.model": options.model,
    "llm.temperature": options.temperature ?? 1,
    "llm.prompt_tokens_estimate": messages.reduce((acc, m) => acc + String(m.content).length / 4, 0),
  });

  try {
    const start = Date.now();
    const response = await openai.chat.completions.create({
      model: options.model,
      messages,
      temperature: options.temperature,
    });

    const latencyMs = Date.now() - start;
    const usage = response.usage!;

    tracer.endSpan(span, {
      "llm.prompt_tokens": usage.prompt_tokens,
      "llm.completion_tokens": usage.completion_tokens,
      "llm.total_tokens": usage.total_tokens,
      "llm.latency_ms": latencyMs,
      "llm.finish_reason": response.choices[0].finish_reason,
      "llm.cost_usd": calculateCost(options.model, usage.prompt_tokens, usage.completion_tokens),
    });

    return {
      content: response.choices[0].message.content ?? "",
      promptTokens: usage.prompt_tokens,
      completionTokens: usage.completion_tokens,
      model: options.model,
      latencyMs,
    };
  } catch (err) {
    tracer.endSpan(span, {}, err as Error);
    throw err;
  }
}

function calculateCost(model: string, promptTokens: number, completionTokens: number): number {
  // Prices per million tokens (April 2026 estimates)
  const pricing: Record<string, { input: number; output: number }> = {
    "gpt-4o": { input: 2.5, output: 10 },
    "gpt-4o-mini": { input: 0.15, output: 0.6 },
    "claude-3-5-sonnet-20241022": { input: 3, output: 15 },
  };
  const rates = pricing[model] ?? { input: 2.5, output: 10 };
  return (promptTokens / 1_000_000) * rates.input + (completionTokens / 1_000_000) * rates.output;
}

For tool calls, add a span per invocation and capture the tool name, input arguments (sanitized), and result status. Tool failures are one of the primary agent failure modes and you need them in the trace:

async function instrumentedToolCall(
  tracer: LLMTracer,
  parentSpanId: string,
  toolName: string,
  args: Record<string, unknown>,
  executor: (args: Record<string, unknown>) => Promise<unknown>
): Promise<unknown> {
  const span = tracer.startSpan("tool.call", parentSpanId, {
    "tool.name": toolName,
    "tool.args": JSON.stringify(args),
  });

  try {
    const result = await executor(args);
    tracer.endSpan(span, {
      "tool.result_size_bytes": JSON.stringify(result).length,
      "tool.status": "success",
    });
    return result;
  } catch (err) {
    tracer.endSpan(span, { "tool.status": "error" }, err as Error);
    throw err;
  }
}

Measuring Token Usage and Cost Per Request

Token usage is your primary cost signal and your primary prompt health signal. Track it at two levels: per LLM call and per user request (aggregated across all calls in the chain).

Per-request aggregation lets you answer: what did this user interaction actually cost, and how does that compare to the budget assumption. If your expected cost per request is $0.002 and you are seeing requests at $0.04, you have a context window problem or a loop problem.

interface RequestMetrics {
  requestId: string;
  userId?: string;
  feature: string;
  totalPromptTokens: number;
  totalCompletionTokens: number;
  totalCostUsd: number;
  llmCallCount: number;
  toolCallCount: number;
  totalLatencyMs: number;
  spans: SpanData[];
}

function aggregateRequestMetrics(tracer: LLMTracer, requestId: string, feature: string): RequestMetrics {
  const spans = tracer.getTrace();

  const llmSpans = spans.filter((s) => s.name === "llm.call");
  const toolSpans = spans.filter((s) => s.name === "tool.call");
  const rootSpan = spans[0];

  return {
    requestId,
    feature,
    totalPromptTokens: llmSpans.reduce((acc, s) => acc + ((s.attributes["llm.prompt_tokens"] as number) ?? 0), 0),
    totalCompletionTokens: llmSpans.reduce((acc, s) => acc + ((s.attributes["llm.completion_tokens"] as number) ?? 0), 0),
    totalCostUsd: llmSpans.reduce((acc, s) => acc + ((s.attributes["llm.cost_usd"] as number) ?? 0), 0),
    llmCallCount: llmSpans.length,
    toolCallCount: toolSpans.length,
    totalLatencyMs: (rootSpan.endTime ?? Date.now()) - rootSpan.startTime,
    spans,
  };
}

Ship these metrics to your time-series database alongside the traces. Alert on:

• total_cost_usd p95 per feature, 20% above rolling 7-day average
• llm_call_count per request exceeding your expected maximum (agent loop detection)
• completion_tokens per call approaching your max_tokens limit (truncation risk)

Detecting Prompt Regressions and Quality Drift

Quality drift is the hardest problem in LLM production. The application did not crash. Users are just getting worse answers, and you cannot tell when it started or why.

The minimum viable approach is to log every LLM interaction with enough structure to run evaluations offline. You need the full prompt, the full completion, the model version, and the timestamp. That alone lets you run a batch eval against historical data when someone reports a regression.

interface LLMInteractionLog {
  interactionId: string;
  traceId: string;
  timestamp: string;
  feature: string;
  model: string;
  modelVersion?: string;
  systemPrompt: string;
  userMessages: OpenAI.Chat.ChatCompletionMessageParam[];
  completion: string;
  promptTokens: number;
  completionTokens: number;
  latencyMs: number;
  metadata: Record<string, unknown>;
}

async function logInteraction(log: LLMInteractionLog): Promise<void> {
  // Write to append-only storage: S3, BigQuery, Clickhouse, etc.
  // Never log to general application logs — access control matters here
  await writeToInteractionStore(log);
}

The step beyond logging is automated scoring. For each interaction, run a lightweight eval after the fact: does the response answer the question, does it stay in scope, does it avoid known failure patterns. Store the score alongside the log.

A practical approach uses a cheaper model as a judge:

async function scoreInteraction(
  interaction: LLMInteractionLog
): Promise<{ score: number; flags: string[] }> {
  const prompt = `You are evaluating an AI assistant response.

User question: ${interaction.userMessages.at(-1)?.content}
Assistant response: ${interaction.completion}

Score from 0-1 on these criteria:
- Relevance: does it answer what was asked?
- Groundedness: does it avoid unsupported claims?
- Completeness: is the answer meaningfully complete?

Return JSON: { "score": <0-1>, "flags": ["<issue>", ...] }`;

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: prompt }],
    response_format: { type: "json_object" },
  });

  return JSON.parse(response.choices[0].message.content ?? "{}");
}

Track the rolling average score per feature. A 5% drop sustained over 48 hours is your regression signal. Compare timestamps against model updates, prompt changes, and retrieved document updates.

Observability Tooling Landscape

You can build the above from scratch or adopt a platform. The practical tradeoffs:

Tool	Approach	Strengths	Limitations
LangSmith	Managed, LangChain-native	Deep LangChain integration, built-in evals	Vendor lock-in, LangChain dependency
Langfuse	Open-source, self-hostable	Full data ownership, SDK-agnostic, cost attribution	Smaller ecosystem than LangSmith
Helicone	Proxy-based	Zero instrumentation change, cost tracking	Less trace depth, proxy adds latency
OpenTelemetry	Open standard	Works with existing infra, no vendor lock	Requires custom semantic conventions for LLMs
Custom logging	Full control	Fits exact data model	Highest build cost, no built-in evals UI

For teams already running OpenTelemetry, the GenAI semantic conventions working group defines standard attribute names (gen_ai.request.model, gen_ai.usage.prompt_tokens, etc.). Using those names lets your LLM traces flow into the same backends as your service traces with proper correlation.

For teams without existing observability infrastructure, Langfuse is the lowest-friction path to a useful UI without lock-in.

Debugging Agent Failure Modes

Agent systems have failure modes that single-call LLM applications do not. The three you will encounter most:

Infinite loops. The agent keeps calling tools or re-prompting itself because it cannot determine it has reached a terminal state. The trace shows llm_call_count far above the expected maximum. The fix is both a circuit breaker in code and an alert on call count.

class AgentLoop {
  private callCount = 0;
  private readonly maxCalls: number;

  constructor(private tracer: LLMTracer, maxCalls = 10) {
    this.maxCalls = maxCalls;
  }

  async step(
    messages: OpenAI.Chat.ChatCompletionMessageParam[],
    parentSpanId: string
  ): Promise<string | null> {
    if (this.callCount >= this.maxCalls) {
      const span = this.tracer.startSpan("agent.circuit_breaker", parentSpanId);
      this.tracer.endSpan(span, {
        "agent.call_count": this.callCount,
        "agent.max_calls": this.maxCalls,
        "agent.status": "aborted",
      });
      return null; // surface as an error to the user
    }

    this.callCount++;
    return instrumentedLLMCall(this.tracer, parentSpanId, messages, { model: "gpt-4o" }).then(
      (r) => r.content
    );
  }
}

Tool errors that mislead the model. When a tool fails and you return the error message to the model, it often tries to work around the error by hallucinating what the tool result should have been. The trace shows a tool span with status: error followed by an LLM span with high completion tokens. The fix is to make tool errors unambiguous in the message format and add a span attribute that flags the “tool error followed by LLM call” pattern.

Hallucination chains. The model generates a fact in step 2 that it then cites as context in step 5, compounding the error. This is harder to detect at runtime. The practical mitigation is retrieval grounding: require that any factual claim in a chain step links back to a retrieved document, and flag responses that reference information not present in the retrieval span’s output.

interface RetrievalSpanData extends SpanData {
  retrievedDocumentIds: string[];
}

function detectGroundednessViolation(
  llmSpan: SpanData,
  priorRetrievalSpans: RetrievalSpanData[],
  completion: string
): boolean {
  // Simplified: in practice, use embedding similarity or NLI model
  const retrievedContent = priorRetrievalSpans
    .flatMap((s) => s.retrievedDocumentIds)
    .join(" ");

  // Flag if completion is long but retrieval context was empty
  const hasRetrieval = retrievedContent.length > 0;
  const hasLongCompletion = (llmSpan.attributes["llm.completion_tokens"] as number) > 200;

  return hasLongCompletion && !hasRetrieval;
}

Production Considerations

A few things that only become apparent after shipping:

Prompt hashing. Hash the system prompt and include it in every trace. When a regression appears, you can immediately filter by prompt version to see if a prompt change correlates. Without this, you are doing archaeology.

import { createHash } from "crypto";

function hashPrompt(systemPrompt: string): string {
  return createHash("sha256").update(systemPrompt).digest("hex").slice(0, 12);
}

Sampling strategy. You cannot log every interaction at full fidelity at scale. Log 100% of errors and circuit-breaker events. Sample successes at 10-20% for cost tracking and evals. Keep the full trace for any request where total_cost_usd is above your p95 threshold.

Separate access control for interaction logs. LLM interaction logs contain user inputs and model outputs. These are sensitive in ways that application logs typically are not. Do not route them to the same destination as your service logs. Treat them with the same access control as user data.

Latency decomposition. Total latency for an LLM request has three components: time in your code before the call, time-to-first-token from the model, and streaming duration. Track all three separately. If TTFT is fine but total latency is high, the issue is in your postprocessing, not the model.

interface LatencyBreakdown {
  preCallMs: number;       // Your code: prompt assembly, retrieval, context building
  ttftMs: number;          // Time from API call to first token
  streamingMs: number;     // Time from first token to last token
  postCallMs: number;      // Your code: parsing, storage, response formatting
}

Alert on quality score drops, not just errors. An LLM application with a 0.2% error rate but a 20% quality score decline is in worse shape than one with a 2% error rate and stable quality. Wire your offline eval scores into your alerting pipeline.

Observability for LLM systems is not a nice-to-have you add later. The failure modes are too quiet and too expensive. The instrumentation described here takes a few days to build properly, and it will surface problems that would otherwise take weeks of user complaints to track down.