LLM Observability in Production: Tracing Completions, Measuring Latency, and Debugging Agent Workflows

Traditional application observability maps cleanly onto requests: one request, one database query, one response. You trace the HTTP call, measure latency at the p99, alert on error rate. The mental model works.

LLM-powered applications break that mental model in three specific ways. First, a single user action can trigger a chain of completions: retrieval, classification, generation, validation, tool calls. Each step has its own latency and failure mode. Second, “success” is ambiguous. A 200 response with a coherent-sounding answer may still be wrong, off-topic, or degraded compared to last week. Third, cost is a runtime concern. Token usage is not a deployment metric, it is a per-request operational cost that can spike unexpectedly when a prompt template changes.

Without instrumentation that handles all three, you are flying blind. Users report hallucinations before your dashboards do. A prompt change that looked fine in evaluation quietly doubles your token spend. An agent loop that usually resolves in three steps occasionally spins through twelve before timing out, and you find out from a support ticket.

This article covers how to build LLM observability that actually works in production: tracing agent chains with OpenTelemetry, collecting structured span data per completion, measuring latency at the model and provider level, detecting quality regressions with Langfuse, and wiring everything into dashboards that surface problems early.

The Observability Gap in LLM Applications

The gap is not a tooling problem. It is a modeling problem. Standard APM tools treat LLM calls as HTTP requests to an external service. They record duration, status code, and maybe request size. That is not wrong, it is just incomplete.

What you actually need to observe:

• Prompt inputs and outputs per step, not just the final response to the user
• Token counts broken down by prompt tokens and completion tokens, per call
• Model and provider per call, especially if you route between providers or use fallbacks
• Latency split: time-to-first-token (TTFT) vs total completion time
• Chain structure: which completions are children of which parent agent step
• Quality signals: did the output match expected format, pass validation, or score above a threshold on a rubric

None of these are captured by default. You have to instrument them.

Tracing Agent Chains with OpenTelemetry

OpenTelemetry gives you the span hierarchy and propagation primitives. The key is treating each LLM call as a child span of the agent step that triggered it, and each agent step as a child span of the user-visible operation.

Start with a typed span attribute schema so every span carries consistent metadata:

import { trace, context, SpanStatusCode, Span } from "@opentelemetry/api";

interface LLMSpanAttributes {
  "llm.provider": string;
  "llm.model": string;
  "llm.prompt_tokens": number;
  "llm.completion_tokens": number;
  "llm.total_tokens": number;
  "llm.ttft_ms": number | null; // time to first token
  "llm.latency_ms": number;
  "llm.temperature": number;
  "llm.step_name": string;
  "llm.trace_id": string;
}

const tracer = trace.getTracer("llm-service", "1.0.0");

async function tracedCompletion(
  stepName: string,
  fn: (span: Span) => Promise<{ text: string; usage: TokenUsage; ttftMs: number | null }>
): Promise<{ text: string; usage: TokenUsage; ttftMs: number | null }> {
  return tracer.startActiveSpan(`llm.completion.${stepName}`, async (span) => {
    const startMs = Date.now();
    try {
      const result = await fn(span);
      const latencyMs = Date.now() - startMs;

      span.setAttributes({
        "llm.step_name": stepName,
        "llm.latency_ms": latencyMs,
        "llm.prompt_tokens": result.usage.promptTokens,
        "llm.completion_tokens": result.usage.completionTokens,
        "llm.total_tokens": result.usage.totalTokens,
        "llm.ttft_ms": result.ttftMs ?? -1,
      });

      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
      span.recordException(err as Error);
      throw err;
    } finally {
      span.end();
    }
  });
}

Now wire this into a multi-step agent. The agent span wraps all child completion spans, which means your trace viewer will show the full chain with timing:

interface AgentStep {
  name: string;
  run: () => Promise<string>;
}

async function runAgentTrace(
  agentName: string,
  sessionId: string,
  steps: AgentStep[]
): Promise<string[]> {
  return tracer.startActiveSpan(`agent.run.${agentName}`, async (agentSpan) => {
    agentSpan.setAttribute("agent.session_id", sessionId);
    agentSpan.setAttribute("agent.step_count", steps.length);

    const results: string[] = [];

    for (const step of steps) {
      const result = await tracer.startActiveSpan(
        `agent.step.${step.name}`,
        async (stepSpan) => {
          try {
            const output = await step.run();
            stepSpan.setStatus({ code: SpanStatusCode.OK });
            return output;
          } catch (err) {
            stepSpan.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
            stepSpan.recordException(err as Error);
            throw err;
          } finally {
            stepSpan.end();
          }
        }
      );
      results.push(result);
    }

    agentSpan.setAttribute("agent.completed_steps", results.length);
    agentSpan.end();
    return results;
  });
}

The parent-child span relationship is handled automatically by OpenTelemetry’s context propagation. When you call tracer.startActiveSpan inside an existing active span, it becomes a child. The trace hierarchy in Jaeger, Honeycomb, or Grafana Tempo will reflect the actual execution tree.

Measuring Latency at the Right Granularity

Wall-clock latency on an LLM call is misleading on its own. A call that takes 4 seconds with a 200ms time-to-first-token feels fast to a user if you are streaming. A call that takes 2 seconds with a 1,800ms TTFT feels broken.

Split your latency measurement:

import OpenAI from "openai";

interface TokenUsage {
  promptTokens: number;
  completionTokens: number;
  totalTokens: number;
}

interface CompletionResult {
  text: string;
  usage: TokenUsage;
  ttftMs: number | null;
  totalMs: number;
}

async function streamingCompletion(
  client: OpenAI,
  model: string,
  messages: OpenAI.Chat.ChatCompletionMessageParam[]
): Promise<CompletionResult> {
  const startMs = Date.now();
  let ttftMs: number | null = null;
  let text = "";

  const stream = await client.chat.completions.create({
    model,
    messages,
    stream: true,
    stream_options: { include_usage: true },
  });

  let usage: TokenUsage = { promptTokens: 0, completionTokens: 0, totalTokens: 0 };

  for await (const chunk of stream) {
    if (ttftMs === null && chunk.choices[0]?.delta?.content) {
      ttftMs = Date.now() - startMs;
    }

    const content = chunk.choices[0]?.delta?.content ?? "";
    text += content;

    if (chunk.usage) {
      usage = {
        promptTokens: chunk.usage.prompt_tokens,
        completionTokens: chunk.usage.completion_tokens,
        totalTokens: chunk.usage.total_tokens,
      };
    }
  }

  return {
    text,
    usage,
    ttftMs,
    totalMs: Date.now() - startMs,
  };
}

Record both TTFT and total latency as separate span attributes. Aggregate them separately in your dashboards. TTFT degradation before total latency degradation is an early signal that the model is under load or that your prompt size increased.

Also measure latency per model and provider. If you route between gpt-4o and claude-3-5-sonnet-20241022 depending on task complexity, you need per-model p50 and p99 to understand where slowdowns are coming from:

async function routedCompletion(
  task: "simple" | "complex",
  messages: OpenAI.Chat.ChatCompletionMessageParam[]
): Promise<CompletionResult> {
  const model = task === "complex" ? "gpt-4o" : "gpt-4o-mini";

  return tracer.startActiveSpan(`llm.routed_completion`, async (span) => {
    span.setAttribute("llm.model", model);
    span.setAttribute("llm.task_type", task);

    const result = await streamingCompletion(openai, model, messages);

    span.setAttributes({
      "llm.ttft_ms": result.ttftMs ?? -1,
      "llm.total_ms": result.totalMs,
      "llm.prompt_tokens": result.usage.promptTokens,
      "llm.completion_tokens": result.usage.completionTokens,
    });

    span.end();
    return result;
  });
}

With this in place, you can slice your latency dashboards by llm.model and immediately see if a model-level SLA is being violated independently from application-level latency.

Structured Logging for Agent State

Spans handle timing and hierarchy. For content and state, use structured log events correlated to the same trace ID. This keeps span payloads small while making prompt inputs and outputs queryable.

import { trace } from "@opentelemetry/api";

interface LLMLogEvent {
  traceId: string;
  spanId: string;
  event: "llm.prompt" | "llm.completion" | "llm.tool_call" | "llm.tool_result";
  stepName: string;
  model?: string;
  messages?: unknown[];
  output?: string;
  toolName?: string;
  toolArgs?: unknown;
  toolResult?: unknown;
  timestampMs: number;
}

function logLLMEvent(event: Omit<LLMLogEvent, "traceId" | "spanId" | "timestampMs">): void {
  const activeSpan = trace.getActiveSpan();
  const spanContext = activeSpan?.spanContext();

  const logEvent: LLMLogEvent = {
    ...event,
    traceId: spanContext?.traceId ?? "none",
    spanId: spanContext?.spanId ?? "none",
    timestampMs: Date.now(),
  };

  // Ship to your log aggregator (Axiom, Loki, CloudWatch Logs, etc.)
  console.log(JSON.stringify(logEvent));
}

By attaching traceId and spanId to every log line, you can pivot from a slow span in your trace viewer directly to the prompt that caused it. This is the workflow that matters when debugging a specific user complaint: find the trace, click the slow span, filter logs by traceId.

Do not log raw user content to production log storage without a data retention policy. Prompt logging is a compliance surface.

Quality Regression Detection with Langfuse

Latency and token counts tell you the application is running. They do not tell you whether it is running well. Quality regression detection requires a different layer.

Langfuse integrates directly into your completion pipeline and supports scoring, evaluation runs, and dataset comparisons. The SDK wraps your completions and records everything as “traces” in Langfuse’s model, separate from your OTel traces.

import { Langfuse } from "langfuse";

const langfuse = new Langfuse({
  secretKey: process.env.LANGFUSE_SECRET_KEY!,
  publicKey: process.env.LANGFUSE_PUBLIC_KEY!,
  baseUrl: "https://cloud.langfuse.com",
});

interface EvaluatedCompletion {
  text: string;
  traceId: string;
  usage: TokenUsage;
}

async function instrumentedCompletion(
  sessionId: string,
  stepName: string,
  input: string,
  systemPrompt: string
): Promise<EvaluatedCompletion> {
  const trace = langfuse.trace({
    name: stepName,
    sessionId,
    input,
    metadata: { environment: process.env.NODE_ENV },
  });

  const generation = trace.generation({
    name: `${stepName}.generation`,
    model: "gpt-4o",
    input: [
      { role: "system", content: systemPrompt },
      { role: "user", content: input },
    ],
  });

  const result = await streamingCompletion(openai, "gpt-4o", [
    { role: "system", content: systemPrompt },
    { role: "user", content: input },
  ]);

  generation.end({
    output: result.text,
    usage: {
      input: result.usage.promptTokens,
      output: result.usage.completionTokens,
    },
  });

  trace.update({ output: result.text });
  await langfuse.flushAsync();

  return {
    text: result.text,
    traceId: trace.id,
    usage: result.usage,
  };
}

Once completions are recorded, you can attach scores programmatically. Scores can come from automated evaluators (format validation, output parsers, rubric-based LLM judges) or from user feedback signals (thumbs up/down, explicit ratings):

async function scoreCompletion(
  traceId: string,
  name: "format_valid" | "relevance" | "user_feedback",
  value: number, // 0 to 1
  comment?: string
): Promise<void> {
  await langfuse.score({
    traceId,
    name,
    value,
    comment,
  });
}

// Example: validate that the output is parseable JSON before scoring
async function validateJsonOutput(
  traceId: string,
  output: string
): Promise<boolean> {
  try {
    JSON.parse(output);
    await scoreCompletion(traceId, "format_valid", 1.0);
    return true;
  } catch {
    await scoreCompletion(traceId, "format_valid", 0.0, "Output is not valid JSON");
    return false;
  }
}

The value of Langfuse over pure OTel for quality work is the dataset and evaluation run workflow. You can take a set of production traces that scored low, promote them to a dataset, run a new prompt version against the same inputs, and compare scores side by side before deploying. That workflow is what separates teams that catch regressions in staging from teams that catch them in production.

Dashboard Structure for LLM Applications

Raw metrics are not enough. The dashboard structure determines what you notice and when. For LLM applications, organize dashboards around these four views:

Operational health (refresh every 30s):

• Completion success rate by model and provider
• p50 / p95 / p99 TTFT and total latency, segmented by model
• Error rate with error type breakdown (timeout, rate limit, context length exceeded, content filter)
• Token usage per minute, segmented by model and step name

Cost tracking (daily/weekly):

• Total tokens by model, with dollar cost using current provider pricing
• Cost per session and cost per completed task
• Outlier detection: sessions in the top 5% of token spend

Quality trends (daily):

• Average score per step name over the last 14 days
• Score distribution shift alerts (if p25 score drops, that is a regression)
• Format validation pass rate per step

Agent behavior (for debugging):

• Step count distribution per agent run: flag sessions where step count exceeds expected max
• Tool call frequency: if a specific tool is being called at 10x the expected rate, the agent may be looping
• Trace waterfall for recent high-latency or low-score sessions

The last view is the one that enables actual debugging. When a user reports a bad response, you need to pull the trace, see every completion in the chain, read the prompts, and understand what the agent was reasoning about. That is only possible if your spans carry step names and your log correlation is working.

Tradeoffs

Dimension	OTel-only	OTel + Langfuse	Custom instrumentation
Setup time	Low	Medium	High
Latency / cost visibility	Yes	Yes	Yes
Quality scoring	No	Yes	Yes (manual)
Dataset + eval runs	No	Yes	No (build it yourself)
Vendor dependency	None	Langfuse SaaS or self-host	None
Prompt content retention	Opt-in via log events	Built-in (with config)	Full control
Compliance surface	Low	Medium (data leaves your infra in SaaS mode)	Low

If you are self-hosting Langfuse (Docker Compose or Kubernetes), the compliance surface is the same as OTel. If you are using Langfuse Cloud, your prompt inputs and outputs are leaving your infrastructure. That matters for applications handling sensitive data.

Production Considerations

Sampling strategy: Do not trace every completion at full detail in high-volume production. Use head-based sampling at 10-20% for operational dashboards, but keep 100% sampling for error paths and low-score completions. OTel supports this with custom samplers.

Async flushing: Langfuse’s flushAsync() call should not be in the critical path of your response. If you are using serverless functions with short timeouts, use langfuse.shutdownAsync() or configure a background flush queue.

Span size limits: OTel collectors and backends reject spans above certain attribute sizes. Do not attach full prompt text as span attributes. Log them separately as structured log events correlated by trace ID. Keep span attributes for numeric metrics and short string identifiers.

Alert thresholds: Set alerts on TTFT p95 exceeding your SLA, on format validation pass rate dropping below 95%, and on agent step count exceeding 2x the expected max. Those three alerts will catch most production LLM problems before users report them.

Prompt versioning: Every completion span should carry a llm.prompt_version attribute that identifies which version of the prompt template was used. Without this, a quality regression after a prompt update is invisible: you cannot correlate the score drop with the change.

Retention and cost: Storing full prompt inputs and outputs is expensive at scale. Define retention tiers: full content for 7 days, metadata-only (scores, token counts, latency) for 90 days, aggregates forever.

The Instrumentation-First Principle

The temptation when building LLM applications is to optimize the prompt first and instrument later. That is backwards. Without instrumentation, you do not know which steps are slow, which steps are expensive, which prompt versions caused a regression, or why a specific user’s session went wrong.

The single most valuable thing you can do before shipping an LLM feature to production is to wire up span-level tracing with step names, token counts, and TTFT. Everything else, the quality scoring, the eval runs, the cost dashboards, builds on top of that foundation. You cannot add it after the fact and expect to understand what happened last Tuesday.

Start with tracedCompletion wrapping every model call. Add traceId to every log line. Push token counts and TTFT as span attributes. That takes a few hours and it is the difference between debugging in the dark and debugging with a map.