LLM Observability in Production: Token Tracking, Latency Budgets, and Prompt Drift Detection

Basic logging tells you the LLM responded. It does not tell you whether the response was correct, whether the cost is trending toward a budget overrun, or whether your prompts are silently degrading as the model provider updates its weights. You will not catch these problems until a user files a bug, a bill arrives, or a metric already has a weeks-old drift baked in.

This guide covers what production LLM observability actually requires: structured logging, token cost attribution per feature, latency budget design, prompt drift detection, quality regression alerting, tracing through multi-step agent chains, and how the available tooling compares.

---

Structured Logging for LLM Requests

console.log(response.choices[0].message.content) is not observability. You need a structured record per call that captures everything relevant to cost, latency, and correctness.

interface LLMCallRecord {
  traceId: string;
  spanId: string;
  model: string;
  feature: string;          // e.g. "document-summary", "chat-reply"
  userId?: string;
  promptTokens: number;
  completionTokens: number;
  totalTokens: number;
  costUsd: number;
  latencyMs: number;
  ttfbMs: number;           // time to first byte (streaming)
  finishReason: string;     // "stop" | "length" | "content_filter"
  promptVersion: string;    // semantic version of your prompt template
  requestedAt: string;      // ISO 8601
  completedAt: string;
  success: boolean;
  errorCode?: string;
}

The feature field is the piece most teams skip. Without it, you cannot attribute cost or latency to a specific product surface. Within three months of shipping LLM features you will have five or more calling the same model, and you will want to know which one is responsible for the spike.

import OpenAI from "openai";

const client = new OpenAI();

async function callLLM(params: {
  feature: string;
  promptVersion: string;
  messages: OpenAI.ChatCompletionMessageParam[];
  userId?: string;
}): Promise<{ content: string; record: LLMCallRecord }> {
  const traceId = crypto.randomUUID();
  const requestedAt = new Date().toISOString();
  const start = performance.now();

  const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: params.messages,
  });

  const latencyMs = Math.round(performance.now() - start);
  const usage = response.usage!;
  const costUsd = computeCost("gpt-4o", usage.prompt_tokens, usage.completion_tokens);

  const record: LLMCallRecord = {
    traceId,
    spanId: crypto.randomUUID(),
    model: "gpt-4o",
    feature: params.feature,
    userId: params.userId,
    promptTokens: usage.prompt_tokens,
    completionTokens: usage.completion_tokens,
    totalTokens: usage.total_tokens,
    costUsd,
    latencyMs,
    ttfbMs: 0,  // populate from streaming events when applicable
    finishReason: response.choices[0].finish_reason,
    promptVersion: params.promptVersion,
    requestedAt,
    completedAt: new Date().toISOString(),
    success: true,
  };

  logger.info("llm.call", record);

  return {
    content: response.choices[0].message.content ?? "",
    record,
  };
}

function computeCost(
  model: string,
  promptTokens: number,
  completionTokens: number
): number {
  // Prices in USD per 1M tokens (as of March 2026)
  const pricing: Record<string, { input: number; output: number }> = {
    "gpt-4o":        { input: 2.50,  output: 10.00 },
    "gpt-4o-mini":   { input: 0.15,  output: 0.60 },
    "claude-3-5-sonnet": { input: 3.00, output: 15.00 },
    "gemini-2.0-flash":  { input: 0.10, output: 0.40 },
  };
  const p = pricing[model];
  if (!p) return 0;
  return (promptTokens / 1_000_000) * p.input + (completionTokens / 1_000_000) * p.output;
}

Use a structured logger (pino, winston) rather than console.log so these records are queryable in your log aggregator. Ship them as JSON with consistent field names and you can build dashboards, alerts, and cost reports without a third-party LLM observability product.

---

Token Cost Attribution

Token cost becomes visible at the feature level only if you tag every call. Once you have the feature field, aggregation is straightforward:

SELECT
  feature,
  DATE_TRUNC('day', requested_at) AS day,
  SUM(cost_usd)                   AS daily_cost_usd,
  SUM(total_tokens)               AS total_tokens,
  COUNT(*)                        AS call_count,
  AVG(total_tokens)               AS avg_tokens_per_call
FROM llm_calls
WHERE requested_at > NOW() - INTERVAL '30 days'
GROUP BY 1, 2
ORDER BY 1, 2;

Two numbers matter beyond the raw total: cost per user and cost per conversion. A document summary feature costing $0.03 per call is fine if it runs once per user session. The same feature is a problem if it runs 40 times during a session due to an overly aggressive refresh trigger. You will only catch this by joining your LLM call logs with session data.

Set budget alerts at the feature level:

async function checkCostBudget(feature: string, windowHours: number = 1): Promise<void> {
  const cost = await db.query<{ total: number }>(
    `SELECT SUM(cost_usd) AS total
     FROM llm_calls
     WHERE feature = $1
       AND requested_at > NOW() - INTERVAL '${windowHours} hours'`,
    [feature]
  );
  const budgets: Record<string, number> = {
    "document-summary": 50,   // $50/hour max
    "chat-reply": 200,
    "code-review": 30,
  };
  const limit = budgets[feature] ?? 100;
  if ((cost.rows[0]?.total ?? 0) > limit) {
    alerting.fire(`llm.cost.budget_exceeded`, { feature, windowHours, limit });
  }
}

---

Latency Budget Design

LLM calls are slow relative to everything else in your stack. A gpt-4o call averages 800ms to 2000ms for typical prompt sizes. If your overall API endpoint has a P99 SLO of 1500ms, you have already blown the budget before your database queries run.

The fix is to design latency budgets explicitly, not discover them in production.

interface LatencyBudget {
  feature: string;
  p50Ms: number;
  p95Ms: number;
  p99Ms: number;
  hardTimeoutMs: number;   // circuit-break here, return fallback
}

const LATENCY_BUDGETS: LatencyBudget[] = [
  {
    feature: "chat-reply",
    p50Ms: 900,
    p95Ms: 3000,
    p99Ms: 6000,
    hardTimeoutMs: 10000,
  },
  {
    feature: "document-summary",
    p50Ms: 2000,
    p95Ms: 5000,
    p99Ms: 9000,
    hardTimeoutMs: 15000,
  },
  {
    feature: "autocomplete-suggestion",
    p50Ms: 400,
    p95Ms: 1200,
    p99Ms: 2500,
    hardTimeoutMs: 3000,
  },
];

For latency-sensitive features, use streaming with ttfbMs as your primary signal. Time to first byte is what users perceive when tokens stream in; total latency matters for non-streaming calls and for downstream processing. Track both.

If a call exceeds the hard timeout, you have two choices: surface a graceful fallback (cached result, partial response, “try again” message) or abort and increment a circuit-breaker counter. Do not let the LLM timeout silently cascade into a user-facing 504.

Alert on P95 drift, not P50. P50 latency can look fine while P95 quietly degrades as your prompt grows longer with each feature iteration. A rising P95 is often the first signal that your prompt is growing out of control.

---

Prompt Drift Detection

Prompt drift happens in two forms. First, your prompts change intentionally through feature development, but untracked: a developer edits the system message “just slightly,” the change ships without a version bump, and now you cannot correlate behavioral changes with the edit. Second, your prompts stay the same, but the model changes beneath them: providers do rolling weight updates that alter behavior without changing the model name.

Tracking prompt versions is the foundation:

const PROMPT_REGISTRY: Record<string, { version: string; template: string }> = {
  "document-summary-v1.2": {
    version: "1.2",
    template: `Summarize the following document in 3-5 sentences.
Focus on key decisions and action items.
Document: {{document}}`,
  },
};

function buildPrompt(key: string, vars: Record<string, string>): {
  messages: OpenAI.ChatCompletionMessageParam[];
  version: string;
} {
  const entry = PROMPT_REGISTRY[key];
  if (!entry) throw new Error(`Unknown prompt key: ${key}`);
  const content = Object.entries(vars).reduce(
    (t, [k, v]) => t.replace(`{{${k}}}`, v),
    entry.template
  );
  return {
    messages: [{ role: "user", content }],
    version: entry.version,
  };
}

Store every prompt version in source control and deploy it as application code, not as a database record that can be edited without a deploy. This gives you a git-level audit trail.

For detecting behavioral drift when model weights change underneath you, you need an evaluation baseline: a fixed set of test inputs with known expected outputs, run on a schedule.

interface EvalCase {
  id: string;
  promptKey: string;
  input: Record<string, string>;
  expectedKeywords: string[];    // must appear in output
  forbiddenKeywords: string[];   // must NOT appear in output
  maxTokens: number;
}

async function runDriftEval(cases: EvalCase[]): Promise<{
  passed: number;
  failed: number;
  driftScore: number;  // 0 = no drift, 1 = complete drift
}> {
  let passed = 0;
  let failed = 0;

  for (const c of cases) {
    const { messages, version } = buildPrompt(c.promptKey, c.input);
    const { content } = await callLLM({
      feature: "eval",
      promptVersion: version,
      messages,
    });

    const lower = content.toLowerCase();
    const keywordsPresent = c.expectedKeywords.every((kw) =>
      lower.includes(kw.toLowerCase())
    );
    const forbidden = c.forbiddenKeywords.some((kw) =>
      lower.includes(kw.toLowerCase())
    );
    const withinLength = content.split(/\s+/).length <= c.maxTokens;

    if (keywordsPresent && !forbidden && withinLength) {
      passed++;
    } else {
      failed++;
      logger.warn("eval.drift_detected", { caseId: c.id, content });
    }
  }

  const total = cases.length;
  return {
    passed,
    failed,
    driftScore: failed / total,
  };
}

Run this eval suite daily (or on every model version change if you pin versions explicitly). Alert when driftScore crosses 0.1 (10% of cases failing). A drift score of zero two weeks ago and 0.3 today is a signal to investigate, whether the cause is a model update, a prompt regression, or data distribution shift in your inputs.

---

Quality Regression Alerting

Drift detection catches behavioral change. Quality regression alerting catches output degradation in production traffic, where you cannot run a fixed eval suite because inputs are arbitrary.

Two tractable signals without human labels:

Finish reason distribution. If finish_reason: "length" starts appearing where it did not before, your prompts are hitting the context window limit and outputs are being cut off.

User action correlation. If users can reject, regenerate, or thumbs-down a response, track that signal against your LLM call logs. A regeneration rate above 15% on a feature is a quality regression, regardless of whether the output looks fine to the model.

async function trackUserFeedback(params: {
  traceId: string;
  action: "accepted" | "rejected" | "regenerated";
  featureName: string;
}): Promise<void> {
  await db.query(
    `INSERT INTO llm_feedback (trace_id, action, feature, recorded_at)
     VALUES ($1, $2, $3, NOW())`,
    [params.traceId, params.action, params.featureName]
  );

  // Recompute rolling rejection rate for alerting
  const rate = await db.query<{ rejection_rate: number }>(
    `SELECT
       ROUND(
         COUNT(*) FILTER (WHERE action IN ('rejected', 'regenerated'))::numeric
         / NULLIF(COUNT(*), 0),
         3
       ) AS rejection_rate
     FROM llm_feedback
     WHERE feature = $1
       AND recorded_at > NOW() - INTERVAL '1 hour'`,
    [params.featureName]
  );

  if ((rate.rows[0]?.rejection_rate ?? 0) > 0.15) {
    alerting.fire("llm.quality.high_rejection_rate", {
      feature: params.featureName,
      rate: rate.rows[0].rejection_rate,
    });
  }
}

---

Tracing Multi-Step Agent Chains

Single LLM calls are debuggable with per-call logs. Multi-step agents are not. When an agent calls three tools and two model completions before producing a final answer, you need a trace that shows the entire execution path, not five independent log lines.

Use OpenTelemetry spans with parent-child relationships:

import { trace, context, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("llm-agent");

async function runAgentChain(input: string, userId: string): Promise<string> {
  return tracer.startActiveSpan("agent.run", async (rootSpan) => {
    rootSpan.setAttributes({
      "agent.input_length": input.length,
      "agent.user_id": userId,
    });

    try {
      // Step 1: classify intent
      const intent = await tracer.startActiveSpan("agent.classify_intent", async (span) => {
        const result = await callLLM({
          feature: "agent-intent-classifier",
          promptVersion: "1.0",
          messages: [
            { role: "system", content: "Classify the user intent as: search | summarize | qa" },
            { role: "user", content: input },
          ],
        });
        span.setAttributes({
          "llm.tokens": result.record.totalTokens,
          "llm.cost_usd": result.record.costUsd,
          "llm.latency_ms": result.record.latencyMs,
        });
        span.end();
        return result.content.trim();
      });

      // Step 2: retrieve context (tool call)
      const context_ = await tracer.startActiveSpan("agent.retrieve_context", async (span) => {
        const docs = await vectorStore.search(input, { topK: 5 });
        span.setAttributes({ "retrieval.doc_count": docs.length });
        span.end();
        return docs;
      });

      // Step 3: generate final answer
      const answer = await tracer.startActiveSpan("agent.generate_answer", async (span) => {
        const result = await callLLM({
          feature: "agent-answer-generation",
          promptVersion: "2.1",
          messages: [
            {
              role: "system",
              content: `Answer based on the retrieved context. Intent: ${intent}`,
            },
            {
              role: "user",
              content: `Context:\n${context_.map((d) => d.text).join("\n\n")}\n\nQuestion: ${input}`,
            },
          ],
        });
        span.setAttributes({
          "llm.tokens": result.record.totalTokens,
          "llm.cost_usd": result.record.costUsd,
        });
        span.end();
        return result.content;
      });

      rootSpan.setStatus({ code: SpanStatusCode.OK });
      return answer;
    } catch (err) {
      rootSpan.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
      throw err;
    } finally {
      rootSpan.end();
    }
  });
}

The key attributes to set on each LLM span: llm.tokens, llm.cost_usd, llm.latency_ms, and llm.prompt_version. With these, a trace visualization shows you the exact cost and latency contribution of each step. When a user reports a slow response, you can open the trace and see that step 2 (retrieval) took 1.2 seconds and step 3 (generation) took 3.8 seconds, and act on the right problem.

---

Dashboard Design for LLM Health

An LLM health dashboard needs five panels, in priority order:

1. Cost by feature (rolling 24h and 7d): The primary cost attribution view. A line chart per feature, annotated with prompt version deploys.
2. Latency P50/P95/P99 by feature: Separate percentiles matter because P95 degrades before P50. Alert thresholds should be on P95.
3. Token distribution (prompt vs. completion): If prompt tokens are growing over time, your context is accumulating state (often a bug). If completion tokens spike, a prompt change caused the model to generate longer outputs.
4. Drift score over time: The output of your eval suite, plotted daily. A flat line at zero is the goal.
5. Rejection/regeneration rate: Per feature, rolling 1h and 24h. This is the user-facing quality signal.

---

Observability Tooling Comparison

Dimension	Langfuse	Helicone	Custom OpenTelemetry
Setup time	Low (SDK + hosted)	Very low (proxy wrapper)	High (instrument manually)
Prompt versioning	Built-in, visual UI	Limited	You build it
Eval framework	Built-in (datasets, scorers)	None	You build it
Cost tracking	Automatic per model	Automatic per model	Manual (compute in code)
Trace visualization	LLM-specific UI	Basic request view	Jaeger/Grafana (generic)
Agent chain tracing	Full span tree	Limited	Full (OpenTelemetry native)
Self-hosting	Yes (open source)	No (SaaS only)	Yes
Privacy / data residency	Controlled if self-hosted	Requests pass through proxy	Full control
Alerting	Webhook-based	Basic	Full (PagerDuty, Grafana)
Cost (hosted)	Free tier + paid	Free tier + paid	Infra cost only

When to use Langfuse: You want prompt versioning, eval datasets, and visual traces without building the infrastructure yourself. The open-source self-hosted option means you keep your data. The best fit for teams that want to move fast without owning LLM observability infrastructure.

When to use Helicone: You want cost tracking and request logging with near-zero setup. The proxy model means you redirect your OpenAI base URL and get logs immediately. The tradeoff is that all requests pass through Helicone’s infrastructure, which is a non-starter for regulated industries or any data that should not leave your environment.

When to use custom OpenTelemetry: You already run a distributed tracing stack (Grafana Tempo, Jaeger, Honeycomb). Adding LLM spans to your existing traces keeps LLM calls inside the same trace context as your database queries and HTTP calls, which is invaluable for debugging multi-service flows where an LLM call is one step. The cost is instrumentation time and the absence of LLM-specific UI features.

The practical answer for most teams: start with Langfuse (self-hosted if data residency matters), then extend to OpenTelemetry spans as your agent chains grow complex enough to need cross-service traces.

---

Production Considerations

Context window growth. Prompt tokens accumulate as you iterate on system messages, add few-shot examples, and inject retrieved context. Set a token budget per feature and add a hard assertion before sending:

function assertTokenBudget(messages: OpenAI.ChatCompletionMessageParam[], maxTokens: number): void {
  // Rough estimate: 4 chars per token
  const estimated = JSON.stringify(messages).length / 4;
  if (estimated > maxTokens) {
    throw new Error(`Prompt exceeds token budget: ~${Math.round(estimated)} > ${maxTokens}`);
  }
}

Idempotency for retries. If you retry a failed LLM call, you will sometimes get two different responses for the same input. This is usually fine, but if you are writing the response to a database (summarization pipelines, async processing), make sure you use an idempotency key tied to the input hash so retries do not create duplicate records.

Prompt injection surface. Any user-controlled input that flows into a prompt is a potential injection vector. Log the raw input alongside the structured record so you can audit for abuse patterns in your traces.

Model version pinning. OpenAI and Anthropic offer dated model aliases (e.g., gpt-4o-2024-11-20). Pin to a dated alias in production so you control when model behavior changes, and run your eval suite before promoting to a new version.

Cost anomaly detection. A runaway loop in an agent chain can generate thousands of tokens per second. Set a per-user hourly token cap enforced before the API call, not just as an alert after the fact.

---

Reliable LLM observability is not a single tool purchase. It is the combination of structured call records with cost attribution, latency budgets that account for model latency in your overall SLOs, eval suites that run on a schedule to detect model drift, and traces that span across the full execution path of your agent chains. The tooling choice matters less than the discipline of tagging every call with feature, version, and user context from the start. Retrofitting that structure into a production system with six undifferentiated LLM calls is significantly harder than building it in from the first call.