Distributed Tracing in Practice: OpenTelemetry, Context Propagation, and Debugging Across Services

Logs are good at telling you something is wrong. They are terrible at telling you why, especially once your system is spread across multiple services, workers, and queues. You see an error in service B, but the root cause is in service A, buried under an async boundary you crossed 400ms ago. Distributed tracing fixes this by giving every request a shared identity that travels with it across every hop.

This article covers how tracing actually works, how to set it up in TypeScript with OpenTelemetry, how to propagate context across HTTP and message queues, how to choose a backend, and how to use traces to debug real problems.

Why Tracing Is Different from Logging and Metrics

Metrics tell you your p99 latency spiked. Logs tell you that a specific request failed with a 500. Traces tell you which service in a chain of six added 800ms of latency, which downstream call returned an unexpected result, and how often that pattern repeats across all requests.

The core abstraction is a trace: a tree of spans, each representing a unit of work in your system. A span has a start time, a duration, a status, and a set of attributes. When service A calls service B, B’s span is a child of A’s span. This parent-child relationship is what makes traces useful for debugging causality.

The problem is maintaining that relationship across service boundaries. Each hop, whether HTTP, gRPC, or a message queue, requires explicit work to carry the trace context forward.

OpenTelemetry: One SDK to Rule the Instrumentation Layer

OpenTelemetry is the standard. It replaced both OpenTracing and OpenCensus and is now the CNCF project that most vendors support. The core insight behind it is separating instrumentation from export: your code emits trace data into the SDK, and the SDK ships it wherever you configure.

The TypeScript SDK (@opentelemetry/sdk-node) handles initialization, automatic instrumentation for common libraries, and batched export.

Setting Up OpenTelemetry in a TypeScript Service

Install the core packages:

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions

Create an instrumentation bootstrap file that runs before anything else:

// src/instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';

const exporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318/v1/traces',
  headers: {
    'x-honeycomb-team': process.env.HONEYCOMB_API_KEY ?? '',
  },
});

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: process.env.SERVICE_NAME ?? 'unknown-service',
    [SEMRESATTRS_SERVICE_VERSION]: process.env.SERVICE_VERSION ?? '0.0.0',
  }),
  spanProcessor: new BatchSpanProcessor(exporter),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false }, // too noisy
      '@opentelemetry/instrumentation-http': { enabled: true },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-pg': { enabled: true },
    }),
  ],
});

sdk.start();

process.on('SIGTERM', () => {
  sdk.shutdown().then(() => process.exit(0));
});

Load it before your app entry point. In Node.js, use --require:

node --require ./dist/instrumentation.js dist/index.js

Or in your package.json:

{
  "scripts": {
    "start": "node --require ./dist/instrumentation.js dist/index.js"
  }
}

Creating Manual Spans

Auto-instrumentation handles HTTP and DB calls. For your own business logic, create spans manually:

import { trace, context, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service', '1.0.0');

async function processOrder(orderId: string): Promise<void> {
  const span = tracer.startSpan('processOrder', {
    attributes: {
      'order.id': orderId,
      'order.source': 'api',
    },
  });

  // Set the span as active so child spans can find their parent
  return context.with(trace.setSpan(context.active(), span), async () => {
    try {
      await validateInventory(orderId);
      await chargePayment(orderId);
      await fulfillOrder(orderId);
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

async function validateInventory(orderId: string): Promise<void> {
  // This span is a child of processOrder because of context.with above
  const span = tracer.startSpan('validateInventory', {
    attributes: { 'order.id': orderId },
  });

  try {
    // ... actual logic
    span.setStatus({ code: SpanStatusCode.OK });
  } finally {
    span.end();
  }
}

Context Propagation: The Hard Part

Setting up tracing inside one service is straightforward. Getting trace context to survive a hop to another service is where most teams run into trouble.

Propagation Over HTTP

When service A makes an HTTP request to service B, the trace context travels in HTTP headers. The W3C Trace Context specification defines traceparent and tracestate as the standard headers. OpenTelemetry uses this format by default.

Auto-instrumentation with @opentelemetry/instrumentation-http handles this automatically for outgoing http and https requests. It injects the current span context into the headers. On the receiving side, it extracts the headers and creates a child span with the correct parent.

For fetch or axios, add explicit instrumentation:

import { context, propagation } from '@opentelemetry/api';

// Outgoing request with manual header injection
async function callDownstreamService(url: string, payload: unknown): Promise<Response> {
  const headers: Record<string, string> = {
    'content-type': 'application/json',
  };

  // Inject current trace context into headers
  propagation.inject(context.active(), headers);

  return fetch(url, {
    method: 'POST',
    headers,
    body: JSON.stringify(payload),
  });
}

On the receiving side, extract context from incoming headers:

import { propagation, context, trace } from '@opentelemetry/api';

// Express middleware to extract trace context
function tracingMiddleware(req: Request, res: Response, next: NextFunction): void {
  const extractedContext = propagation.extract(context.active(), req.headers);

  context.with(extractedContext, () => {
    const span = tracer.startSpan(`${req.method} ${req.path}`, {
      attributes: {
        'http.method': req.method,
        'http.url': req.url,
      },
    });

    context.with(trace.setSpan(extractedContext, span), () => {
      res.on('finish', () => {
        span.setAttribute('http.status_code', res.statusCode);
        span.end();
      });
      next();
    });
  });
}

Propagation Over Message Queues

Message queues break the synchronous call chain. There is no request-response cycle, so you have to store the trace context as a message attribute and extract it on the consumer side.

import { context, propagation, trace, SpanKind } from '@opentelemetry/api';

// Producer: inject context into message attributes
async function publishOrderEvent(orderId: string): Promise<void> {
  const messageAttributes: Record<string, string> = {};
  propagation.inject(context.active(), messageAttributes);

  await sqsClient.send(new SendMessageCommand({
    QueueUrl: process.env.ORDER_QUEUE_URL,
    MessageBody: JSON.stringify({ orderId }),
    MessageAttributes: Object.fromEntries(
      Object.entries(messageAttributes).map(([key, value]) => [
        key,
        { DataType: 'String', StringValue: value },
      ])
    ),
  }));
}

// Consumer: extract context and create a linked span
async function handleOrderMessage(message: SQSMessage): Promise<void> {
  const carrier = Object.fromEntries(
    Object.entries(message.MessageAttributes ?? {}).map(([key, attr]) => [
      key,
      attr.StringValue ?? '',
    ])
  );

  const parentContext = propagation.extract(context.active(), carrier);

  // Use SpanKind.CONSUMER for message queue processing
  const span = tracer.startSpan('handleOrderMessage', {
    kind: SpanKind.CONSUMER,
    attributes: {
      'messaging.system': 'sqs',
      'messaging.destination': 'order-queue',
      'messaging.message_id': message.MessageId,
    },
  }, parentContext);

  return context.with(trace.setSpan(parentContext, span), async () => {
    try {
      const body = JSON.parse(message.Body ?? '{}');
      await processOrder(body.orderId);
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

The key difference from HTTP propagation: you use SpanKind.CONSUMER on the receiving end, which tells the backend that this span represents async message processing rather than an inbound HTTP call. The trace is still connected to the producer’s trace through the parent context.

Propagation in Serverless Environments

In Lambda or similar environments, the propagation pattern is the same, but the SDK initialization is different. You cannot use BatchSpanProcessor because the process exits after each invocation. Use SimpleSpanProcessor instead, which exports synchronously:

import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';

// In Lambda: flush synchronously before the handler returns
const sdk = new NodeSDK({
  spanProcessor: new SimpleSpanProcessor(exporter),
  // ...
});

Also ensure your handler awaits shutdown before returning:

export const handler = async (event: APIGatewayEvent): Promise<APIGatewayProxyResult> => {
  const span = tracer.startSpan('handler', { attributes: { 'faas.trigger': 'http' } });

  try {
    return context.with(trace.setSpan(context.active(), span), async () => {
      const result = await processEvent(event);
      span.end();
      await sdk.shutdown(); // flush before Lambda freezes the process
      return result;
    });
  } catch (err) {
    span.recordException(err as Error);
    span.end();
    await sdk.shutdown();
    throw err;
  }
};

Choosing a Backend

The OTLP protocol (OpenTelemetry’s native wire format) decouples your instrumentation from your backend. You can switch backends without changing application code.

Backend	Hosting	Cost model	Best for
Jaeger	Self-hosted	Infrastructure cost only	Teams that want control and have ops capacity
Zipkin	Self-hosted	Infrastructure cost only	Simpler setup, smaller teams
Tempo (Grafana)	Self-hosted or managed	Storage-based	Teams already on Grafana stack
Cloud-native (X-Ray, Cloud Trace)	Managed	Per-span ingestion	Teams fully committed to one cloud
Honeycomb	SaaS	Events-based	High-cardinality querying, fast iteration
Datadog APM	SaaS	Host-based + volumes	Teams already paying for Datadog

Jaeger is a solid default for self-hosted. It has a mature UI, supports OTLP natively since v1.35, and runs well on Kubernetes with Cassandra or Elasticsearch as storage.

Tempo makes sense if you are already running Grafana and Loki. It stores traces efficiently using object storage and integrates directly with Grafana dashboards.

SaaS options reduce operational burden at the cost of per-event pricing, which can get expensive at high volume without aggressive sampling.

The deployment choice matters less than getting your sampling strategy right.

Sampling Strategies

Tracing every request is expensive. A busy service handling 10k RPS generates enormous volumes of trace data. Sampling controls how much of that you actually record.

Head-Based Sampling

The decision to sample is made at the root span, before processing completes. Simple and cheap, but you cannot make decisions based on what happened downstream.

import { ParentBasedSampler, TraceIdRatioBased } from '@opentelemetry/sdk-trace-base';

// Sample 10% of new traces, always follow parent decision
const sampler = new ParentBasedSampler({
  root: new TraceIdRatioBased(0.1),
});

const sdk = new NodeSDK({
  sampler,
  // ...
});

Tail-Based Sampling

The decision is made after the full trace is assembled. This lets you always sample errors, slow traces, and traces involving specific attributes, while dropping routine healthy traces.

Tail sampling requires a collector that can buffer spans and make decisions after the fact. The OpenTelemetry Collector supports this with the tail_sampling processor:

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces-policy
        type: latency
        latency: { threshold_ms: 1000 }
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

This configuration always keeps error traces and traces slower than 1 second. Everything else is sampled at 5%. This is a common starting point for production systems.

Adaptive Sampling

Some commercial backends offer adaptive sampling that adjusts rates automatically based on traffic volume and error rates. This reduces configuration overhead but gives up explicit control.

The practical recommendation: start with head-based sampling at 10%, add tail-based sampling once you have enough traffic that storage costs become meaningful, and always sample 100% of errors.

Debugging Workflows

A trace is most useful when you know what question to ask.

Latency Spikes

When p99 latency rises, start with a waterfall view of a slow trace. Look for:

• Spans that are wide but have no children (work happening that is not broken down)
• Sequential spans that could be parallelized
• Database spans with high row counts (N+1 query pattern)
• Gaps between spans (time that is unaccounted for in the trace)

Add attributes to your spans to answer follow-up questions without re-investigating:

span.setAttributes({
  'db.rows_returned': result.rowCount,
  'cache.hit': cacheResult !== null,
  'queue.depth': queueDepth,
  'retry.count': retryCount,
});

Error Attribution

When a request fails, the trace shows you which service threw the error, whether it propagated up from a downstream call, and whether the error was retried before reaching the user.

Set structured error information on spans:

try {
  await downstreamCall();
} catch (err) {
  const error = err as Error;
  span.recordException(error); // captures stack trace, message, type
  span.setStatus({
    code: SpanStatusCode.ERROR,
    message: error.message,
  });
  span.setAttribute('error.type', error.constructor.name);
  throw err;
}

This gives you enough information in the trace to distinguish between a timeout in your own code, a 429 from a downstream API, and a connection reset from a database.

Service Dependency Analysis

Most tracing backends let you view service dependency graphs: which services call which other services, and what the error rate and latency distribution looks like on each edge. This view is useful for finding hidden dependencies and for understanding what breaks when a specific service degrades.

Generate this view by aggregating span data, not by reading your architecture diagrams. What you think your architecture looks like and what it actually looks like in production often diverge.

Production Considerations

Span cardinality. Attributes with unbounded cardinality (user IDs, request IDs, full URLs with query parameters) can cause storage backends to blow up. Group high-cardinality values or avoid them as span attributes entirely. Use a fixed set of known attribute values where possible.

Instrumentation overhead. OpenTelemetry’s SDK adds latency for every span creation and export. In benchmarks, this is typically under 1ms per span for the critical path, but batch export buffers matter. Test your p99 latency before and after enabling instrumentation under load.

Propagation gaps. If any service in your chain does not propagate trace context, the trace breaks at that point. Audit all service-to-service communication paths, including background jobs, scheduled tasks, and webhooks. Missing context is usually silent: you just see disconnected traces instead of an error.

Clock skew. In distributed systems, clocks across machines are not perfectly synchronized. Some backends handle this gracefully by using the parent span’s clock as a reference. If you see spans that appear to start before their parent, this is usually clock skew, not a bug in your code.

Sensitive data. Span attributes end up in your tracing backend, which may be a third-party SaaS. Never put PII, credentials, or payment data in span attributes. Scrub or truncate values that might contain sensitive information before recording them.

Distributed tracing is not a silver bullet for understanding production systems. It works best when combined with structured logging (which gives you detail inside a single request) and metrics (which give you aggregate signals across all requests). The value is in the connections: seeing how a user-visible failure traces back through your services to its root cause, without reconstructing the story from scattered log lines across five different services.

When tracing is set up well, debugging a latency spike that would have taken hours becomes a ten-minute exercise of finding the slow trace and reading the waterfall.