OpenTelemetry Collector Architecture: Pipeline Design, Processors, and Scaling Telemetry Ingestion in Production

Most teams add the OpenTelemetry Collector because the getting-started docs told them to. Few treat it as a production component with its own capacity model, failure modes, and cost implications. That gap is where observability spending spirals and on-call alerts get noisy.

This article covers the Collector as a real infrastructure piece: how its internal pipeline model works, which processors actually matter in production, how to deploy it across a cluster without creating a bottleneck, and how a well-configured processor chain can cut your observability backend bill by 40-60% before a single byte leaves your datacenter.

Why You Need a Collector at All

The naive path is to configure your instrumented application to export directly to your backend (Datadog, Honeycomb, Jaeger, whatever). This works until it does not.

Direct export couples your application’s shutdown behavior to backend availability. If the backend is slow or unreachable, your app thread blocks on export. You also give up any ability to process telemetry data before it lands: no sampling, no field enrichment, no cost filtering. Every noisy health-check trace and high-cardinality debug log ships at full rate.

The Collector sits between your application and your backend. It receives telemetry over OTLP (or a dozen other protocols), applies a pipeline of processors, and exports to one or more backends. The application’s export path stays local and fast. The Collector handles backpressure, batching, retry, and all the processing you would otherwise need to push into every service.

Core Architecture: Receivers, Processors, Exporters

The Collector’s internal model is straightforward. A receiver accepts telemetry data from some source. A processor transforms or filters data in a pipeline. An exporter sends data to a destination. Pipelines connect them:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, tail_sampling]
      exporters: [otlphttp/honeycomb]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch, filter/drop_debug]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp, fluentforward]
      processors: [memory_limiter, batch, attributes/add_env]
      exporters: [loki]

Each pipeline type (traces, metrics, logs) is independent. You can have multiple pipelines of the same type with different processor chains. The memory_limiter should always be first in every processor chain, before anything that could accumulate data.

Receiver Configuration

OTLP Receiver

OTLP is the native OpenTelemetry protocol. Configure both gRPC and HTTP endpoints:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 4
        keepalive:
          server_parameters:
            max_connection_idle: 11s
            max_connection_age: 12s
            time: 30s
            timeout: 5s
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - "https://*.yourdomain.com"

Set max_recv_msg_size_mib explicitly. The default is 4 MiB which is fine for most spans, but some SDKs batch aggressively and you will see truncation errors if you exceed it without knowing why.

Prometheus Receiver

For scraping existing Prometheus endpoints, including your own applications and infrastructure components:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: kubernetes-pods
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: "true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: (.+)
              replacement: $1
          scrape_interval: 30s
          scrape_timeout: 10s

The Prometheus receiver runs a full Prometheus scraper internally. That means it holds state (last scrape timestamp, counter resets). If you run multiple Collector replicas with Prometheus receivers pointed at the same targets, you will double-scrape and produce incorrect rate calculations. Use a single Collector instance or a dedicated Prometheus for scraping, then forward via remote write.

Fluentd / FluentForward Receiver

Many teams have existing Fluentd or Fluent Bit agents writing logs. The fluentforward receiver speaks the Fluent Forward protocol:

receivers:
  fluentforward:
    endpoint: 0.0.0.0:24224

This lets you migrate logs to OpenTelemetry incrementally without replacing all your log shippers at once.

Processors That Matter in Production

Memory Limiter: Always First

The memory limiter is your circuit breaker. Without it, a spike in inbound telemetry volume will OOM the Collector process:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 400

When memory usage exceeds limit_mib - spike_limit_mib (1100 MiB in this example), the processor starts refusing new data and returns backpressure signals to receivers. When it exceeds limit_mib, it forces garbage collection. Set limit_mib to about 80% of your container memory limit. Set spike_limit_mib to about 25% of limit_mib. The check happens on check_interval, so there is a window where memory can exceed the limit; that is expected behavior.

Batch Processor: Throughput vs Latency

The batch processor accumulates spans/metrics/logs and sends them downstream in larger chunks. This is essential for throughput: your exporter makes fewer HTTP/gRPC calls, your backend receives data in more efficient sizes:

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
    send_batch_max_size: 2000

The batch processor sends when either send_batch_size items are accumulated OR timeout elapses, whichever comes first. The tradeoff: larger batches and longer timeouts improve throughput and reduce per-call overhead, but increase tail latency for traces. A span emitted at second 0 may not reach your backend until second 5.

For traces this is usually acceptable, trace backends are not real-time dashboards. For metrics and logs that feed alerting, keep timeout lower (1-2s). Configure separate batch processors per pipeline if your latency requirements differ:

processors:
  batch/traces:
    timeout: 5s
    send_batch_size: 1000
  batch/metrics:
    timeout: 1s
    send_batch_size: 500
  batch/logs:
    timeout: 2s
    send_batch_size: 500

Filter Processor: Drop Before You Pay

The filter processor is where you claw back observability costs. Telemetry backends typically charge per volume ingested. If 60% of your traces are health check endpoints returning 200, you are paying to store noise.

processors:
  filter/drop_health_checks:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/health"'
        - 'attributes["http.route"] == "/ready"'
        - 'attributes["http.route"] == "/metrics"'
    metrics:
      metric:
        - 'name == "go_gc_duration_seconds"'
        - 'name == "process_cpu_seconds_total" and resource.attributes["service.name"] == "internal-cron"'
    logs:
      log_record:
        - 'severity_number < SEVERITY_NUMBER_WARN and body contains "DEBUG"'

The error_mode: ignore setting means if an OTTL expression fails to evaluate (malformed attribute, type mismatch), the item passes through rather than crashing. Use propagate in development to catch expression errors.

Filter processors using OTTL (OpenTelemetry Transformation Language) are evaluated per span/metric/log record. At high volume this adds CPU cost. Profile your pipeline under load before deploying aggressive filter rules.

Attributes Processor: Enrichment and Redaction

The attributes processor adds, removes, renames, or hashes span attributes:

processors:
  attributes/enrich:
    actions:
      - key: deployment.environment
        value: production
        action: insert
      - key: k8s.cluster.name
        value: ${K8S_CLUSTER_NAME}
        action: insert
      - key: user.email
        from_attribute: http.request.header.x-user-email
        action: extract
      - key: http.request.header.x-user-email
        action: delete

  attributes/redact_pii:
    actions:
      - key: db.statement
        action: hash
      - key: http.request.body
        action: delete

The extract action copies a value from one attribute to another. The hash action replaces a value with its SHA-1 hash, useful for PII fields where you need correlation (same user, different requests) without storing raw values.

Environment variables are resolved at startup (${K8S_CLUSTER_NAME}). In Kubernetes, inject these via the Downward API in your DaemonSet spec.

Tail Sampling: Keep the Interesting Traces

Head-based sampling (deciding at trace start whether to sample) is simple but blind. You will drop slow requests and errors at the same rate as fast, successful ones.

Tail sampling buffers complete traces and makes the sampling decision after all spans have arrived. This lets you keep 100% of errors, 100% of slow requests, and sample down the boring successful fast requests:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces-policy
        type: latency
        latency:
          threshold_ms: 1000
      - name: sample-everything-else
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

The decision_wait is how long to wait for all spans of a trace before making a sampling decision. Distributed traces can have spans arriving from different services at different times. 10 seconds is conservative but safe for most deployments.

num_traces is the in-memory buffer size. At expected_new_traces_per_sec: 1000 and decision_wait: 10s, you need at least 10,000 trace slots. Set num_traces higher (50,000) to absorb bursts.

Tail sampling requires all spans for a single trace to arrive at the same Collector instance. This has significant implications for horizontal scaling, covered in the next section.

Deployment Patterns

Pattern	Topology	Pros	Cons	When to Use
Agent (DaemonSet)	One Collector per node	Low network hop, simple auth, resource isolation per node	No tail sampling across nodes, more total CPU	Default for metrics and logs
Agent (Sidecar)	One Collector per pod	Per-service config, strongest isolation	Resource overhead per pod, complex management	Compliance-sensitive services
Gateway	Centralized pool behind load balancer	Tail sampling possible, single config, batching at scale	Network hop, SPOF if misconfigured, stateful trace buffer	Traces with tail sampling
Two-tier	DaemonSet agents + Gateway	Best of both: local buffering + centralized processing	Most complex, two configs to maintain	Large clusters with tail sampling requirement

For most Kubernetes deployments the two-tier pattern is the right default once you hit enough scale for tail sampling to matter. Agents on each node receive OTLP from local pods and forward traces to the Gateway cluster. The Gateway holds the tail sampling state. Metrics and logs can be processed at the agent tier and exported directly to backends without hitting the gateway.

Agent DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector-agent
spec:
  selector:
    matchLabels:
      app: otel-collector-agent
  template:
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.96.0
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol-contrib/config.yaml
              subPath: config.yaml
      volumes:
        - name: config
          configMap:
            name: otel-collector-agent-config

Gateway Deployment with Sticky Load Balancing

Tail sampling requires all spans for a trace to reach the same Collector replica. Standard round-robin load balancing breaks this. Use the loadbalancingexporter in your agent tier to route traces by trace ID to a consistent gateway replica:

exporters:
  loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
        timeout: 5s
    resolver:
      k8s:
        service: otel-collector-gateway
        ports:
          - 4317

The load balancing exporter hashes the trace ID and consistently routes to the same backend endpoint. When you scale the gateway Deployment up or down, the consistent hash rebalances, and some in-flight traces will be split across replicas during the transition window. This causes those traces to be sampled incorrectly once. For production this is acceptable. Avoid scaling the gateway during peak traffic.

Scaling and Backpressure

The Collector uses a bounded queue between the processor pipeline and the exporter. Configure it explicitly:

exporters:
  otlphttp/honeycomb:
    endpoint: https://api.honeycomb.io
    headers:
      x-honeycomb-team: ${HONEYCOMB_API_KEY}
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

num_consumers controls how many goroutines drain the queue and make export calls. Increase this if your backend supports high concurrency and your bottleneck is outbound throughput. queue_size is the in-memory queue depth. If this fills up (backend is slow, network is degraded), the Collector will start dropping data. Size it to absorb the expected lag at your backend’s worst-case latency.

Monitor otelcol_exporter_queue_size and otelcol_exporter_send_failed_spans metrics. The queue size metric tells you how close you are to the limit; failed spans tells you you have already dropped data.

For the ingestion side, the memory limiter provides backpressure to receivers. Receivers propagate this back to clients via gRPC status codes (RESOURCE_EXHAUSTED). Well-behaved SDKs will back off and retry. Not all SDKs handle this correctly, so test your SDK’s behavior under memory pressure before relying on it.

Cost Reduction via Processor Pipelines

A typical production service generates spans for every HTTP request, including health checks, metrics scrapes, and internal heartbeats. Add high-frequency database queries, cache lookups, and internal gRPC calls and you have a high volume of low-value traces.

A processor pipeline that filters health check spans, applies 5% tail sampling on successful fast requests, drops debug-level logs, and removes high-cardinality attributes (user IDs, session tokens) before shipping can realistically reduce your ingested telemetry volume by 40-60%. The math:

• Health check and readiness probe spans (called every 10-30 seconds per pod): eliminated entirely
• Successful, fast (under 200ms) requests: 5% sample rate, 95% reduction
• DEBUG and INFO logs below WARN threshold: eliminated based on environment
• High-cardinality label dimensions on metrics: reduced via attribute filtering

The Collector processes all of this before any byte leaves your cluster. You pay for Collector compute (cheap: a few vCPUs and GiBs of RAM across a DaemonSet) and save on backend ingestion costs (which scale with volume and often include per-GB pricing).

Production Checklist

Before promoting a Collector configuration to production:

Memory and resource limits

• memory_limiter is the first processor in every pipeline
• Container memory limit set, limit_mib is 80% of it, spike_limit_mib is 25% of limit_mib
• CPU requests and limits set on all Collector pods; tail sampling and filter processors are CPU-intensive

Pipeline correctness

• batch processor appears after memory_limiter and before exporters in every pipeline
• Filter OTTL expressions tested with sample payloads using otelcol validate --config
• Attributes processor delete actions confirmed not removing fields needed downstream

Tail sampling (if used)

• loadbalancingexporter configured in agent tier with trace-ID routing
• decision_wait long enough for your slowest services to emit their spans
• num_traces sized to decision_wait * expected_new_traces_per_sec * 1.5 headroom
• Gateway Deployment has pod disruption budget; avoid rolling restarts during peak traffic

Exporter reliability

• sending_queue enabled with explicit queue_size
• retry_on_failure enabled with sensible max_elapsed_time (5 minutes is reasonable)
• Exporter API keys injected from secrets, not hardcoded in ConfigMap

Observability of the Collector itself

• Collector’s own metrics scraped (exposed on port 8888 by default)
• Alerts on otelcol_exporter_queue_size approaching queue_size limit
• Alerts on otelcol_exporter_send_failed_spans greater than zero sustained for over 1 minute
• Alerts on otelcol_processor_dropped_spans to detect overly aggressive filter rules

Configuration management

• Collector config stored in version control alongside application configuration
• ConfigMap changes trigger Collector pod rollout (use a hash annotation or a config watcher sidecar)
• Separate configs for agent tier and gateway tier; do not share a single ConfigMap between both

The Collector is not a set-and-forget component. As your service count grows, your traffic patterns change, and your backends evolve, the pipeline configuration needs to evolve with them. Treating it as infrastructure with the same rigor as your database or message queue is the difference between an observability platform that scales with you and one that becomes a reliability problem.