Designing a Logging and Log Aggregation Pipeline: Collection, Transport, Storage, and Search at Scale

Most logging systems work fine until they don’t. The failure mode is predictable: you start with console.log to stdout, a sidecar scrapes it, everything lands in Elasticsearch. Then volume grows, query latency climbs into seconds, index costs start showing up on the cloud bill, and someone gets paged at 2am because the ingest pipeline backed up and you lost thirty minutes of logs.

The problem is not any individual tool. It is that each component in a logging pipeline makes tradeoffs, and those tradeoffs compound across the full stack. This article walks through each layer of a production logging pipeline, the concrete configuration choices that matter, and the tradeoffs you actually have to make.

---

The Pipeline Shape

Before getting into components, it helps to have the shape in mind:

Application
    |
    v
[Log emission: structured JSON to stdout]
    |
    v
[Collection: Fluent Bit / Vector on each node]
    |
    v
[Transport: Kafka (async, buffered) or direct ship]
    |
    v
[Indexing / Storage: Elasticsearch / ClickHouse / Loki]
    |
    v
[Query layer: Kibana / Grafana / custom API]

Every arrow in that diagram is a place where logs can be dropped, delayed, or duplicated. Designing for that reality means making explicit decisions at each stage rather than accepting defaults.

---

Structured Log Emission

The most important decision you make about logging happens before any pipeline component is involved: what shape are your log records?

Unstructured logs ("User 1234 logged in at 14:32:01") require regex parsing downstream to be queryable. That parsing is fragile, slow, and needs to be maintained every time the message format changes. Structured logging emits JSON from the start.

interface LogRecord {
  timestamp: string;        // ISO 8601, UTC
  level: "debug" | "info" | "warn" | "error" | "fatal";
  service: string;
  version: string;
  traceId: string;
  spanId: string;
  message: string;
  context: Record<string, unknown>;
  error?: {
    type: string;
    message: string;
    stack: string;
  };
}

function createLogger(service: string, version: string) {
  return {
    info(message: string, context: Record<string, unknown> = {}) {
      const record: LogRecord = {
        timestamp: new Date().toISOString(),
        level: "info",
        service,
        version,
        traceId: getTraceId(),
        spanId: getSpanId(),
        message,
        context,
      };
      process.stdout.write(JSON.stringify(record) + "\n");
    },
    error(message: string, err: unknown, context: Record<string, unknown> = {}) {
      const record: LogRecord = {
        timestamp: new Date().toISOString(),
        level: "error",
        service,
        version,
        traceId: getTraceId(),
        spanId: getSpanId(),
        message,
        context,
        error: err instanceof Error ? {
          type: err.constructor.name,
          message: err.message,
          stack: err.stack ?? "",
        } : { type: "unknown", message: String(err), stack: "" },
      };
      process.stdout.write(JSON.stringify(record) + "\n");
    },
  };
}

Two details here that matter in production: always write to stdout (not a file), and always terminate with a newline. Collectors assume line-delimited JSON. Multiline logs (stack traces, for example) require special handling at the collection layer if they are not flattened first.

---

Collection: Fluent Bit, Fluentd, and Vector

Once logs are on stdout, something needs to collect them from each node, parse them if needed, and forward them downstream. The three serious options are Fluent Bit, Fluentd, and Vector.

Fluent Bit is written in C, has a minimal memory footprint (roughly 650KB resident with basic config), and is the standard sidecar/DaemonSet choice for Kubernetes environments. It handles parsing, filtering, and routing but has a more limited plugin ecosystem than Fluentd.

Fluentd is written in Ruby, heavier (roughly 40-60MB idle), but has hundreds of community plugins and handles complex routing logic well. It sits between Fluent Bit and a downstream store in many architectures: Fluent Bit collects from nodes, ships to Fluentd for enrichment and routing, Fluentd writes to storage.

Vector is written in Rust, designed as a full observability pipeline (logs, metrics, traces), and has consistently lower latency and memory than either Fluent option. Its configuration model is more expressive than Fluent Bit but it has less operational history in large Kubernetes deployments.

A minimal Fluent Bit config for a Kubernetes DaemonSet collecting from container stdout:

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Name        kafka
    Match       *
    Brokers     kafka-broker-0:9092,kafka-broker-1:9092,kafka-broker-2:9092
    Topics      logs.raw
    rdkafka.request.required.acks -1
    rdkafka.message.max.bytes     1000000

The Mem_Buf_Limit setting is critical. Without it, if the output is slow (Kafka is catching up, network is saturated), Fluent Bit will buffer indefinitely in memory until the node OOMs. Set it to a value your DaemonSet can afford given node memory constraints.

---

Transport: Kafka vs Direct Shipping

Whether you put Kafka between collectors and storage is a real architectural decision, not a default choice.

Direct shipping (Fluent Bit writes directly to Elasticsearch or Loki) is operationally simpler and has lower end-to-end latency. It is the right choice for smaller deployments where log volume is predictable and you can afford occasional lag during storage upgrades or index rotation.

Kafka in the middle decouples producers from consumers. Collectors write to Kafka regardless of what is happening downstream. Storage can be upgraded, replaced, or scaled without losing data. You can add consumers (a second storage tier, a real-time alerting system) without touching collectors. The cost is operational: Kafka is a real infrastructure component that needs monitoring, tuning, and maintenance.

The crossover point is roughly 50GB/day of log volume, or any time you need more than one downstream consumer reading the same log stream.

When using Kafka, topic partitioning matters. A single topic with eight partitions allows eight parallel consumer threads writing to storage. Partition by a hash of service if you want per-service ordering guarantees, or by a round-robin key if you just want parallelism. Log pipelines almost never need strict global ordering, so round-robin is usually correct.

// Consumer group reading from Kafka, writing to Elasticsearch
import { Kafka, EachMessagePayload } from "kafkajs";

const kafka = new Kafka({
  clientId: "log-consumer",
  brokers: ["kafka-0:9092", "kafka-1:9092", "kafka-2:9092"],
});

const consumer = kafka.consumer({ groupId: "elasticsearch-sink" });

async function run() {
  await consumer.connect();
  await consumer.subscribe({ topic: "logs.raw", fromBeginning: false });

  const batch: LogRecord[] = [];

  await consumer.run({
    eachMessage: async ({ message }: EachMessagePayload) => {
      if (!message.value) return;
      try {
        const record = JSON.parse(message.value.toString()) as LogRecord;
        batch.push(record);
        if (batch.length >= 500) {
          await flushToElasticsearch(batch.splice(0));
        }
      } catch {
        // malformed records: write to dead-letter topic, do not throw
      }
    },
  });
}

Batching at the consumer is important. Elasticsearch and ClickHouse both perform far better with bulk writes (hundreds to thousands of records per request) than with individual document inserts.

---

Storage Engines: Elasticsearch, ClickHouse, and Loki

The three serious options for log storage have genuinely different strengths.

Elasticsearch / OpenSearch is the most full-featured for search. Full-text search, field-level queries, aggregations, and the Kibana UI are all first-class. The costs are also real: index mappings consume memory proportional to field cardinality, shard counts matter for query parallelism, and storage is expensive because every field is indexed by default. At high volume, inverted index maintenance creates significant write amplification.

ClickHouse is a column-oriented OLAP database that was not designed specifically for logs but handles them well at scale. Storage costs are 5-10x lower than Elasticsearch for the same data volume due to column compression. Queries over large time ranges are fast because ClickHouse skips granules (blocks of 8192 rows) that do not match the WHERE clause. The tradeoff is that full-text search is slower than Elasticsearch, and the ecosystem tooling (UI, alerting) is less mature.

Grafana Loki takes a different position: it indexes only labels (metadata fields like service, level, namespace) and stores the raw log lines as compressed chunks in object storage (S3, GCS). Query cost is low because storage is cheap and you only index what you need to filter on. The limitation is that searching within log content requires scanning compressed chunks, which is slow for high-volume unfiltered queries.

Dimension	Elasticsearch	ClickHouse	Loki
Full-text search	Excellent	Acceptable (tokenbf_v1)	Slow (chunk scan)
Storage cost	High	Low	Very low
Write throughput	Moderate	High	High
Query flexibility	High (DSL)	High (SQL)	Limited (LogQL)
Operational complexity	High	Medium	Low (object storage)
Best for	Search-heavy, ad-hoc queries	Analytics, long-term retention	Cost-sensitive, label-filtered lookups

---

Indexing Strategy

For Elasticsearch, the default dynamic mapping indexes every field, which is fine for development and expensive in production. Explicit mappings reduce memory usage and prevent cardinality explosions from high-cardinality fields like userId or requestId.

{
  "mappings": {
    "dynamic": "false",
    "properties": {
      "timestamp":  { "type": "date" },
      "level":      { "type": "keyword" },
      "service":    { "type": "keyword" },
      "version":    { "type": "keyword" },
      "traceId":    { "type": "keyword", "index": false },
      "spanId":     { "type": "keyword", "index": false },
      "message":    { "type": "text", "norms": false },
      "context":    { "type": "object", "dynamic": false }
    }
  }
}

Setting "index": false on traceId and spanId means they are stored but not indexed. You can retrieve them in results, but you cannot filter by them directly. If you need to look up logs by trace ID, add it to the indexed fields. Otherwise, save the memory.

For ClickHouse, the equivalent optimization is choosing the right ORDER BY key and using skip indexes. A table ordered by (service, toStartOfHour(timestamp), level) allows Elasticsearch-style filtered queries (service + time range + level) to skip large chunks of data:

CREATE TABLE logs (
  timestamp     DateTime64(3, 'UTC'),
  level         LowCardinality(String),
  service       LowCardinality(String),
  version       String,
  traceId       String,
  message       String,
  context       String  -- JSON stored as string
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (service, toStartOfHour(timestamp), level)
TTL timestamp + INTERVAL 90 DAY DELETE
SETTINGS index_granularity = 8192;

LowCardinality(String) for fields like level and service applies dictionary encoding automatically, reducing storage and improving scan performance. For context, storing as a JSON string and using JSONExtract* functions at query time is cheaper than nested types for infrequently queried fields.

---

Retention Policies and Cost Tradeoffs

Hot-warm-cold tiering is the standard approach to retention cost.

For Elasticsearch, Index Lifecycle Management (ILM) automates transitions. A typical policy: hot tier (fast SSDs, full replicas) for the last 7 days; warm tier (slower disks, one replica) for days 8-30; cold tier (object storage snapshots) for days 31-90; delete at 91 days.

For ClickHouse, the TTL clause handles deletion automatically. Tiering to cheaper storage uses the MOVE TO VOLUME syntax with tiered storage policies.

For Loki, data lives in object storage from day one, so there is no tiering decision. Retention is configured per-stream with a global default and label-based overrides.

The non-obvious cost driver is not storage but query compute. Running a 30-day aggregation query over an unpartitioned Elasticsearch index will use as much resource as ten days of ingestion. Partition aggressively (daily partitions, not weekly), and set index aliases that automatically route queries to the relevant partition range.

---

Query Patterns

For incident response, queries are narrow in time and wide in fields. A typical pattern: filter by service and a 15-minute window, then group errors by context.statusCode. Both Elasticsearch and ClickHouse handle this fast when the time range is indexed.

For debugging a specific request, you need trace-correlated log lookup: all logs where traceId = "abc123". This requires traceId to be indexed (or to use a Loki label). Without indexing, you are doing a full-scan over the time window, which is slow past a few hours of data.

For capacity planning and trend analysis, you need aggregations over long time ranges. ClickHouse outperforms Elasticsearch here significantly because column scans skip non-matching granules and compressed storage means less IO.

A useful ClickHouse query for error rate trends:

SELECT
  toStartOfHour(timestamp) AS hour,
  service,
  countIf(level = 'error') AS errors,
  count() AS total,
  round(countIf(level = 'error') / count() * 100, 2) AS error_rate_pct
FROM logs
WHERE timestamp >= now() - INTERVAL 7 DAY
  AND service IN ('api', 'worker', 'scheduler')
GROUP BY hour, service
ORDER BY hour DESC, service;

---

Production Considerations

Backpressure and data loss. Every buffer in the pipeline (Fluent Bit’s Mem_Buf_Limit, Kafka’s retention, the consumer batch size) defines a window in which data can be lost if something fails. Know what that window is and make it explicit. For most teams, losing a few minutes of non-critical logs during an incident is acceptable; losing logs from the incident itself is not. Kafka with a retention of at least 24 hours gives you a replay window for debugging consumer failures.

Schema evolution. Log schemas change as services evolve. Elasticsearch with dynamic mapping silently breaks when a field changes type (e.g., a field that was keyword becomes an object). Use strict mappings and version your index templates. ClickHouse requires ALTER TABLE to add columns, which can be done online but requires coordination. Loki avoids this entirely by not indexing content fields.

Observability of the pipeline itself. Monitor Fluent Bit’s mem_buf_overflow counter and Kafka consumer group lag. A consumer group lag growing without bound means your downstream write path is slower than ingest. Catch this at hundreds of thousands of messages, not millions.

Sampling for debug logs. Emitting every debug-level log in production is expensive. A tail-based sampling approach: buffer all log levels locally, but only flush debug-level records for traces that contain an error or meet a sampling rate. This requires the collector to buffer by trace ID and make flush decisions on trace completion, which Vector handles natively but requires custom logic with Fluent Bit.

Multi-tenancy and access control. If multiple teams or products share a logging cluster, index-level RBAC (Elasticsearch) or row-level filtering (ClickHouse row policy) prevents one team from reading another team’s logs. Define this at cluster setup, not as an afterthought.

---

Choosing Your Stack

Start here: how much log volume do you generate per day, and what is your primary query pattern?

Under 10GB/day with mostly service-level filtering: Loki with a single Kafka topic and basic Fluent Bit DaemonSets. Operational overhead is low, storage is cheap, and Grafana gives you a working UI.

10-100GB/day with incident-driven search requirements: Elasticsearch with ILM, daily index rollover, explicit mappings for your top 20 fields, and hot-warm tiering. Accept the operational cost.

Over 100GB/day, or if you need long-term retention for analytics or compliance: ClickHouse for primary storage, with a shorter-retention Elasticsearch cluster for recent data if you need full-text search. The storage cost difference at this volume is significant enough to justify running two systems.

In all cases: Kafka in the middle as soon as you have more than one consumer or want replay capability. Direct shipping is simpler but you will regret the coupling the first time you need to migrate storage backends.

A well-designed logging pipeline is invisible when it works. The goal is not to build something clever, it is to ensure that when something breaks in production, the data you need to understand what happened is available, queryable, and complete.