Designing a Real-Time Analytics Pipeline: Streaming Ingestion, Aggregation, and Dashboards at Scale

Batch ETL made sense when “fresh data” meant yesterday’s report. You’d run a nightly job, load a data warehouse, and serve dashboards off that snapshot. That model still works for finance and compliance reporting where the question is “what happened last quarter.” It breaks down completely when the question is “what’s happening right now.”

Real-time analytics pipelines are a different class of system. You’re not moving data from A to B on a schedule. You’re processing a continuous stream of events, computing aggregates over sliding time windows, handling late-arriving records, and pushing results to dashboards with sub-second latency. Every component in the stack has to be designed for this model, because batch-optimized tooling will become your bottleneck the moment traffic spikes.

This article covers the full architecture: ingestion, processing, storage, and dashboard delivery. The goal is to give you a mental model for each layer and the tradeoffs that matter in production.

Why Batch ETL Breaks for Real-Time Use Cases

Batch pipelines process data in chunks: extract from source, transform, load into warehouse, refresh BI layer. The refresh interval is the fundamental constraint. Even with aggressive scheduling, you’re looking at 15-minute minimum cycles in well-tuned systems. In practice, most teams ship hourly or nightly batches.

The failure modes are predictable: a spike in errors at 2 AM shows up in the dashboard at 6 AM when the nightly job finishes, a payment anomaly goes undetected for 45 minutes while the next batch runs, and “how many users completed checkout in the last 5 minutes” has no answer.

The deeper problem is that batch pipelines are stateful in the wrong way. They accumulate state in files and warehouse snapshots. Reprocessing means re-running the entire job. Adding a new metric means changing the ETL and waiting for the next run. Real-time pipelines invert this: state lives in a stream processor, new metrics are added by deploying new consumers, and latency is measured in seconds.

Streaming Ingestion Layer

The ingestion layer is a distributed log: an ordered, durable, replayable sequence of events. Producers write events; consumers read from any offset. The log buffers between producers and processors, absorbing bursts without dropping data.

Kafka is the default choice for most teams. It partitions topics across brokers, retains messages on disk for configurable periods, and scales to millions of events per second with proper tuning. Consumers track their own offsets, which enables replay, A/B testing between consumer versions, and crash recovery without data loss.

Amazon Kinesis is the managed equivalent for AWS shops. Lower operational overhead at the cost of less flexibility. Shard limits (1 MB/s write, 2 MB/s read per shard) require planning, and retention caps at 365 days maximum.

Redpanda is a Kafka-compatible broker written in C++ without the JVM. Lower latency (single-digit milliseconds), simpler operations (no ZooKeeper, no separate controller), and better predictability under load. Worth evaluating if Kafka’s JVM tuning overhead is a real cost for your team.

The choice between these three is mostly about operational context. All three will handle the ingestion load for most applications. The stream processor matters more.

Stream Processing Patterns

Once events are in the log, you need to compute aggregates. Raw events are not useful to dashboards. You need counts, sums, percentiles, and rates, computed over time windows, continuously updated as new events arrive.

Windowed Aggregation

Most analytics questions are window-based: “events in the last 60 seconds,” “revenue in the last 5 minutes,” “error rate per 30-second bucket.” Stream processors model this explicitly.

There are three window types:

Tumbling windows divide time into fixed, non-overlapping buckets. A 1-minute tumbling window produces one result per minute. Simple to reason about, but the edge is abrupt: a query for “last 60 seconds” at 12:00:59 and 12:01:01 produces completely different buckets.

Sliding windows move continuously. A 60-second sliding window updated every 10 seconds produces 6 overlapping windows per minute. Higher compute cost, smoother output.

Session windows close when activity gaps exceed a threshold. Useful for user session analytics but complex to implement correctly at scale.

Here is a TypeScript consumer implementing tumbling window aggregation over a Kafka stream using the kafkajs library:

import { Kafka, Consumer, EachMessagePayload } from "kafkajs";

interface ClickEvent {
  userId: string;
  pageId: string;
  timestamp: number; // epoch ms
}

interface WindowBucket {
  windowStart: number;
  pageId: string;
  count: number;
  uniqueUsers: Set<string>;
}

const WINDOW_SIZE_MS = 60_000; // 1-minute tumbling windows
const FLUSH_INTERVAL_MS = 5_000; // flush to storage every 5 seconds

const kafka = new Kafka({
  clientId: "analytics-consumer",
  brokers: ["kafka-broker-1:9092", "kafka-broker-2:9092"],
});

const consumer: Consumer = kafka.consumer({
  groupId: "analytics-aggregator",
  sessionTimeout: 30_000,
  heartbeatInterval: 3_000,
});

// In-memory window state: windowKey -> bucket
const windows = new Map<string, WindowBucket>();

function getWindowStart(timestamp: number): number {
  return Math.floor(timestamp / WINDOW_SIZE_MS) * WINDOW_SIZE_MS;
}

function windowKey(windowStart: number, pageId: string): string {
  return `${windowStart}:${pageId}`;
}

function accumulate(event: ClickEvent): void {
  const windowStart = getWindowStart(event.timestamp);
  const key = windowKey(windowStart, event.pageId);

  if (!windows.has(key)) {
    windows.set(key, {
      windowStart,
      pageId: event.pageId,
      count: 0,
      uniqueUsers: new Set(),
    });
  }

  const bucket = windows.get(key)!;
  bucket.count++;
  bucket.uniqueUsers.add(event.userId);
}

async function flushExpiredWindows(storage: StorageClient): Promise<void> {
  const now = Date.now();
  const cutoff = getWindowStart(now) - WINDOW_SIZE_MS; // flush windows older than 1 period

  for (const [key, bucket] of windows.entries()) {
    if (bucket.windowStart <= cutoff) {
      await storage.upsert({
        windowStart: new Date(bucket.windowStart),
        pageId: bucket.pageId,
        clickCount: bucket.count,
        uniqueUsers: bucket.uniqueUsers.size,
      });
      windows.delete(key);
    }
  }
}

async function run(storage: StorageClient): Promise<void> {
  await consumer.connect();
  await consumer.subscribe({ topic: "click-events", fromBeginning: false });

  // Periodic flush loop independent of message processing
  const flushTimer = setInterval(() => flushExpiredWindows(storage), FLUSH_INTERVAL_MS);

  await consumer.run({
    eachMessage: async ({ message }: EachMessagePayload) => {
      if (!message.value) return;

      const event: ClickEvent = JSON.parse(message.value.toString());
      accumulate(event);
    },
  });

  process.on("SIGTERM", async () => {
    clearInterval(flushTimer);
    await flushExpiredWindows(storage); // drain remaining windows on shutdown
    await consumer.disconnect();
  });
}

This is deliberately simple. Production implementations add partition-aware state, checkpoint offsets atomically with flushes for exactly-once semantics, and handle out-of-order events.

Exactly-Once Semantics

At-least-once delivery is the Kafka default. Consumers commit offsets after processing, but crashes between processing and committing cause reprocessing. For analytics, duplicate counts produce wrong numbers.

Exactly-once requires coordinating offset commits with output writes atomically. In Kafka, this means using transactional producers and consumers: the processor writes output and commits its input offset in the same transaction. Kafka’s transaction API handles this, but it adds complexity and reduces throughput by 10-20%.

An alternative is idempotent writes: design your output storage to handle duplicate writes gracefully. If flushing a window aggregate uses an UPSERT keyed on (windowStart, pageId), duplicate flushes produce the same result. This is often simpler than full transactional processing.

Late Arrivals

Events arrive late. Mobile apps batch events and send them on reconnect. CDN logs arrive with 30-second delays. Sensor telemetry buffers in edge devices.

You have two options. The first is to set a watermark: a timestamp threshold beyond which late events are dropped. A 2-minute watermark means events older than 2 minutes are discarded when they arrive. Simple, predictable, data loss is explicit.

The second is to allow corrections: maintain a slightly longer window in state, accept late events into the appropriate bucket, and emit corrected aggregates. This complicates downstream consumers (they need to handle updates, not just inserts) but avoids data loss for predictable latency distributions.

In practice, most teams use watermarks with monitoring. Track the distribution of event-arrival lag. Set the watermark at the 99th percentile. Alert when the late-event drop rate exceeds a threshold.

Time-Series Storage

Processed aggregates need a storage layer optimized for time-range queries: “give me all rows where timestamp is between X and Y, grouped by page_id.” Standard relational databases handle this poorly at scale because they’re not designed for append-heavy workloads or columnar access patterns.

ClickHouse is the strong default for analytics workloads. Columnar storage, vectorized query execution, and aggressive compression make it fast for aggregation queries across billions of rows. It handles both raw event storage and pre-aggregated results. The MergeTree engine and its variants (ReplacingMergeTree, AggregatingMergeTree) are designed specifically for the append-and-query pattern of analytics pipelines. Ingestion throughput is high, and query latency for typical analytics queries is in the milliseconds even at terabyte scale.

TimescaleDB extends PostgreSQL with hypertables, which automatically partition time-series data by time intervals. You keep the full PostgreSQL ecosystem (extensions, joins, ACID transactions) while gaining time-series optimizations. Better choice when your analytics data lives alongside relational data and you don’t want to manage a separate system. Query performance is good but benchmarks behind ClickHouse for pure analytics at very high cardinality.

Apache Druid is designed for sub-second queries on real-time and historical data simultaneously. It ingests from Kafka directly, pre-aggregates during ingestion, and serves queries from memory-mapped segment files. The operational complexity is high (multiple node types: broker, coordinator, historical, middle manager), but the query speed is exceptional for pre-defined aggregation patterns. Common in SaaS products where the analytics dashboard is customer-facing and latency SLAs are tight.

Materialized Views for Dashboards

Dashboards query the same aggregates repeatedly. If your storage layer supports materialized views, define them. A materialized view pre-computes the result of a query and stores it, updating incrementally as new data arrives.

In ClickHouse, a materialized view is a special table that is populated by a trigger on inserts to a source table:

-- Source table: raw aggregated windows
CREATE TABLE page_click_windows (
  window_start DateTime,
  page_id String,
  click_count UInt64,
  unique_users UInt64
) ENGINE = MergeTree()
ORDER BY (window_start, page_id);

-- Materialized view: hourly rollup for historical charts
CREATE MATERIALIZED VIEW page_clicks_hourly
ENGINE = SummingMergeTree()
ORDER BY (hour, page_id)
AS
SELECT
  toStartOfHour(window_start) AS hour,
  page_id,
  sum(click_count) AS click_count,
  sum(unique_users) AS unique_users -- approximation; for exact, use HyperLogLog
FROM page_click_windows
GROUP BY hour, page_id;

The dashboard queries page_clicks_hourly for the time-range chart and page_click_windows for the last-60-minutes view. Both queries are fast because data is pre-aggregated.

WebSocket Push to the Dashboard

Polling the storage layer from the dashboard frontend introduces latency and unnecessary load. For live dashboards, push updates to connected clients via WebSocket.

Here is a TypeScript Node.js server that reads from ClickHouse and pushes updates to subscribers using the ws library:

import { WebSocketServer, WebSocket } from "ws";
import { createClient } from "@clickhouse/client";
import { IncomingMessage } from "http";

interface DashboardUpdate {
  pageId: string;
  windowStart: string;
  clickCount: number;
  uniqueUsers: number;
}

const ch = createClient({
  host: process.env.CLICKHOUSE_HOST ?? "http://localhost:8123",
  database: "analytics",
});

const wss = new WebSocketServer({ port: 8080 });
const subscribers = new Set<WebSocket>();

wss.on("connection", (ws: WebSocket, _req: IncomingMessage) => {
  subscribers.add(ws);

  ws.on("close", () => {
    subscribers.delete(ws);
  });

  ws.on("error", () => {
    subscribers.delete(ws);
  });
});

function broadcast(payload: DashboardUpdate[]): void {
  const message = JSON.stringify(payload);

  for (const ws of subscribers) {
    if (ws.readyState === WebSocket.OPEN) {
      ws.send(message);
    }
  }
}

async function pollAndPush(): Promise<void> {
  const since = new Date(Date.now() - 5 * 60 * 1000); // last 5 minutes

  const result = await ch.query({
    query: `
      SELECT
        page_id,
        window_start,
        click_count,
        unique_users
      FROM page_click_windows
      WHERE window_start >= {since:DateTime}
      ORDER BY window_start DESC
    `,
    query_params: { since: since.toISOString().replace("T", " ").slice(0, 19) },
    format: "JSONEachRow",
  });

  const rows = await result.json<DashboardUpdate>();

  if (rows.length > 0 && subscribers.size > 0) {
    broadcast(rows);
  }
}

// Push updates every 2 seconds to connected clients
setInterval(() => {
  pollAndPush().catch(console.error);
}, 2_000);

console.log("Dashboard WebSocket server running on port 8080");

In production, add authentication on the WebSocket upgrade request, per-client subscriptions so clients only receive the page IDs they care about, and a circuit breaker around the ClickHouse query to prevent a slow query from cascading into broadcast failures.

Backpressure Handling

Backpressure is what happens when your consumer processes slower than the producer writes. Without management, the consumer falls behind, the consumer group lag grows, memory pressure builds, and eventually the consumer crashes.

The Kafka partitioning model provides natural backpressure: a consumer can only process as fast as it processes. Lag accumulates in the broker, which has ample disk. The risk is that lag grows unbounded during a spike and takes hours to drain.

Strategies to handle this:

Horizontal scaling: Add consumer instances up to the number of partitions. More partitions allow more parallelism. Over-partition topics at creation time (you can’t reduce partition count without recreating the topic).

Separate fast and slow paths: High-volume events (page views, clicks) go through a lightweight consumer that only counts. Low-volume events (purchases, signups) go through a richer consumer that joins against user data. Don’t let expensive processing block high-volume throughput.

Async flush with bounded queues: The accumulate-and-flush pattern in the consumer above decouples in-memory accumulation from storage writes. If storage is slow, the flush queue builds. Add a queue depth metric and alert before it becomes a problem.

Shed load deliberately: If lag exceeds a threshold, drop non-critical events rather than falling further behind. For analytics (not billing), dropping 0.1% of events during a spike is acceptable. Model this explicitly in your monitoring rather than letting it happen accidentally through consumer crashes.

Lambda vs. Kappa Architecture: Tradeoffs

Two architectural patterns dominate real-time analytics systems. The choice affects your operational complexity, reprocessing capability, and query consistency.

Dimension	Lambda	Kappa
Core idea	Parallel batch and stream layers, merge at query time	Single stream layer; reprocess by replaying the log
Query consistency	Batch layer provides “correct” results; stream layer provides “recent” approximation	Single source of truth; consistency depends on stream processor correctness
Reprocessing	Run batch job over historical data; stream catches up	Replay Kafka topic from offset 0 with updated consumer
Operational complexity	Two separate pipelines to maintain, test, and deploy	One pipeline; simpler operations
Storage requirements	Batch layer (data warehouse) + stream layer (fast store)	Stream store only; log retention must cover reprocessing window
Historical query performance	Excellent (batch layer is warehouse-optimized)	Depends on stream store; ClickHouse handles this well
Common failure mode	Batch and stream results diverge; debugging is painful	Stream processor bug corrupts all results until reprocessed
Best fit	Teams already running a data warehouse; accuracy requirements differ between recent and historical	Teams starting fresh; want operational simplicity

Most new systems should start with kappa. The lambda architecture’s batch layer adds real cost: a second pipeline to maintain, a second set of transformations to keep in sync, and subtle divergences between batch and stream outputs that are painful to debug. Kappa eliminates that class of problem. The tradeoff is that your stream processor has to be correct, because it’s the only source of truth.

Production Considerations

Consumer group lag monitoring is non-negotiable. Export Kafka consumer group lag to your metrics system and alert when lag exceeds your acceptable recovery time. A lag of 6 hours means a problem that takes hours to drain even after fixing the root cause.

Schema evolution is the silent killer of analytics pipelines. Use a schema registry (Confluent or AWS Glue) with a compatibility mode that prevents breaking changes. Design consumers to handle missing fields with defaults, not panics.

Retention policy alignment: Kafka topic retention must cover your worst-case reprocessing window. If you need to replay 7 days of events, set retention to 8 days minimum. Storage is cheap; losing replay capability is not.

Clock skew handling: Event timestamps come from producer machines and those clocks drift. Always use the event’s own timestamp field for windowing, not the Kafka message timestamp or the consumer system clock.

Dead letter queues: Route malformed events and processing failures to a separate topic rather than crashing the consumer or dropping data silently. Alert when DLQ growth rate spikes.

The ingestion layer is largely commodity at this point. The interesting decisions are in the processing model (windowing, watermarks, exactly-once), the storage choice, and how you handle the operational realities of continuous processing at scale. Kappa with Kafka, a well-tuned stream consumer, and ClickHouse covers the overwhelming majority of real-time analytics requirements. Monitor consumer lag, schema drift, and late-arrival rates as first-class signals, and that stack handles millions of events per minute without exotic infrastructure.