Designing a Real-Time Event Processing System: Complex Event Processing, Windowed Aggregations, and Pattern Detection at Scale

Most systems start by processing events one at a time. A payment comes in, you validate it, you persist it. An order arrives, you reserve inventory, you send a confirmation. This works fine until the question changes from “what happened?” to “what pattern is happening across many events over time?” That shift is where real-time event processing gets architecturally interesting, and where naive implementations collapse under load.

Complex event processing (CEP) is the discipline of deriving meaning from event streams by evaluating patterns, sequences, and aggregations across time windows. It shows up in fraud detection, anomaly detection, recommendation systems, network monitoring, and anywhere else where a single event is insufficient and the relationship between events matters.

This article walks through the core building blocks: window types, pattern matching, and the framework tradeoffs you face when choosing between Kafka Streams, Apache Flink, and lighter alternatives.

The Problem with Stateless Stream Processing

A stateless stream processor transforms each event independently. It is easy to reason about, easy to scale horizontally, and easy to test. The problem is that most useful analytical queries are inherently stateful.

“Did this user attempt login more than five times in the last two minutes?” requires state. “Is this transaction sequence consistent with a card testing attack?” requires state. “Is the 95th-percentile API latency over the last 30 seconds crossing a threshold?” requires state.

Once you accept that you need state, the next question is: over what time boundary? This is where window semantics become load-bearing.

Window Types

There are three primary window patterns. Each fits a different query shape.

Tumbling windows partition time into fixed, non-overlapping buckets. A 5-minute tumbling window produces one result per 5-minute interval: [0:00-5:00), [5:00-10:00), and so on. Events belong to exactly one window. This is the simplest model and works well for metrics aggregation: request counts per minute, error rates per hour, revenue per day.

Sliding windows produce overlapping results. A 5-minute window sliding every 1 minute means at any given second, the active window contains the last 5 minutes of events, and a new window closes every minute. An event can appear in multiple windows. This is appropriate when you need continuous coverage, such as “alert if the error rate exceeds 5% over any 5-minute period,” not just at fixed boundaries.

Session windows are defined by activity gaps rather than clock time. A session window stays open as long as events continue arriving within a gap threshold. If no event arrives within 30 seconds, the session closes. This models user behavior naturally: a checkout flow, a video watch session, a sequence of API calls from a single client.

Implementing Windowed Aggregation in TypeScript

Below is a self-contained implementation of all three window types against an in-memory event buffer. This is not production Flink, but it illustrates the mechanics clearly and is directly testable.

type Event = {
  timestamp: number; // Unix ms
  userId: string;
  type: string;
  value: number;
};

type WindowResult = {
  windowStart: number;
  windowEnd: number;
  count: number;
  sum: number;
};

// Tumbling window: fixed non-overlapping buckets
function tumblingWindow(
  events: Event[],
  windowSizeMs: number
): WindowResult[] {
  const buckets = new Map<number, { count: number; sum: number }>();

  for (const event of events) {
    const bucketKey =
      Math.floor(event.timestamp / windowSizeMs) * windowSizeMs;
    const bucket = buckets.get(bucketKey) ?? { count: 0, sum: 0 };
    bucket.count++;
    bucket.sum += event.value;
    buckets.set(bucketKey, bucket);
  }

  return Array.from(buckets.entries())
    .sort(([a], [b]) => a - b)
    .map(([start, data]) => ({
      windowStart: start,
      windowEnd: start + windowSizeMs,
      count: data.count,
      sum: data.sum,
    }));
}

// Sliding window: overlapping windows at regular intervals
function slidingWindow(
  events: Event[],
  windowSizeMs: number,
  slideMs: number
): WindowResult[] {
  if (events.length === 0) return [];

  const earliest = Math.min(...events.map((e) => e.timestamp));
  const latest = Math.max(...events.map((e) => e.timestamp));
  const results: WindowResult[] = [];

  // Align start to slide boundary
  let windowStart = Math.floor(earliest / slideMs) * slideMs;

  while (windowStart <= latest) {
    const windowEnd = windowStart + windowSizeMs;
    const windowEvents = events.filter(
      (e) => e.timestamp >= windowStart && e.timestamp < windowEnd
    );

    if (windowEvents.length > 0) {
      results.push({
        windowStart,
        windowEnd,
        count: windowEvents.length,
        sum: windowEvents.reduce((acc, e) => acc + e.value, 0),
      });
    }

    windowStart += slideMs;
  }

  return results;
}

// Session window: gap-based grouping
function sessionWindow(events: Event[], gapMs: number): WindowResult[] {
  if (events.length === 0) return [];

  const sorted = [...events].sort((a, b) => a.timestamp - b.timestamp);
  const sessions: WindowResult[] = [];

  let sessionStart = sorted[0].timestamp;
  let sessionEnd = sorted[0].timestamp;
  let count = 1;
  let sum = sorted[0].value;

  for (let i = 1; i < sorted.length; i++) {
    const event = sorted[i];
    if (event.timestamp - sessionEnd <= gapMs) {
      sessionEnd = event.timestamp;
      count++;
      sum += event.value;
    } else {
      sessions.push({
        windowStart: sessionStart,
        windowEnd: sessionEnd,
        count,
        sum,
      });
      sessionStart = event.timestamp;
      sessionEnd = event.timestamp;
      count = 1;
      sum = event.value;
    }
  }

  sessions.push({ windowStart: sessionStart, windowEnd: sessionEnd, count, sum });
  return sessions;
}

In a real stream processor, windows are maintained incrementally rather than computed over a buffered set. The buffer approach here is only appropriate for batch-mode backfill or unit testing.

Event Pattern Detection

Aggregations answer “how much?” Pattern detection answers “did this sequence happen?” These are different problems with different data structures.

The classic example is fraud detection. A card testing attack typically looks like: multiple small-value authorization attempts in rapid succession, often across different merchants, followed by a high-value purchase if the small ones succeed. No single event triggers the rule. The sequence does.

CEP engines model this with finite automata or rule graphs. Each rule describes a sequence of events with temporal constraints. The engine maintains partial match state for every active sequence across every entity.

Here is a lightweight pattern detector that models a sequence requirement:

type PatternStep = {
  type: string;
  predicate?: (event: Event) => boolean;
  maxDelayMs?: number; // Max time allowed since the previous matched event
};

type PatternMatch = {
  matched: boolean;
  events: Event[];
  completedAt?: number;
};

function detectSequencePattern(
  events: Event[],
  pattern: PatternStep[],
  entityKey: keyof Event
): Map<string, PatternMatch[]> {
  const sorted = [...events].sort((a, b) => a.timestamp - b.timestamp);

  // partialMatches[entityId] = list of in-progress match states
  const partialMatches = new Map<
    string,
    Array<{ step: number; events: Event[] }>
  >();
  const completedMatches = new Map<string, PatternMatch[]>();

  for (const event of sorted) {
    const entityId = String(event[entityKey]);

    if (!partialMatches.has(entityId)) partialMatches.set(entityId, []);
    if (!completedMatches.has(entityId)) completedMatches.set(entityId, []);

    const inProgress = partialMatches.get(entityId)!;
    const nextInProgress: typeof inProgress = [];

    for (const match of inProgress) {
      const nextStep = pattern[match.step];
      const prevEvent = match.events[match.events.length - 1];

      // Drop partial matches that exceeded their temporal constraint
      if (
        nextStep.maxDelayMs !== undefined &&
        event.timestamp - prevEvent.timestamp > nextStep.maxDelayMs
      ) {
        continue;
      }

      if (
        event.type === nextStep.type &&
        (nextStep.predicate === undefined || nextStep.predicate(event))
      ) {
        const updatedMatch = {
          step: match.step + 1,
          events: [...match.events, event],
        };

        if (updatedMatch.step === pattern.length) {
          completedMatches.get(entityId)!.push({
            matched: true,
            events: updatedMatch.events,
            completedAt: event.timestamp,
          });
        } else {
          nextInProgress.push(updatedMatch);
        }
      } else {
        nextInProgress.push(match);
      }
    }

    // Check if this event starts a new pattern attempt
    const firstStep = pattern[0];
    if (
      event.type === firstStep.type &&
      (firstStep.predicate === undefined || firstStep.predicate(event))
    ) {
      if (pattern.length === 1) {
        completedMatches.get(entityId)!.push({
          matched: true,
          events: [event],
          completedAt: event.timestamp,
        });
      } else {
        nextInProgress.push({ step: 1, events: [event] });
      }
    }

    partialMatches.set(entityId, nextInProgress);
  }

  return completedMatches;
}

// Example: detect a failed login followed by a successful login within 60 seconds
const loginPattern: PatternStep[] = [
  { type: "login_failed" },
  { type: "login_success", maxDelayMs: 60_000 },
];

In production CEP systems like Apache Flink, the NFA (non-deterministic finite automaton) state is managed per-key and checkpointed to durable storage. The partial match pruning on timeout is critical: without it, long-lived partial matches accumulate indefinitely and cause memory pressure that grows proportionally to the number of unique entities in the stream.

Framework Tradeoffs

Choosing the right processing framework depends on your scale, operational maturity, and the complexity of your event patterns.

Framework	Strengths	Weaknesses	Best Fit
Apache Flink	Native CEP library, exactly-once semantics, large window state, event time support	Operational overhead, JVM tuning required, steep learning curve	High-throughput workloads, complex pattern matching, teams with JVM expertise
Kafka Streams	Native Kafka integration, exactly-once with transactions, simple deployment model	Limited CEP expressiveness, no native complex pattern API	Already on Kafka, simpler aggregations, teams comfortable with Java/Scala
Redis Streams + custom	Low latency, simple operations, flexible schema	Manual state management, limited fault tolerance, DIY windowing	Low scale, startup phase, simple and well-defined patterns
Node.js / TypeScript (in-process)	Full control, fast iteration, no infrastructure	Not distributed, limited throughput, no built-in checkpointing	Prototype, low event volume, single-instance services
Materialize / RisingWave	SQL over streams, incremental view maintenance, familiar interface	Less mature, fewer integrations, managed cost at scale	Teams comfortable with SQL, moderate query complexity

The JVM frameworks carry real operational cost. You are managing heap sizing, garbage collection pressure under high state volume, checkpoint tuning, and topic partition rebalancing. For a startup processing under 10,000 events per second with modest state, a well-structured TypeScript worker backed by Redis or Postgres is often the right call. The cost is not licensing, it is engineering time spent on infrastructure rather than product.

Production Considerations

Watermarks and late-arriving events. In event time processing, events arrive out of order. A request logged at 14:03:00 may reach the processor at 14:03:45 due to network jitter, mobile client buffering, or upstream queue lag. Without watermarks, you either wait indefinitely for stragglers or close windows prematurely and lose data. Flink’s watermark mechanism advances a per-partition logical clock to signal “events before time T are unlikely to arrive.” Setting the watermark lag is a direct tradeoff: higher lag improves correctness, higher lag increases result latency. For most fraud detection use cases, a 5-30 second watermark lag is acceptable.

State store sizing. Session windows and pattern detection hold partial match state per entity. If you have 10 million active users and each user can have multiple in-flight pattern matches, your state store grows large quickly. Profile your state-per-entity before committing to a storage backend. RocksDB-backed state in Flink handles this well at scale. Redis works if you set TTLs aggressively and monitor key count. An in-process map without eviction will eventually cause an OOM on any non-trivial workload.

Checkpoint and recovery latency. Exactly-once processing requires checkpointing stream position and state atomically. In Flink, checkpoint duration directly impacts recovery time after failure. If your checkpoints take 2 minutes to complete, your worst-case recovery time is at least 2 minutes plus replay time. For systems with latency SLAs, checkpoint intervals of 10-30 seconds with incremental checkpointing are a reasonable starting point. Monitor checkpoint duration as a first-class metric.

Backpressure propagation. When processing falls behind the ingest rate, backpressure must propagate upstream rather than buffering events indefinitely. Kafka naturally provides this by slowing consumer poll rates. In custom workers, avoid unbounded in-memory queues between your consumer and your processing loop. A large buffer between I/O stages will grow silently until it exhausts available memory, and you will not notice until the process crashes.

Testing windowed logic. Window semantics are notoriously hard to test against a live stream. The right approach is to inject events with explicit timestamps and verify results against expected window boundaries in unit tests, completely decoupled from wall-clock time. The TypeScript functions above are straightforward to test this way. In Flink, TestHarness provides the equivalent mechanism for testing stateful operators without running a full cluster.

Schema evolution. Events are produced by many services and their schemas evolve independently. A pattern rule that checks event.type === "checkout_completed" breaks silently if the producing service renames that field to event.eventType. Use a schema registry (Confluent Schema Registry, AWS Glue) and enforce compatibility modes. Avro or Protobuf with a schema registry catches breaking changes at produce time rather than at processing time, which is far cheaper to debug.

Where Simple Beats Complex

It is tempting to reach for Flink because the problem sounds like it requires Flink. Often it does not.

If your event volume is under a few thousand per second and your patterns are expressible as SQL window functions, a streaming SQL query against a materialized view in Postgres will serve you well. Postgres window functions over append-only tables with a time-based index handle surprising amounts of analytical load before you need a dedicated stream processor.

If you need pattern detection over moderate event volumes but your patterns are a small finite set, a Redis-backed state machine with keyed expiry is often simpler to operate and debug than a Flink cluster. The operational surface is much smaller, and the failure modes are easier to reason about.

The question is not “can Flink handle this?” It can. The question is “does the operational complexity of Flink justify the scale and pattern complexity I actually have?” For most products below a certain scale, the answer is no.

The Core Constraint

A real-time event processing system is fundamentally a stateful computation over time. The complexity is not in the processing logic itself but in the boundary conditions: how you handle late data, how you bound state growth, how you recover from failures without reprocessing everything, and how you validate that your window semantics match your actual business requirements.

Get the window semantics wrong and your fraud detection fires on patterns that do not exist. Get the watermark wrong and you drop events that should have triggered alerts. Get state management wrong and your system degrades silently until it falls over under load.

The frameworks abstract these details but do not eliminate them. Understanding the mechanics underneath is what separates a CEP system that works in production from one that passes demos.