System Design ·

Event-Driven Architecture: When to Use It and When Not To

A practical guide to event-driven architecture for senior engineers covering pub/sub, event sourcing, and CQRS patterns, with real tradeoffs around eventual consistency, debugging challenges, idempotency, broker selection, and a concrete decision framework for when EDA earns its complexity.

Event-Driven Architecture: When to Use It and When Not To

Event-driven architecture gets oversold. Teams adopt it because it sounds modern, because their tech lead read about how Netflix uses Kafka, because microservices seem to demand it. Then six months later they’re debugging a production incident where a payment was processed twice and nobody can tell you why because the audit trail is spread across four services and three message queues.

This is not an argument against EDA. It is genuinely the right architecture for certain problems. The issue is that most teams reach for it before they have a concrete reason to, without understanding what they’re trading away.

This guide covers the core patterns, the specific conditions where EDA earns its complexity, the real tradeoffs you need to understand before committing, and a decision framework you can use in practice.

What “Event-Driven” Actually Means

An event is a record that something happened. Not a command telling another service what to do: a fact about the past. OrderPlaced, PaymentFailed, UserEmailVerified. Events are immutable by definition because you cannot un-happen something.

In an event-driven system, services communicate by publishing events and subscribing to them, rather than calling each other directly. This one change ripples through your entire system design in ways that are easy to underestimate.

There are three distinct patterns that get grouped under the “event-driven” label, and they solve different problems.

Pub/Sub

The simplest pattern. A producer publishes a message to a topic. Multiple consumers subscribe and each receives a copy. The producer has no knowledge of who is listening.

// Producer: Order service publishes a fact, not a command
async function placeOrder(order: Order): Promise<void> {
  await db.orders.insert(order);

  // Publish after the DB write succeeds
  await messageBus.publish("order.placed", {
    orderId: order.id,
    userId: order.userId,
    items: order.items,
    totalCents: order.totalCents,
    placedAt: new Date().toISOString(),
  });
}

// Consumer: Inventory service — unaware of other subscribers
messageBus.subscribe("order.placed", async (event) => {
  for (const item of event.items) {
    await inventory.decrement(item.sku, item.quantity);
  }
});

// Consumer: Email service — also subscribing, producer is unaware
messageBus.subscribe("order.placed", async (event) => {
  await email.sendOrderConfirmation(event.userId, event.orderId);
});

The order service no longer needs to know about inventory or email. You can add a third consumer (fraud detection, loyalty points, analytics) without touching the order service. This is the core value proposition of pub/sub: decoupled fan-out.

Event Sourcing

Instead of storing current state, you store every event that led to that state. The current state is derived by replaying the event log.

// Events are the source of truth, not rows in an accounts table
type AccountEvent =
  | { type: "AccountOpened"; accountId: string; ownerId: string; openedAt: string }
  | { type: "MoneyDeposited"; accountId: string; amountCents: number; at: string }
  | { type: "MoneyWithdrawn"; accountId: string; amountCents: number; at: string }
  | { type: "AccountFrozen"; accountId: string; reason: string; at: string };

// Pure function: no side effects, easy to test
function applyEvent(state: AccountState, event: AccountEvent): AccountState {
  switch (event.type) {
    case "AccountOpened":
      return { accountId: event.accountId, balanceCents: 0, status: "active" };
    case "MoneyDeposited":
      return { ...state, balanceCents: state.balanceCents + event.amountCents };
    case "MoneyWithdrawn":
      return { ...state, balanceCents: state.balanceCents - event.amountCents };
    case "AccountFrozen":
      return { ...state, status: "frozen" };
  }
}

async function getAccountState(accountId: string): Promise<AccountState> {
  const events = await eventStore.getEventsForAccount(accountId);
  return events.reduce(applyEvent, null as unknown as AccountState);
}

// You can reconstruct state at any point in time
async function getAccountStateAt(accountId: string, at: Date): Promise<AccountState> {
  const events = await eventStore.getEventsForAccountBefore(accountId, at);
  return events.reduce(applyEvent, null as unknown as AccountState);
}

You get a complete audit log and the ability to reconstruct state at any point in time. You pay for this with query complexity and the overhead of replaying events on every read. Snapshots mitigate the replay cost: periodically persist the derived state and only replay events after the snapshot.

CQRS

Command Query Responsibility Segregation separates your write model from your read model. Commands mutate state, queries read from a separately maintained projection.

// Write side: validates business rules, emits events
async function handleWithdrawMoney(cmd: WithdrawMoneyCommand): Promise<void> {
  const events = await eventStore.getEventsForAccount(cmd.accountId);
  const state = events.reduce(applyEvent, null as unknown as AccountState);

  if (state.status === "frozen") {
    throw new Error("Account is frozen");
  }
  if (state.balanceCents < cmd.amountCents) {
    throw new Error("Insufficient funds");
  }

  await eventStore.append({
    type: "MoneyWithdrawn",
    accountId: cmd.accountId,
    amountCents: cmd.amountCents,
    at: new Date().toISOString(),
  });
}

// Read side: a denormalized projection, rebuilt from the event stream
// This can be a Postgres table, Elasticsearch index, Redis hash — whatever
// serves the query patterns, not the write model
async function getAccountSummary(accountId: string): Promise<AccountSummary> {
  return await readDb.accountSummaries.findOne({ accountId });
}

CQRS lets you optimize reads and writes independently. Your read model can be structured for exactly the query patterns you need. You pay for this with the complexity of keeping the read model in sync and accepting that reads are eventually consistent.

CQRS and event sourcing pair naturally but are independent choices. You can use CQRS without event sourcing (separate read/write databases, synced by CDC). You can use event sourcing without CQRS (rebuild state from events on every read, add snapshots for performance). The combination is powerful and more complex than either alone.

When EDA Is the Right Call

Fan-out without coupling

You have one producer and multiple independent consumers. Adding a new consumer should not require changing the producer or coordinating a joint deployment. Classic cases: an order placed triggers inventory updates, email confirmation, analytics tracking, fraud scoring, and loyalty point accrual. With synchronous calls, the order service would need to know about all of them and handle partial failures from each. With pub/sub, it publishes one event and each team owns their subscription independently.

Workload isolation between a fast producer and a slow processor

A slow consumer should not block a fast producer. If your image processing job takes 30 seconds per image and your upload service accepts 100 uploads per minute, you need a queue between them. The queue absorbs the burst and the processor works at its own pace. Without a queue, you either block uploads (bad UX) or spawn unbounded goroutines and threads (OOM crash under load).

Audit requirements as a first-class concern

In regulated industries (finance, healthcare, anything with compliance obligations), you often need to answer “what was the state of this record on a specific date, and what caused each change?” Event sourcing makes this trivial. With a mutable database, you need to bolt on audit tables after the fact, and they’re always incomplete. If your domain naturally requires this kind of history, event sourcing pays off its complexity faster than in domains where you are adding it artificially.

Cross-boundary integration at scale

When integrating with external systems or across bounded contexts in a large organization, events are a natural contract. A PaymentSettled event published by the payments team is consumed by the accounting team, the analytics team, and the product team, each projecting it into their own model. This scales better than shared databases or synchronous API chains where a single slow consumer brings down others.

When Synchronous Calls Are the Better Choice

The operation needs to be atomic

A user signs up. You create an account, send a welcome email, and charge their credit card. If any step fails, you need to roll back. With synchronous calls in a single transaction, this is straightforward. With events, you are in the world of sagas and compensating transactions. This is solvable, but it is significantly more complex. For a signup flow, the complexity is almost never worth it.

You need the result immediately

A user submits a form and expects a response: success or failure. If the answer determines the HTTP response, you cannot defer the work to a queue. You can publish an event after completing the synchronous work, but the work itself has to be synchronous.

Your team is small and the domain is simple

Event-driven architecture requires operational maturity: you need to monitor queue depths, handle dead letter queues, reason about message ordering, and debug failures that span multiple services and time windows. For a team of two building a SaaS product with three services, this overhead costs more than it saves. Start with direct calls and well-defined service boundaries. You can introduce a message bus later when you have a concrete pain point it would solve.

You do not have idempotency handled

Every consumer in an event-driven system must be idempotent. Messages are delivered at least once in most brokers (Kafka, RabbitMQ, SQS). Your consumer will receive the same message more than once. If you process a PaymentFailed event twice and send the user two “your payment failed” emails, that is a support ticket. If you decrement inventory twice for the same order, that is a correctness bug.

// Idempotency via a processed-events table — runs inside a transaction
async function handleOrderPlaced(event: OrderPlacedEvent): Promise<void> {
  const alreadyProcessed = await db.processedEvents.findOne({
    eventId: event.id,
    consumer: "inventory-service",
  });

  if (alreadyProcessed) {
    // Safe to ignore: we already handled this delivery
    return;
  }

  await db.transaction(async (tx) => {
    for (const item of event.items) {
      await tx.inventory.decrement(item.sku, item.quantity);
    }
    // Insert inside the same transaction — atomic with the business logic
    await tx.processedEvents.insert({
      eventId: event.id,
      consumer: "inventory-service",
      processedAt: new Date(),
    });
  });
}

The idempotency check and the business logic must be in the same transaction. If you check outside the transaction and then insert inside, a concurrent duplicate delivery can slip through. If you skip idempotency handling because “it probably won’t happen twice in practice,” you will spend a weekend debugging a production incident.

The Real Tradeoffs

Eventual consistency is not optional

In a synchronous system, after a function call returns, all state changes are visible. In an event-driven system, a consumer may lag by milliseconds or minutes. A user places an order and immediately navigates to “My Orders.” Will the new order appear? It depends on whether the read model has been updated by the time the query arrives. If the read model is powered by a consumer processing events from a queue, the answer is “probably, but not guaranteed.”

You need to either accept this lag and design your UI accordingly, or build read-your-writes guarantees. One approach: for a short window after a write, read directly from the write database. After the window expires, fall back to the read replica.

// Read-your-writes: bypass the read replica for the originating user
// for a short window after they mutate data
async function getUserOrders(
  userId: string,
  sessionContext: { lastWriteAt: number | null }
): Promise<Order[]> {
  const readYourWritesWindowMs = 2000;
  const now = Date.now();

  if (
    sessionContext.lastWriteAt !== null &&
    now - sessionContext.lastWriteAt < readYourWritesWindowMs
  ) {
    // Read from the write database to ensure we see the user's own writes
    return await writeDb.orders.findByUserId(userId);
  }

  // Safe to use the read replica (eventually consistent view)
  return await readDb.orders.findByUserId(userId);
}

Message ordering is harder than it looks

Most brokers give you ordering within a partition or queue, but not across partitions. Kafka partitions by key, so all events for a given orderId will be ordered if you partition by orderId. But if you need ordering across different entity types in the same consumer, you need to design for it explicitly.

The failure mode is subtle: an OrderShipped event arrives at the consumer before the OrderPlaced event because they were on different Kafka partitions and the consumer was restarted between them. Your consumer sees a shipped order with no corresponding placed order and either crashes or silently corrupts state.

Design your consumers to handle out-of-order delivery. Either sort events before applying them, or make your apply logic tolerant of receiving events in any order (treating unknown state as a reason to defer rather than fail).

Debugging is harder

In a synchronous call stack, a bug has a traceable path: request in, error out, stack trace. In an event-driven system, cause and effect are separated in time and potentially in service. A bug might manifest as a consumer silently failing to process an event, with no visible error to the user.

You need distributed tracing from day one. Every event should carry a correlationId that links it back to the originating request.

// Propagate trace context through the event, not just within a service
async function placeOrder(
  order: Order,
  ctx: { traceId: string; spanId: string }
): Promise<void> {
  await messageBus.publish("order.placed", {
    orderId: order.id,
    items: order.items,
    totalCents: order.totalCents,
    placedAt: new Date().toISOString(),
    _meta: {
      traceId: ctx.traceId,       // links to the originating HTTP request
      causationId: ctx.spanId,    // the specific span that caused this event
      publishedAt: new Date().toISOString(),
      source: "order-service",
      version: "v1",
    },
  });
}

// Consumer logs with the same traceId for cross-service correlation
messageBus.subscribe("order.placed", async (event) => {
  logger.info("Processing order.placed", {
    traceId: event._meta.traceId,
    orderId: event.orderId,
    consumer: "inventory-service",
  });

  await handleOrderPlaced(event);
});

Without propagated trace context, when something goes wrong in production you’re looking at logs from four services and trying to correlate them by timestamp. That is a miserable experience.

Dead letter queues are not optional

When a consumer fails to process a message (due to a bug, a transient database error, a dependency being unavailable), the message needs to go somewhere. If you let it retry indefinitely, you get an infinite loop. If you discard it, you silently lose data.

Dead letter queues hold messages that failed after N retry attempts. You need to monitor them, alert when they accumulate, and have a process for replaying them after fixing the underlying bug. This is operational overhead that synchronous systems do not have.

The transactional outbox problem

Publishing events to a broker and calling it done introduces a subtle reliability gap. If the broker is unavailable between your database write and your publish call, you’ve written state but lost the event. Subscribers will never know the order was placed.

The transactional outbox pattern closes this gap:

// Write the event to the database in the same transaction as the business state.
// A separate relay process publishes from the outbox table to the broker.
async function placeOrder(order: Order): Promise<void> {
  await db.transaction(async (tx) => {
    await tx.orders.insert(order);

    // This row is committed atomically with the order row
    await tx.outbox.insert({
      id: crypto.randomUUID(),
      topic: "order.placed",
      payload: JSON.stringify({
        orderId: order.id,
        items: order.items,
        totalCents: order.totalCents,
        placedAt: new Date().toISOString(),
      }),
      createdAt: new Date(),
      publishedAt: null,  // relay sets this after successful publish
    });
  });
}

// Separate relay process (runs on a schedule or triggered by Postgres LISTEN/NOTIFY)
async function relayOutboxEvents(): Promise<void> {
  const pending = await db.outbox.findPending({ limit: 100 });

  for (const row of pending) {
    await messageBus.publish(row.topic, JSON.parse(row.payload));
    await db.outbox.markPublished(row.id);
  }
}

If you skip the outbox pattern and publish directly in application code, you will eventually drop events during broker outages or application crashes. For high-stakes events (payments, user actions that trigger billing), this is not acceptable.

Choosing a Broker

The broker choice affects delivery guarantees, ordering, and operational overhead.

  • Kafka: durable, ordered within partitions, high throughput, replay built-in. Good for high-volume streams, audit logs, event sourcing. Operationally heavier than the alternatives.
  • RabbitMQ: flexible routing, lower throughput than Kafka, simpler to operate at smaller scale. Good for task queues and fan-out with moderate volume. Does not retain messages after delivery by default.
  • AWS SQS / Google Pub/Sub: managed, scales automatically, good enough delivery guarantees for most use cases. Ordering is limited (SQS FIFO for ordered delivery, with throughput caps). Good choice if you’re already on a cloud provider and want to reduce ops burden.

For most products that are not operating at Kafka-scale, SQS or a managed Pub/Sub service reduces operational overhead significantly. The routing flexibility of RabbitMQ is useful when you need complex exchange topologies. Kafka is worth the operational cost when you need log retention, replay, or very high throughput.

Common Mistakes Teams Make

Adopting EDA as the default. Every service-to-service call becomes a message. This maximizes decoupling and maximizes operational complexity simultaneously. Use events for fan-out and workload isolation. Use synchronous calls for simple request/response where you need the result immediately.

Not versioning events. An event schema is a public contract. When you change it (rename a field, add a required field, remove a field), you break every consumer that has not been updated. Version your events from the start: order.placed.v1, order.placed.v2. Maintain backward compatibility for at least one version while consumers migrate.

// Versioned event schema — consumers declare which versions they handle
type OrderPlacedV1 = {
  version: "v1";
  orderId: string;
  userId: string;
  totalCents: number;
};

type OrderPlacedV2 = {
  version: "v2";
  orderId: string;
  userId: string;
  totalCents: number;
  currencyCode: string;  // added in v2
  lineItems: { sku: string; quantity: number; unitCents: number }[];  // restructured in v2
};

type OrderPlacedEvent = OrderPlacedV1 | OrderPlacedV2;

function normalizeOrderPlaced(event: OrderPlacedEvent): OrderPlacedV2 {
  if (event.version === "v1") {
    return {
      version: "v2",
      orderId: event.orderId,
      userId: event.userId,
      totalCents: event.totalCents,
      currencyCode: "USD",  // default for legacy events
      lineItems: [],         // not available in v1 — handle downstream
    };
  }
  return event;
}

Treating events like commands. An event is OrderPlaced, not ProcessOrder. If your events are named as commands, you’re building an RPC system with extra steps, not an event-driven system. The distinction matters: commands have exactly one handler and you care whether they succeed. Events have zero or many handlers and the producer does not care what happens downstream.

Skipping the outbox. Publishing events directly from application code after a database write will eventually drop events during outages or crashes. For business-critical events, use the transactional outbox pattern.

A Decision Framework

Before adding a message broker to your architecture, answer these questions:

Do you have a concrete fan-out problem? One write triggers multiple independent downstream actions, and you want to add more over time without touching the producer. If yes, pub/sub is likely worth it.

Do you have a workload isolation problem? A slow processing step is blocking a faster upstream step, or you need to absorb traffic spikes without scaling the processor. If yes, a queue solves this directly.

Do you need a full audit history or time-travel queries? Regulatory requirements, debugging needs, or business logic that depends on history. If yes, event sourcing earns its complexity. If no, it almost certainly does not.

Is your team ready to operate it? Dead letter queues, consumer lag monitoring, idempotency handling, distributed tracing, broker availability. These are new operational concerns. If your team does not yet have runbooks for these, adding EDA will slow you down before it helps you.

Could you solve the problem with a well-placed database trigger or CDC? Change data capture (tools like Debezium) can emit events from your existing database without requiring you to change application code. For integrations that are purely about reacting to data changes, CDC is often simpler than a full EDA overhaul.

If you answer yes to the first two questions and your team is operationally ready, EDA is probably the right call for that specific flow. If you answered no to the first two questions, a message bus will add complexity without a proportional benefit.

Closing Thoughts

Event-driven architecture earns its complexity in specific situations: decoupled fan-out, workload isolation, and domains where audit history is a first-class requirement. It is not a general upgrade to synchronous architectures.

The move that pays off most consistently is to design services with clean boundaries first and use direct synchronous calls. When you hit a concrete fan-out problem or a workload isolation problem, introduce a message bus for that specific flow. You’ll have a clear sense of what problem you’re solving and what you’re trading away to solve it.

The engineers who are happiest with their event-driven systems are the ones who introduced it to solve a specific pain point, not the ones who adopted it wholesale as an architectural philosophy. Start with the pain point. The pattern follows.

More in System Design

How Amazon Aurora Works Internally: The Log-Is-the-Database Architecture, Quorum Writes, and Storage-Compute Separation Behind Cloud-Native SQL
System Design ·

How Amazon Aurora Works Internally: The Log-Is-the-Database Architecture, Quorum Writes, and Storage-Compute Separation Behind Cloud-Native SQL

A deep dive into Aurora's storage-compute separation, the log-is-the-database design that pushes redo log processing to storage nodes, quorum-based replication across 6 copies in 3 AZs, protection group architecture, fast cloning, Serverless v2 scaling mechanics, and honest tradeoffs versus RDS, self-managed PostgreSQL, CockroachDB, and Cloud Spanner.

How CockroachDB Works Internally: Distributed SQL, Raft Consensus Per Range, and the Architecture Behind Serializable Transactions at Global Scale
System Design ·

How CockroachDB Works Internally: Distributed SQL, Raft Consensus Per Range, and the Architecture Behind Serializable Transactions at Global Scale

A deep dive into CockroachDB's internals: the 512MB range-based data model with automatic splitting, per-range Raft replication, MVCC timestamp ordering with hybrid logical clocks, DistSQL physical planning, serializable transactions with timestamp refreshes, closed timestamps for follower reads, online schema changes using multi-version schema descriptors, and production considerations for hotspots and write amplification.

How Container Runtimes Work Internally: Namespaces, Cgroups, and the OCI Stack From Docker Run to Process Isolation
System Design ·

How Container Runtimes Work Internally: Namespaces, Cgroups, and the OCI Stack From Docker Run to Process Isolation

Containers are not virtual machines and they are not magic. They are a thin composition of Linux kernel primitives: namespaces, cgroups, and a layered filesystem. This article traces exactly what happens between docker run and a running process, covering the OCI spec, containerd, runc, overlay filesystems, and container networking at the kernel level.

How YugabyteDB Works Internally: DocDB Storage, Tablet Splitting, and the Dual API Architecture That Scales PostgreSQL Horizontally
System Design ·

How YugabyteDB Works Internally: DocDB Storage, Tablet Splitting, and the Dual API Architecture That Scales PostgreSQL Horizontally

A deep dive into YugabyteDB internals covering the DocDB storage engine built on RocksDB LSM trees, per-tablet Raft consensus, YSQL and YCQL dual query layers, distributed MVCC with hybrid logical clocks, automatic tablet splitting and rebalancing, xCluster multi-region replication, and production considerations for hotspots, connection pooling, and schema design.