Event-Driven Architecture in Practice: Message Brokers, Event Sourcing, and the Trade-offs Nobody Warns You About

Event-driven architecture shows up in system design discussions as the answer to coupling, scalability, and distributed coordination. Sometimes it is the right answer. Sometimes it is a distributed systems complexity tax that a CRUD API and a Postgres transaction would have handled more reliably in one-tenth the code. This article covers the mechanics you need to use EDA effectively: how message brokers actually differ, how event sourcing and CQRS fit together, what delivery semantics mean for your code, how to handle failures that do not throw exceptions, and where EDA quietly becomes the wrong tool.

Message Broker Mechanics: Kafka, RabbitMQ, and Cloud-Native Options

Choosing a broker is less about brand preference and more about the delivery model your system requires.

Kafka is a distributed commit log. Producers write to partitioned topics, brokers persist records durably to disk, and consumers read at their own pace by tracking an offset. The broker does not know or care whether a consumer has processed a record. This means replay is free, consumer groups can independently track progress, and you can add a new consumer tomorrow and backfill months of history. The trade-off is that Kafka does not track per-message acknowledgment state. If your consumer dies mid-batch, it reprocesses from its last committed offset. Exactly-once delivery requires coordinated use of Kafka transactions and idempotent producers, which adds operational weight.

RabbitMQ is a message broker in the traditional sense: it routes messages through exchanges to queues, tracks acknowledgment per message, and deletes messages once acknowledged. This is a push model (the broker pushes to consumers) versus Kafka’s pull model. You get fine-grained ack/nack control, built-in dead letter exchange routing, and simpler exactly-once semantics through publisher confirms and consumer acks. The trade-off is that once a message is acknowledged and deleted, replay requires reprocessing from the upstream source. There is no inherent audit log.

SQS/SNS (AWS-native) handles the operational burden of running a broker. SQS is a simple queue with at-least-once delivery, visibility timeouts, and a dead letter queue configuration. SNS is a pub/sub fanout layer that can push to multiple SQS queues, HTTP endpoints, or Lambda functions. You pay per API call and get managed scaling, but you give up Kafka’s replay semantics and RabbitMQ’s routing flexibility.

// Kafka producer with idempotent write and transaction support
import { Kafka, CompressionTypes } from "kafkajs";

const kafka = new Kafka({ brokers: ["kafka:9092"], clientId: "order-service" });

const producer = kafka.producer({
  idempotent: true,           // enables exactly-once at the producer level
  transactionalId: "order-service-txn",
});

await producer.connect();
await producer.transaction(async (txn) => {
  await txn.send({
    topic: "orders.created",
    compression: CompressionTypes.GZIP,
    messages: [
      {
        key: order.id,
        value: JSON.stringify({ orderId: order.id, customerId: order.customerId, total: order.total }),
        headers: { "event-type": "OrderCreated", "schema-version": "1" },
      },
    ],
  });
  await txn.sendOffsets({
    consumerGroupId: "order-processor",
    topics: [{ topic: "payments.confirmed", partitions: [{ partition: 0, offset: "42" }] }],
  });
  await txn.commit();
});

The key structural choice: if your consumers need replay (analytics pipelines, event sourcing projections, new downstream services) use Kafka or a log-based broker. If you need routing flexibility, priority queues, or want the broker to track per-message state, RabbitMQ fits better. If you are on AWS and want zero operational overhead, SQS/SNS is a reasonable starting point with the caveat that you are accepting at-least-once semantics and must handle idempotency yourself.

Event Sourcing and Event Store Design

Event sourcing stores state as an ordered sequence of immutable events rather than mutable rows. Instead of updating a orders row, you append OrderPlaced, PaymentConfirmed, and OrderShipped events. The current state is the fold of all events.

This has two non-obvious consequences. First, your audit log is the primary data, not an afterthought. Second, rebuilding state from scratch (for projections, migrations, or debugging) is always possible. The cost is that reading current state requires replaying events, which is why CQRS is almost always paired with event sourcing.

// Event store interface with optimistic concurrency control
type StoredEvent = {
  streamId: string;
  version: number;   // monotonically increasing per stream
  type: string;
  data: unknown;
  metadata: { causationId: string; correlationId: string; timestamp: string };
};

interface EventStore {
  append(streamId: string, events: Omit<StoredEvent, "version">[], expectedVersion: number): Promise<void>;
  read(streamId: string, fromVersion?: number): AsyncIterable<StoredEvent>;
}

// Aggregate rehydration
async function rehydrateOrder(store: EventStore, orderId: string): Promise<OrderState> {
  let state: OrderState = { status: "none", items: [], total: 0 };
  for await (const event of store.read(`order-${orderId}`)) {
    state = applyEvent(state, event);
  }
  return state;
}

function applyEvent(state: OrderState, event: StoredEvent): OrderState {
  switch (event.type) {
    case "OrderPlaced":
      return { ...state, status: "placed", items: (event.data as any).items, total: (event.data as any).total };
    case "PaymentConfirmed":
      return { ...state, status: "paid" };
    case "OrderShipped":
      return { ...state, status: "shipped" };
    default:
      return state;
  }
}

The expectedVersion parameter in append is your optimistic concurrency guard. If two concurrent processes both read version 5 and try to append at version 5, the second one fails with a concurrency conflict rather than silently winning.

Projection rebuilds are the operational reality nobody warns you about. When you add a new read model or fix a bug in an existing projection, you need to replay every event in the store. For a system with millions of events per stream across thousands of streams, this replay can take hours. You need a rebuild strategy before you hit this in production: checkpoint tables to resume interrupted replays, parallel stream processing, and the ability to serve the old projection until the new one is caught up.

CQRS as the Read Side

Command Query Responsibility Segregation separates the write path (commands that mutate state, validated against the aggregate) from the read path (queries that serve pre-computed views). In an event-sourced system, events published after a successful write drive projection updates.

Write side:                       Read side (projections):
  Command                           OrderSummaryProjection
    -> Aggregate rehydrate             { orderId, status, customerName, total }
    -> Validate + apply events
    -> Append to event store        CustomerOrdersProjection
    -> Publish to event bus            [{ orderId, status, createdAt }]

The practical trade-off: your read models are eventually consistent with your write model. If a customer places an order and immediately refreshes their order list, the projection might not have updated yet. For most UIs this is acceptable. For billing or inventory systems where reads drive business-critical decisions, you need to think carefully about whether to query the write side directly for specific strong-consistency reads.

Saga Patterns: Choreography vs. Orchestration

A saga is a sequence of local transactions across multiple services, where each step publishes an event that triggers the next, and compensating transactions handle rollback on failure.

Choreography: each service listens for events and decides what to do next. There is no central coordinator. Services are loosely coupled at the cost of distributed behavior that is hard to visualize. When something goes wrong, finding why a saga stalled requires correlating events across multiple services and logs.

Orchestration: a single saga orchestrator holds the state machine and issues commands to each service. Services respond with success or failure events. The orchestrator decides what happens next. The behavior is centralized and inspectable; you can query the orchestrator for the current state of any in-flight saga. The cost is that the orchestrator becomes a coordination bottleneck and a single place for saga logic to accumulate.

// Orchestrated saga state machine for order fulfillment
type SagaState =
  | { step: "reserve-inventory"; orderId: string }
  | { step: "charge-payment"; orderId: string; inventoryReservationId: string }
  | { step: "ship-order"; orderId: string; paymentIntentId: string }
  | { step: "completed"; orderId: string }
  | { step: "compensating"; orderId: string; failedAt: string };

async function processOrderSagaEvent(state: SagaState, event: SagaEvent): Promise<SagaState> {
  switch (state.step) {
    case "reserve-inventory":
      if (event.type === "InventoryReserved") {
        await commandBus.send("charge-payment", { orderId: state.orderId, amount: event.data.amount });
        return { step: "charge-payment", orderId: state.orderId, inventoryReservationId: event.data.reservationId };
      }
      if (event.type === "InventoryUnavailable") {
        await commandBus.send("cancel-order", { orderId: state.orderId, reason: "out-of-stock" });
        return { step: "compensating", orderId: state.orderId, failedAt: "reserve-inventory" };
      }
      return state;
    // ... remaining steps
    default:
      return state;
  }
}

For complex, long-running workflows with multiple compensation paths, orchestration wins on debuggability. For simple event chains (A emits, B reacts, B emits, C reacts), choreography keeps the code simpler and avoids the orchestrator dependency.

Delivery Semantics

Three delivery guarantees exist in theory. Only two are practical without significant coordination cost.

At-most-once: the message is delivered zero or one times. If the broker or consumer crashes before acknowledgment, the message is lost. Use this for metrics, heartbeats, and non-critical notifications where data loss is acceptable.

At-least-once: the message is delivered one or more times. If the consumer crashes after processing but before acknowledgment, the broker redelivers. This is the default for most brokers and most production systems. Your consumers must be idempotent.

Exactly-once: the message is processed exactly once. This requires coordination between the producer, broker, and consumer (Kafka’s transactional API is the cleanest implementation). The operational complexity is real, and it often makes more sense to implement at-least-once with idempotent consumers than to use exactly-once end-to-end.

Dead Letter Queues and Poison Message Handling

A poison message is one that consistently causes consumer errors. Without a dead letter queue (DLQ), the consumer retries forever, blocking the queue, and all subsequent messages stall behind the poison one.

The pattern: set a max delivery count (typically 3-5 retries with exponential backoff). After exhausting retries, route to the DLQ. Monitor the DLQ. Alert on growth. Every message in the DLQ is a data loss risk and a business process that did not complete.

// SQS consumer with exponential backoff retry and DLQ routing
// (DLQ configuration is on the SQS queue itself; consumer implements backoff)
async function processMessage(message: SQSMessage): Promise<void> {
  const attemptCount = parseInt(message.attributes.ApproximateReceiveCount, 10);
  const backoffMs = Math.min(1000 * Math.pow(2, attemptCount - 1), 30_000);

  try {
    await handleEvent(JSON.parse(message.Body));
    await sqs.deleteMessage({ QueueUrl: QUEUE_URL, ReceiptHandle: message.ReceiptHandle }).promise();
  } catch (err) {
    if (attemptCount >= 5) {
      // The SQS redrive policy will handle DLQ routing after maxReceiveCount
      // Log the failure with full context for DLQ processing
      logger.error({ messageId: message.MessageId, body: message.Body, error: err }, "Message exceeded retry limit, routing to DLQ");
    } else {
      // Let visibility timeout expire after backoff delay to trigger redelivery
      await sqs.changeMessageVisibility({
        QueueUrl: QUEUE_URL,
        ReceiptHandle: message.ReceiptHandle,
        VisibilityTimeout: Math.ceil(backoffMs / 1000),
      }).promise();
    }
    throw err;
  }
}

DLQ processing is a first-class operational concern. Build a dashboard showing DLQ depth by queue, message age, and error type. Designate ownership for DLQ triage. Automate alerting on DLQ non-empty for critical queues. The messages in the DLQ are not garbage to be cleared; they are failures waiting for a fix.

Idempotency

At-least-once delivery means your consumer will receive duplicates under normal operating conditions (network retries, consumer restarts, broker rebalances). Idempotency is not optional.

The standard approach: use a stable deduplication key from the event itself (event ID, order ID, or a deterministic hash of the payload) and track processed IDs in a store with a TTL longer than your maximum redelivery window.

// Redis-backed idempotency check for event consumers
async function processEventIdempotently(
  event: { id: string; type: string; data: unknown },
  handler: (event: { id: string; type: string; data: unknown }) => Promise<void>
): Promise<void> {
  const key = `processed-event:${event.id}`;
  const alreadyProcessed = await redis.set(key, "1", "EX", 86400, "NX"); // 24h TTL, set-if-not-exists

  if (alreadyProcessed === null) {
    // Key already existed, this is a duplicate
    logger.info({ eventId: event.id }, "Duplicate event skipped");
    return;
  }

  try {
    await handler(event);
  } catch (err) {
    // Remove the key so the event can be retried on redelivery
    await redis.del(key);
    throw err;
  }
}

Watch for the subtle case: if the handler succeeds but the consumer crashes before acknowledging the message, the deduplication key is already set, so the redelivery will be silently skipped. This is usually correct behavior (the work was done), but it requires that your handler commits its state change before returning, not after. If you write to a database and then return, the success path is safe. If you chain multiple side effects, partial completion is possible.

Backpressure and Consumer Lag Monitoring

Consumer lag is the number of messages in the queue (or Kafka partition offset gap) that have not yet been processed. A lag spike can mean the consumer is slow, the consumer is down, or a bad deployment increased processing time. Any of these is a problem.

Kafka exposes consumer group lag via the kafka-consumer-groups.sh tool and the __consumer_offsets topic. Export it to Prometheus via kafka-exporter or jmx-exporter. Alert on lag exceeding a threshold relative to your SLA (not a fixed number; a queue that processes 10K messages per minute has a very different “high lag” threshold than one processing 100 per minute).

Backpressure is what happens when your consumer cannot keep up. Without explicit backpressure mechanisms, unbounded message buffers accumulate in memory, consumers run out of heap, and the crash loop begins. The solution depends on the broker:

• Kafka: reduce the max.poll.records setting so each poll fetches fewer records. Pause partitions (consumer.pause()) while processing to stop fetching. This is pull-based by design and gives you natural backpressure hooks.
• RabbitMQ: set prefetch (QoS) to a small number. The broker will not push more messages to a consumer until it acknowledges the current batch.
• SQS: limit concurrency at the consumer level. Long polling with a concurrency limit is simpler than it sounds.

Message Broker Trade-offs

Dimension	Kafka	RabbitMQ	SQS/SNS
Delivery model	Pull, log-based	Push, queue-based	Pull (SQS), push (SNS)
Default semantics	At-least-once	At-least-once	At-least-once
Exactly-once support	Yes (transactions)	Via publisher confirms + idempotent consumers	No (idempotent consumers required)
Message replay	Yes (configurable retention)	No (messages deleted on ack)	No
Routing flexibility	Topic/partition, no complex routing	Exchanges, bindings, routing keys	SQS queues + SNS topics/subscriptions
Operational overhead	High (ZooKeeper/KRaft, JVM, partitioning)	Medium (Erlang runtime, cluster setup)	Zero (fully managed)
Consumer lag visibility	Native (consumer group offsets)	Management plugin / shovel	CloudWatch metrics
Message ordering	Guaranteed within partition	Guaranteed within single queue	Per-FIFO-queue; standard queues unordered
Throughput ceiling	Millions/sec per partition (append-only log)	Hundreds of thousands/sec	Millions/sec (standard queue)
Cost model	Infrastructure + operational	Infrastructure + operational	Per-request ($0.40/million)

When EDA Is the Wrong Choice

Event-driven architecture adds genuine complexity. It is the wrong choice when:

Your domain does not have natural events. If your system is a CRUD API for managing configuration or user profiles, the “events” you produce are artificial and add no value. A REST endpoint with a Postgres transaction is simpler, more debuggable, and more reliable.

You need strong consistency. EDA is inherently eventually consistent. If your business logic requires reading your own writes immediately with a guarantee, distributing that logic across async event handlers creates correctness problems that compensating transactions do not fully solve.

Your team has not operated async systems before. The debugging model is fundamentally different. A synchronous HTTP call throws an exception. An async consumer failure is a message in a DLQ, a lag counter ticking up, and a business process that silently stopped. The observability and operational tooling required to operate EDA safely takes time to build and maintain.

You have fewer than three services. Two services passing events through Kafka is often a distributed monolith with Kafka overhead. A direct HTTP call or a shared Postgres table is simpler for intra-team communication until you have clear ownership boundaries that justify the broker.

Your events are really just API calls in disguise. Sending a SendEmailCommand event is a request-response pattern dressed as an event. If the producer needs to know whether the email was sent, this is not an event; it is an RPC call over a queue, and you should evaluate whether you actually need the decoupling the broker provides.

Production Considerations

Schema evolution is a first-class concern. Events are immutable and stored forever. Adding a field is safe. Removing or renaming a field breaks consumers that already deployed. Use a schema registry (Confluent Schema Registry for Kafka, or roll your own validation middleware) with forward and backward compatibility checks enforced in CI before any event schema change merges.

Correlation IDs must be propagated. Every event should carry a correlationId (the root request or user action that started the chain) and a causationId (the event or command that directly caused this event). Without these, distributed tracing across an event-driven system is a log-correlation nightmare.

Test your DLQ handling, not just your happy path. In integration tests, inject messages that fail validation, messages that fail during handler execution, and messages that fail after partial state change. Verify that the DLQ receives them, that the alerts fire, and that manual replay produces correct results.

Consumer lag alerting must be per-consumer-group, not global. A single slow consumer in one consumer group should not alert for a fast consumer in another. Set per-group lag thresholds based on each group’s processing SLA.

Idempotency TTLs must exceed your maximum redelivery window. If your broker can redeliver a message 48 hours after original delivery (due to partition leadership changes, consumer restarts, or broker maintenance), a 24-hour deduplication TTL will allow duplicates through. Know your broker’s redelivery ceiling and set TTLs accordingly.

Capacity plan for projection rebuilds. Before you ship event sourcing to production, answer two questions: how long does a full projection rebuild take at current event volume, and what is the plan for serving read traffic during a rebuild? If the answer is “we do not know” and “nothing, reads fail,” these are production incidents waiting to happen.

Monitor event schema version distributions in flight. When you deploy a schema migration with a new schema version, old and new event versions will coexist in the stream during the rollout window. Consumers must handle both. Track the percentage of each schema version in the stream so you know when it is safe to drop old version handling.

EDA done well produces systems that scale, audit, and recover gracefully. EDA done carelessly produces distributed debugging nightmares with more failure modes than the synchronous system it replaced. The difference is mostly in the operational discipline around delivery semantics, idempotency, and observability, not in the framework or broker choice.