Designing a Notification System: Push, Email, SMS, and In-App Delivery at Scale

Most systems start with a single sendEmail() call. It works fine until product adds SMS, then push, then in-app. Suddenly you have four integrations that all do roughly the same thing: accept an event, look up a user, render a template, and fire off a request to a third-party vendor. The duplication is obvious. The problems are not: duplicate sends, no rate limiting, no fallback when Twilio returns a 429, no way to know if users are even receiving what you send.

This guide walks through the architecture of a notification system that handles all four channels without collapsing into a pile of vendor-specific if statements.

The Core Architecture

The system has five logical layers:

1. Event ingestion - Accept a structured notification event from any internal service
2. Preference routing - Determine which channels to use for which user
3. Template rendering - Produce the final content per channel
4. Delivery - Send through the appropriate vendor
5. Observability - Track every state transition

// The canonical event shape
interface NotificationEvent {
  eventId: string;           // idempotency key
  type: string;              // "order.shipped", "alert.high-cpu"
  recipientId: string;
  payload: Record<string, unknown>;
  channels?: Channel[];      // explicit override; omit to use preferences
  priority: "critical" | "high" | "normal" | "low";
  dedupKey?: string;         // optional scoped dedup beyond eventId
  ttlSeconds?: number;       // drop if not delivered within this window
}

type Channel = "email" | "sms" | "push" | "in_app";

Callers emit events. They do not decide channels. That decision belongs to the routing layer.

Event Ingestion and Deduplication

Every notification event gets an eventId. The ingestion layer stores it in Redis with a short TTL before processing begins. If the same eventId arrives twice (upstream retry, at-least-once queue delivery), the second is dropped.

async function ingestEvent(event: NotificationEvent): Promise<"accepted" | "duplicate"> {
  const key = `notif:dedup:${event.eventId}`;
  const set = await redis.set(key, "1", "NX", "EX", 86400); // 24h window
  if (!set) return "duplicate";

  await queue.publish("notification.ingested", event);
  return "accepted";
}

The dedupKey field handles business-level deduplication: “don’t send another ‘your storage is 90% full’ email within 24 hours, regardless of how many events the system emits.”

async function checkBusinessDedup(
  recipientId: string,
  dedupKey: string,
  windowSeconds: number
): Promise<boolean> {
  const key = `notif:bdedup:${recipientId}:${dedupKey}`;
  const set = await redis.set(key, "1", "NX", "EX", windowSeconds);
  return set === null; // true = suppressed
}

Preference Routing

Routing answers: for this event type and this user, which channels should actually fire?

The preference model has three layers: system defaults, notification category defaults, and per-user overrides. Critical alerts bypass user preferences entirely.

interface UserPreferences {
  userId: string;
  channels: {
    email: { enabled: boolean; address: string };
    sms: { enabled: boolean; number: string | null };
    push: { enabled: boolean; tokens: string[] };
    in_app: { enabled: boolean };
  };
  quietHours?: { start: string; end: string; tz: string }; // "22:00", "08:00"
  categoryOverrides: Record<string, Partial<UserPreferences["channels"]>>;
}

async function resolveChannels(
  event: NotificationEvent,
  prefs: UserPreferences
): Promise<Channel[]> {
  // Critical notifications ignore user preferences
  if (event.priority === "critical") {
    return ["email", "sms", "push", "in_app"];
  }

  // Explicit override from caller
  if (event.channels) return event.channels;

  const categoryPrefs = prefs.categoryOverrides[event.type] ?? {};
  const resolved: Channel[] = [];

  for (const ch of ["email", "sms", "push", "in_app"] as Channel[]) {
    const base = prefs.channels[ch];
    const override = categoryPrefs[ch];
    const effective = override ?? base;

    if (effective?.enabled) {
      resolved.push(ch);
    }
  }

  // Respect quiet hours for non-critical
  if (event.priority === "normal" || event.priority === "low") {
    if (await isQuietHours(prefs)) {
      return resolved.filter(ch => ch === "in_app"); // only in-app during quiet hours
    }
  }

  return resolved;
}

The routing layer emits one delivery task per resolved channel. Each task is independent. A push failure does not block email.

Template Rendering

Templates are stored in a database (or a CMS, depending on your team). The render step compiles a Mustache or Handlebars template against the event payload. The key constraint: rendering happens before delivery, and the result is stored. If a delivery attempt fails and retries 5 minutes later, it uses the same pre-rendered content, not a re-render. This matters for time-sensitive copy like “Your code expires in 10 minutes.”

interface NotificationTemplate {
  id: string;
  channel: Channel;
  eventType: string;
  locale: string;
  subject?: string;        // email only
  body: string;            // HTML for email, plain for SMS/push
  pushTitle?: string;
}

async function renderTemplate(
  template: NotificationTemplate,
  payload: Record<string, unknown>
): Promise<RenderedContent> {
  const renderFn = Handlebars.compile(template.body);
  const subject = template.subject
    ? Handlebars.compile(template.subject)(payload)
    : undefined;

  return {
    channel: template.channel,
    subject,
    body: renderFn(payload),
    pushTitle: template.pushTitle
      ? Handlebars.compile(template.pushTitle)(payload)
      : undefined,
  };
}

Template rendering failures are fatal for that delivery task. Log the error with the template ID and payload shape, then dead-letter the task. Do not retry a render failure with the same template.

Rate Limiting Per User

Without per-user rate limiting, a noisy upstream service can flood a user’s inbox in minutes. The limit operates at two scopes: per-user total across all channels, and per-user per-channel.

interface RateLimitConfig {
  global: { maxPerHour: number; maxPerDay: number };
  perChannel: Partial<Record<Channel, { maxPerHour: number }>>;
}

async function checkRateLimit(
  userId: string,
  channel: Channel,
  config: RateLimitConfig
): Promise<"allowed" | "limited"> {
  const now = Date.now();
  const hourKey = `rl:${userId}:${Math.floor(now / 3_600_000)}`;
  const dayKey = `rl:${userId}:${Math.floor(now / 86_400_000)}`;
  const channelKey = `rl:${userId}:${channel}:${Math.floor(now / 3_600_000)}`;

  const [hourCount, dayCount, channelCount] = await redis.mget(hourKey, dayKey, channelKey);

  if (Number(hourCount) >= config.global.maxPerHour) return "limited";
  if (Number(dayCount) >= config.global.maxPerDay) return "limited";

  const channelLimit = config.perChannel[channel];
  if (channelLimit && Number(channelCount) >= channelLimit.maxPerHour) return "limited";

  // Increment all counters atomically
  const pipeline = redis.pipeline();
  pipeline.incr(hourKey).expire(hourKey, 7200);
  pipeline.incr(dayKey).expire(dayKey, 172800);
  pipeline.incr(channelKey).expire(channelKey, 7200);
  await pipeline.exec();

  return "allowed";
}

Rate-limited notifications are not dropped. They are queued with a delay. For SMS and push, this often means a 1-hour bucket. For in-app, they accumulate and display when the user next opens the product.

Delivery and the Retry-Fallback Chain

Each channel has its own delivery adapter. The adapters are thin wrappers over vendor SDKs. Their only job: send the content, map the vendor response to a normalized result, and throw on non-retryable errors.

type DeliveryResult =
  | { status: "delivered"; vendorId: string }
  | { status: "failed"; retryable: boolean; reason: string };

interface ChannelAdapter {
  deliver(content: RenderedContent, recipient: string): Promise<DeliveryResult>;
}

The orchestrator runs the retry-fallback chain. Retryable failures (network timeout, 429, 503) go back on the queue with exponential backoff. Non-retryable failures (invalid address, account suspended) skip to the fallback channel.

async function deliverWithFallback(
  task: DeliveryTask,
  adapters: Record<Channel, ChannelAdapter>,
  fallbackChain: Channel[]
): Promise<void> {
  for (const channel of fallbackChain) {
    const adapter = adapters[channel];
    let attempts = 0;
    const maxAttempts = 4;

    while (attempts < maxAttempts) {
      attempts++;
      const result = await adapter.deliver(task.content, task.recipient);

      if (result.status === "delivered") {
        await recordDelivery(task, channel, result.vendorId);
        return;
      }

      if (!result.retryable) {
        await recordFailure(task, channel, result.reason, "non_retryable");
        break; // move to next channel in fallback chain
      }

      const backoff = Math.min(1000 * 2 ** attempts, 30_000);
      await sleep(backoff);
    }
  }

  // All channels exhausted
  await deadLetter(task, "all_channels_failed");
}

The fallback chain is configured per event priority. A critical alert might chain: push -> sms -> email. A marketing notification might not fallback at all.

Tradeoffs

Decision	Option A	Option B	When to prefer A
Template storage	DB with versioning	Code-deployed templates	Frequent copy changes by non-engineers
Render timing	Pre-render at ingest	Render at delivery	Time-sensitive content (OTPs, expiry windows)
Dedup scope	eventId only	eventId + business key	When upstream emits multiple events for one logical action
Rate limit storage	Redis counters	Token bucket in-process	Multi-instance deployments (always Redis)
Fallback strategy	Auto-fallback to next channel	Notify on first failure	Critical alerts vs. marketing digests
Queue backend	BullMQ (Redis)	SQS / Pub/Sub	Self-hosted vs. cloud-managed tradeoff

Production Considerations

Vendor outages. All four channel vendors will go down at some point. The retry-fallback chain handles transient outages. For sustained outages (30+ minutes), you need a circuit breaker that stops trying the failed vendor and routes directly to the fallback. Track vendor health in a separate background process, not inline with delivery.

Push token management. iOS and Android tokens expire and rotate. Every delivery attempt should check the vendor response for “invalid token” errors and delete the token from your store immediately. Sending to stale tokens wastes quota and inflates your error rates.

Email deliverability. High volumes of transactional email require a warm-up period for new sending domains. Separate transactional from marketing sends at the IP and subdomain level. Marketing email failures (spam trap hits) should not affect transactional reputation.

In-app ordering. In-app notifications are typically queried by the client on page load. They need a stable sort (by createdAt descending), pagination, and a read/unread toggle that is idempotent. Store them in a table with a composite index on (recipient_id, created_at) and a soft-delete for dismissal.

Observability. Every state transition should emit a structured log: ingested, deduplicated, routed, rate_limited, render_failed, delivery_attempted, delivered, failed, dead_lettered. From these events you can build the metrics that matter: delivery rate per channel, p95 time from event to delivery, dead-letter rate by event type.

interface NotificationLifecycleEvent {
  eventId: string;
  recipientId: string;
  channel?: Channel;
  state: NotificationState;
  ts: number;
  meta?: Record<string, unknown>;
}

type NotificationState =
  | "ingested"
  | "deduplicated"
  | "routed"
  | "rate_limited"
  | "render_failed"
  | "delivery_attempted"
  | "delivered"
  | "failed"
  | "dead_lettered";

async function emitLifecycleEvent(event: NotificationLifecycleEvent): Promise<void> {
  // Write to your event store (ClickHouse, BigQuery, Datadog, etc.)
  await analytics.track("notification.lifecycle", event);
}

Set up two alerts: one on dead-letter rate spiking above baseline (broken template or misconfigured vendor), and one on p95 delivery latency for critical-priority events crossing a threshold you care about (typically 30 seconds for push, 60 seconds for SMS).

The Piece Most Systems Skip

The data model for user preferences is almost always an afterthought. It gets added as a JSON column in the users table with no validation, no schema versioning, and no way to run bulk preference changes when you add a new channel.

Treat preferences as first-class entities. Version the schema. Run migrations in the background when you add a new notification category. Give users a dedicated preferences endpoint that they can call from any product surface. The cost is low. The alternative is a preference system that breaks whenever product adds a new event type, which is often.

Notification systems are not glamorous. They are load-bearing infrastructure. Getting dedup, rate limiting, and fallback right early saves you from chasing duplicate-send incidents at 2am.