Designing a Customer Data Platform: Event Collection, Identity Resolution, and Unified Customer Profiles at Scale

A customer data platform sits at the intersection of data engineering, distributed systems, and privacy law. The business case is straightforward: users touch your product across browsers, native apps, and backend workflows, and you need a single coherent view of who they are and what they did. Getting there requires solving some genuinely hard problems: collecting events reliably from untrusted clients, resolving identities across anonymous and authenticated sessions, merging partial profiles without destroying history, and deleting specific data on demand without rebuilding everything.

This article walks through each layer with production-grade TypeScript and honest tradeoffs. It is not a review of Segment, RudderStack, or Jitsu. It is the architecture you would build if you decided to own it yourself.

The Four Core Problems

Before touching code, it helps to be precise about what a CDP actually needs to solve:

1. Event collection — capturing what users do, from client-side SDKs and server-side sources, reliably and without data loss.
2. Identity resolution — matching anonymous sessions to known users, and known users across devices.
3. Profile unification — merging fragmented profile records into a single, queryable representation.
4. Audience computation — evaluating segment membership rules against profiles in real time and in batch.

Privacy is a cross-cutting concern that touches all four.

Event Collection

Client-Side SDK

Client-side event collection has two constraints that server-side doesn’t: you cannot trust the client, and you will lose events. Ad blockers, browser crashes, bad network conditions, and users navigating away mid-request all create gaps. Your SDK needs to handle these gracefully.

// packages/sdk-browser/src/client.ts
interface EventPayload {
  event: string;
  properties: Record<string, unknown>;
  anonymousId: string;
  userId?: string;
  timestamp: string;
  context: {
    page: { url: string; title: string; referrer: string };
    userAgent: string;
    locale: string;
  };
}

class CDPClient {
  private queue: EventPayload[] = [];
  private flushInterval: ReturnType<typeof setInterval> | null = null;
  private anonymousId: string;

  constructor(private writeKey: string, private endpoint: string) {
    this.anonymousId = this.getOrCreateAnonymousId();
    this.setupFlush();
    this.setupBeforeUnload();
  }

  track(event: string, properties: Record<string, unknown> = {}): void {
    const payload: EventPayload = {
      event,
      properties,
      anonymousId: this.anonymousId,
      userId: this.getCurrentUserId(),
      timestamp: new Date().toISOString(),
      context: this.buildContext(),
    };
    this.queue.push(payload);

    // Flush immediately for high-priority events
    if (event === "Order Completed" || event === "Subscription Started") {
      this.flush();
    }
  }

  identify(userId: string, traits: Record<string, unknown> = {}): void {
    this.setCurrentUserId(userId);
    this.track("$identify", { userId, traits });
    // Flush synchronously on identify — this links anonymous to known
    this.flush();
  }

  private async flush(): Promise<void> {
    if (this.queue.length === 0) return;
    const batch = this.queue.splice(0, 100);
    try {
      await fetch(`${this.endpoint}/v1/batch`, {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `Basic ${btoa(this.writeKey + ":")}`,
        },
        body: JSON.stringify({ batch }),
        keepalive: true, // survives page unload
      });
    } catch {
      // Re-queue on failure, cap at 500 to avoid memory bloat
      this.queue.unshift(...batch.slice(0, 500 - this.queue.length));
    }
  }

  private setupBeforeUnload(): void {
    window.addEventListener("visibilitychange", () => {
      if (document.visibilityState === "hidden") this.flush();
    });
  }

  private getOrCreateAnonymousId(): string {
    const key = "cdp_anon_id";
    let id = localStorage.getItem(key);
    if (!id) {
      id = crypto.randomUUID();
      localStorage.setItem(key, id);
    }
    return id;
  }
}

Key decisions here: keepalive: true on the fetch keeps the request alive after navigation. Flushing on visibilitychange rather than beforeunload is more reliable on mobile. The queue cap prevents memory exhaustion on flaky networks.

Server-Side Ingestion

Server-side events arrive through an HTTP API. The ingestion layer needs to validate, enrich, and hand off to the processing pipeline without blocking the response.

// apps/collector/src/ingest.ts
import { Hono } from "hono";
import { zValidator } from "@hono/zod-validator";
import { z } from "zod";

const EventSchema = z.object({
  batch: z.array(
    z.object({
      event: z.string().min(1).max(255),
      properties: z.record(z.unknown()),
      anonymousId: z.string().uuid().optional(),
      userId: z.string().optional(),
      timestamp: z.string().datetime(),
      context: z.record(z.unknown()).optional(),
    })
  ).max(100),
});

const app = new Hono();

app.post("/v1/batch", zValidator("json", EventSchema), async (c) => {
  const writeKey = parseWriteKey(c.req.header("Authorization"));
  if (!writeKey) return c.json({ error: "Unauthorized" }, 401);

  const source = await resolveSource(writeKey);
  if (!source) return c.json({ error: "Unknown write key" }, 401);

  const { batch } = c.req.valid("json");

  // Enrich server-side
  const enriched = batch.map((event) => ({
    ...event,
    sourceId: source.id,
    receivedAt: new Date().toISOString(),
    ip: c.req.header("CF-Connecting-IP") ?? c.req.header("X-Forwarded-For"),
  }));

  // Publish to queue — do not await, respond immediately
  await eventQueue.publishBatch(enriched);

  return c.json({ ok: true, count: enriched.length });
});

The collector should be stateless and horizontally scalable. It does two things: validate the payload and publish to a durable queue (Kafka, SQS, or Pub/Sub). It does not write to the database. That happens downstream.

Identity Resolution

This is the hardest part of a CDP. You have anonymous sessions, device IDs, email addresses, phone numbers, and external IDs, and you need to figure out which ones belong to the same person.

The Identity Graph

The core data structure is a graph where nodes are identifiers and edges represent “same person” relationships. The graph is append-only: you add edges, you never remove them (except for deletion requests, which are a special case).

// packages/identity/src/graph.ts
interface IdentityNode {
  id: string;
  type: "anonymous_id" | "user_id" | "email" | "phone" | "device_id" | "external_id";
  value: string;
  sourceId: string;
  createdAt: Date;
}

interface IdentityEdge {
  nodeAId: string;
  nodeBId: string;
  confidence: number; // 0.0 to 1.0
  method: "deterministic" | "probabilistic";
  createdAt: Date;
}

class IdentityGraph {
  // Find the canonical profile ID for any identifier
  async resolveProfileId(
    type: IdentityNode["type"],
    value: string
  ): Promise<string | null> {
    // BFS through the graph to find the root profile
    const node = await this.db.identityNodes.findOne({ type, value });
    if (!node) return null;

    const visited = new Set<string>();
    const queue = [node.id];

    while (queue.length > 0) {
      const current = queue.shift()!;
      if (visited.has(current)) continue;
      visited.add(current);

      const edges = await this.db.identityEdges.findAll({
        where: { nodeAId: current },
      });

      for (const edge of edges) {
        if (!visited.has(edge.nodeBId)) queue.push(edge.nodeBId);
      }
    }

    // The profile ID is derived from the connected component
    return this.computeCanonicalId([...visited]);
  }

  async mergeIdentities(
    nodeAType: IdentityNode["type"],
    nodeAValue: string,
    nodeBType: IdentityNode["type"],
    nodeBValue: string,
    method: "deterministic" | "probabilistic",
    confidence: number
  ): Promise<void> {
    const [nodeA, nodeB] = await Promise.all([
      this.getOrCreateNode(nodeAType, nodeAValue),
      this.getOrCreateNode(nodeBType, nodeBValue),
    ]);

    // Skip if already in the same component
    const [profileA, profileB] = await Promise.all([
      this.resolveProfileId(nodeAType, nodeAValue),
      this.resolveProfileId(nodeBType, nodeBValue),
    ]);

    if (profileA === profileB && profileA !== null) return;

    await this.db.identityEdges.insert({
      nodeAId: nodeA.id,
      nodeBId: nodeB.id,
      confidence,
      method,
      createdAt: new Date(),
    });

    // Trigger profile merge
    await this.profileMergeQueue.publish({ profileA, profileB });
  }
}

Deterministic matching happens when two identifiers are explicitly linked by user action: a user logs in after an anonymous session, for example. You call mergeIdentities("anonymous_id", anonId, "user_id", userId, "deterministic", 1.0).

Probabilistic matching uses signals like device fingerprint similarity, IP address proximity, and behavioral patterns to infer that two anonymous sessions are the same person. Confidence scores below a threshold (typically 0.7) should not trigger a merge automatically; they should queue for review or be used only for downstream scoring, not hard profile merges.

The Identify Event

When a user authenticates, you emit an identify event that links the current anonymous ID to the authenticated user ID. This is the most common identity resolution trigger.

// packages/identity/src/resolver.ts
async function processIdentifyEvent(event: {
  anonymousId?: string;
  userId: string;
  traits: Record<string, unknown>;
  sourceId: string;
}): Promise<void> {
  if (event.anonymousId) {
    await graph.mergeIdentities(
      "anonymous_id",
      event.anonymousId,
      "user_id",
      event.userId,
      "deterministic",
      1.0
    );
  }

  // Upsert traits onto the profile
  const profileId = await graph.resolveProfileId("user_id", event.userId);
  if (profileId) {
    await profileStore.upsertTraits(profileId, event.traits, event.sourceId);
  }
}

Unified Customer Profile Construction

Once identity resolution links identifiers to profile IDs, you need to build the actual profile. The challenge is that traits come from multiple sources with different precedence rules, and some fields can conflict.

// packages/profiles/src/merger.ts
interface ProfileTrait {
  value: unknown;
  sourceId: string;
  updatedAt: Date;
  confidence: number;
}

interface UnifiedProfile {
  profileId: string;
  identifiers: Record<string, string[]>; // type -> [values]
  traits: Record<string, ProfileTrait>;
  eventCount: number;
  firstSeen: Date;
  lastSeen: Date;
  computedSegments: string[];
}

class ProfileMerger {
  // Merge two profiles when identity resolution determines they're the same person
  async mergeProfiles(
    profileIdA: string,
    profileIdB: string
  ): Promise<UnifiedProfile> {
    const [profileA, profileB] = await Promise.all([
      this.store.getProfile(profileIdA),
      this.store.getProfile(profileIdB),
    ]);

    if (!profileA || !profileB) {
      throw new Error("Cannot merge: one or both profiles not found");
    }

    // Canonical ID is the older profile (preserves history)
    const [canonical, secondary] =
      profileA.firstSeen <= profileB.firstSeen
        ? [profileA, profileB]
        : [profileB, profileA];

    const mergedTraits: Record<string, ProfileTrait> = { ...secondary.traits };

    // Canonical profile traits win on conflict, unless secondary is newer
    for (const [key, trait] of Object.entries(canonical.traits)) {
      const existing = mergedTraits[key];
      if (!existing || trait.updatedAt >= existing.updatedAt) {
        mergedTraits[key] = trait;
      }
    }

    const merged: UnifiedProfile = {
      profileId: canonical.profileId,
      identifiers: this.mergeIdentifiers(canonical.identifiers, secondary.identifiers),
      traits: mergedTraits,
      eventCount: canonical.eventCount + secondary.eventCount,
      firstSeen: canonical.firstSeen,
      lastSeen: new Date(
        Math.max(canonical.lastSeen.getTime(), secondary.lastSeen.getTime())
      ),
      computedSegments: [],
    };

    await this.store.saveProfile(merged);
    await this.store.redirectProfile(secondary.profileId, canonical.profileId);
    await this.segmentEngine.recompute(canonical.profileId);

    return merged;
  }
}

The redirectProfile call writes a pointer from the secondary profile ID to the canonical one. Any downstream system that holds the old ID will follow the redirect on next lookup. This is cleaner than trying to update all references in place.

Audience Segmentation

Segments are boolean rules evaluated against profiles. You need two modes: batch recomputation (rebuild all segment memberships nightly) and real-time evaluation (update segment membership when a profile changes).

// packages/segments/src/engine.ts
type SegmentRule =
  | { type: "trait"; key: string; operator: "eq" | "neq" | "gt" | "lt" | "contains" | "exists"; value?: unknown }
  | { type: "event"; name: string; operator: "performed" | "not_performed"; within?: string }
  | { type: "and"; rules: SegmentRule[] }
  | { type: "or"; rules: SegmentRule[] }
  | { type: "not"; rule: SegmentRule };

class SegmentEngine {
  async evaluate(profile: UnifiedProfile, rule: SegmentRule): Promise<boolean> {
    switch (rule.type) {
      case "trait": {
        const trait = profile.traits[rule.key];
        if (!trait) return rule.operator === "exists" ? false : false;
        const v = trait.value;
        switch (rule.operator) {
          case "eq": return v === rule.value;
          case "neq": return v !== rule.value;
          case "gt": return typeof v === "number" && v > (rule.value as number);
          case "lt": return typeof v === "number" && v < (rule.value as number);
          case "contains": return typeof v === "string" && v.includes(rule.value as string);
          case "exists": return true;
        }
      }
      case "event": {
        const since = rule.within
          ? new Date(Date.now() - parseDuration(rule.within))
          : undefined;
        const count = await this.eventStore.countEvents(
          profile.profileId,
          rule.name,
          since
        );
        return rule.operator === "performed" ? count > 0 : count === 0;
      }
      case "and":
        return (await Promise.all(rule.rules.map((r) => this.evaluate(profile, r)))).every(Boolean);
      case "or":
        return (await Promise.all(rule.rules.map((r) => this.evaluate(profile, r)))).some(Boolean);
      case "not":
        return !(await this.evaluate(profile, rule.rule));
    }
  }

  async recompute(profileId: string): Promise<void> {
    const profile = await this.profileStore.getProfile(profileId);
    if (!profile) return;

    const segments = await this.segmentStore.getAllSegments();
    const memberships = await Promise.all(
      segments.map(async (seg) => ({
        segmentId: seg.id,
        member: await this.evaluate(profile, seg.rule),
      }))
    );

    const joined = memberships.filter((m) => m.member).map((m) => m.segmentId);
    await this.profileStore.updateSegments(profileId, joined);

    // Emit membership change events for downstream sync (CRM, ad platforms, etc.)
    await this.emitSegmentChanges(profileId, profile.computedSegments, joined);
  }
}

For batch recomputation, run this as a background job scanning profiles in pages of 1,000. For real-time, trigger recompute on every profile write. The event-based segment rules ("type": "event") are the expensive part: they require a query against the event store rather than just reading profile traits. Cache aggressively.

Privacy: Consent and GDPR Right to Deletion

Consent Management

Every event must carry consent context. At ingestion time, check the user’s consent record and drop events for processing purposes the user has not consented to.

// packages/privacy/src/consent.ts
type ConsentPurpose = "analytics" | "marketing" | "personalization" | "essential";

interface ConsentRecord {
  profileId: string;
  purposes: Record<ConsentPurpose, boolean>;
  updatedAt: Date;
  version: string; // which consent UI version collected this
}

function filterEventByConsent(
  event: EnrichedEvent,
  consent: ConsentRecord
): EnrichedEvent | null {
  // Map event types to required consent purposes
  const requiredPurpose = getRequiredPurpose(event.event);
  if (!requiredPurpose) return event; // essential events always pass

  if (!consent.purposes[requiredPurpose]) return null;
  return event;
}

GDPR Right to Erasure

Right to deletion is where many CDPs get it wrong. You cannot just delete the profile row. You need to:

1. Delete or anonymize all events associated with the profile.
2. Delete all identity nodes and edges for the profile.
3. Delete the profile itself.
4. Propagate deletion to downstream destinations.
5. Produce a deletion receipt for audit.

// packages/privacy/src/deletion.ts
async function processErasureRequest(
  profileId: string,
  requestId: string
): Promise<ErasureReceipt> {
  // 1. Lock the profile to prevent new writes
  await profileStore.lockProfile(profileId);

  // 2. Collect all identity nodes for this profile
  const identifiers = await identityGraph.getAllIdentifiers(profileId);

  // 3. Delete events — batch delete by profileId
  const deletedEvents = await eventStore.deleteByProfileId(profileId);

  // 4. Remove identity graph nodes and edges
  await identityGraph.deleteProfile(profileId);

  // 5. Delete the profile record
  await profileStore.deleteProfile(profileId);

  // 6. Publish deletion notice to all destinations
  await deletionBus.publish({
    profileId,
    identifiers,
    requestId,
    timestamp: new Date().toISOString(),
  });

  // 7. Write audit receipt to append-only log (do NOT delete this)
  const receipt: ErasureReceipt = {
    requestId,
    profileId,
    deletedEventCount: deletedEvents,
    identifiersRemoved: identifiers.length,
    completedAt: new Date().toISOString(),
  };

  await auditLog.append(receipt);

  return receipt;
}

The audit log is the one thing you cannot delete. You need proof that the deletion happened. Store it in a separate append-only store (write-once S3 bucket, a log table with no DELETE privilege, etc.).

Late-arriving events for a deleted profile must be dropped at ingestion. Check against a deletion blocklist at the point of identity resolution: if the profile ID resolves to a deleted record, discard the event.

Late-Arriving Events and Schema Evolution

Events arrive out of order and sometimes hours or days late. Your processing pipeline must handle both.

For late events, track a receivedAt timestamp separately from the client-supplied timestamp. Use receivedAt for partition routing but timestamp for behavioral ordering. Never trust a client timestamp for anything that affects billing or compliance.

Schema evolution is trickier. Events are write-once, but your profile schema needs to evolve as you add new traits. Two practices that help:

• Store traits as a semi-structured column (JSONB in Postgres, or a document in a document store) rather than rigid columns. This lets you add new traits without migrations.
• Version your segment rule definitions. When a rule references a trait that no longer exists, fail gracefully to false rather than throwing.

Tradeoffs: Build vs. Buy

Dimension	Build	Segment / RudderStack / Jitsu
Time to first event	Weeks	Hours
Identity resolution quality	You control it	Black box
Data residency	Full control	Depends on tier
Cost at scale	Infrastructure cost only	Per-MTU pricing gets expensive fast
GDPR/CCPA compliance surface	You own it	Shared responsibility
Custom segment logic	Arbitrary	Limited by their rule builder
Engineering maintenance	Ongoing	Vendor manages

The per-MTU (monthly tracked user) pricing model of managed CDPs looks reasonable at 10,000 users and becomes painful at 500,000. If your user base is large or growing fast, the math usually favors building at 18-24 months of scale. If your team is small or moving quickly, the managed option is rational for the first year.

A middle path: use RudderStack or Jitsu for event collection (their SDKs are solid and handle the edge cases), but build identity resolution and profile storage yourself. You get reliable ingestion without the per-MTU tax on the expensive parts.

Production Considerations

Profile storage. Postgres with JSONB works well up to a few million profiles. At tens of millions, you will want to shard by profileId or move the trait store to a document database. Keep the identity graph in Postgres (it has good graph traversal with recursive CTEs).

Segment computation. Evaluate trait-based rules synchronously on profile write. Evaluate event-based rules asynchronously in a background job. The synchronous path covers 80% of use cases; the async path handles the expensive 20%.

Fan-out to destinations. Each destination (email platform, ad platform, CRM) should receive segment membership changes via a queue, not a synchronous call. Use the outbox pattern on the profile write to guarantee at-least-once delivery.

Rate limiting at ingestion. A misconfigured SDK can flood your collector with thousands of events per second from a single client. Rate limit by write key (not by IP) and return 429 when the limit is exceeded. The SDK should back off exponentially.

Closing Thought

The identity graph is the core of a CDP, and it is the part that is hardest to get right. Deterministic merges are straightforward; probabilistic merges require careful confidence thresholds and the humility to accept that you will make mistakes. The architecture that survives is one that makes merges reversible (via the redirect chain), deletion complete (via the blocklist), and identity resolution auditable (via edge metadata). Everything else is plumbing.