Designing a Tokenization Service: PCI Compliance, Format-Preserving Encryption, and Secure Data Vaults at Scale

Most engineering teams treat PCI compliance as a procurement problem. Buy a certified gateway, route card numbers through it, and assume the scope problem is solved. That assumption collapses the first time you need to store a card for recurring billing, display the last four digits on a receipt, or pass a card reference through five internal microservices. Suddenly “send it to the gateway” is not an answer.

Tokenization is the architectural answer. The idea is deceptively simple: replace a sensitive value with a non-sensitive stand-in that can be stored, logged, and passed around freely, while the real value lives in one place with strict access controls. But the implementation details are where compliance and performance either come together or fall apart.

This guide covers the architecture of a production tokenization service: what makes a token safe, how format-preserving encryption changes the tradeoff, how to design the vault, how to authorize detokenization, and how to keep latency acceptable at scale.

Tokenization vs. Encryption

Before the architecture, the distinction matters.

Encryption transforms a value using a key. With the key, anyone can reverse the operation. If the key is compromised, all encrypted values are compromised. The ciphertext also does not resemble the original, which breaks any system that expects a specific format (a 16-digit card number, for example).

Tokenization replaces a value with a token that has no mathematical relationship to the original. There is no key that transforms a token back into a card number. The mapping exists only in the vault. A compromised token reveals nothing about the underlying data, and a compromised vault that leaks the mapping table is the only meaningful attack surface.

This distinction matters for PCI DSS scope. PCI DSS scope includes every system that stores, processes, or transmits cardholder data. A system that only ever sees tokens is out of scope, provided the tokens are generated and managed correctly. This is the core value proposition: tokenization shrinks your PCI compliance perimeter by moving card data out of your application systems.

The practical implication is that your order service, your analytics database, your customer support tools, and your data warehouse can all store the token. None of them are in PCI scope. Only the tokenization service itself, and the vault it talks to, are in scope.

Token Types and Their Tradeoffs

There are three token types used in practice. The right choice depends on what you need to do with the token.

Random Tokens

A random token is a cryptographically random value (typically 128 or 256 bits, base64url-encoded) stored in a mapping table alongside the original value.

import { randomBytes } from "crypto";

async function tokenize(
  cardNumber: string,
  tenantId: string,
  db: VaultDb
): Promise<string> {
  // check if a token already exists for this card + tenant
  const existing = await db.findToken({ cardNumber, tenantId });
  if (existing) return existing.token;

  const token = randomBytes(32).toString("base64url"); // 43-char URL-safe token
  await db.insertMapping({ token, cardNumber, tenantId, createdAt: new Date() });
  return token;
}

Random tokens are the most secure option because there is no structural information in the token itself. An attacker who obtains a token learns only that it is a token. The attack surface is limited to the vault mapping table.

The limitation is that they look nothing like what they replace. A system expecting a 16-digit card number cannot store a random token without a schema change. This is usually fine for new systems but creates migration friction in legacy applications.

Deterministic Tokens

A deterministic token is derived from the sensitive value using a keyed pseudorandom function (PRF). The same input always produces the same output, but the output is not reversible without the key.

import { createHmac } from "crypto";

function deterministicToken(
  cardNumber: string,
  tenantId: string,
  secretKey: Buffer
): string {
  // HMAC-SHA256 with a per-tenant namespace to prevent cross-tenant collisions
  const hmac = createHmac("sha256", secretKey);
  hmac.update(`${tenantId}:${cardNumber}`);
  return hmac.digest("base64url");
}

The benefit of deterministic tokens is deduplication without a vault lookup. The same card presented twice in the same tenant produces the same token. This matters for fraud detection (identify duplicate card use across sessions) and analytics (count unique cards without decryption).

The risk is that an attacker who knows the token space could perform offline attacks. For card numbers, the space is small enough that brute force is feasible if the PRF key is compromised. You also lose the ability to rotate tokens without re-deriving all of them.

Use deterministic tokens when your application needs to group or deduplicate by sensitive value and you accept the key-rotation complexity.

Format-Preserving Encryption (FPE)

Format-preserving encryption produces ciphertext in the same format as the plaintext. A 16-digit card number in, a 16-digit token out. The result can pass Luhn validation if you implement that, which means legacy systems often require no schema changes.

The standard algorithms are FF1 and FF3-1, both specified in NIST SP 800-38G. FF1 uses CBC-mode AES internally. FF3-1 uses a tweakable block cipher construction. Both operate over arbitrary alphabets and lengths, not just binary data.

// FF1 splits the plaintext into two halves and runs 10 Feistel rounds.
// Use a spec-compliant library (node-ff1, or your HSM's PKCS#11 FPE API)
// rather than implementing this yourself. The structure is:
//
//   for i in 0..9:
//     C = (numRadix(A) + PRF(numRadix(B), roundKey(key, tweak, i))) mod radix^len(A)
//     A = B
//     B = str(C, radix, len(A))
//   return A + B
//
// The output has the same length and alphabet as the input.
function fpeEncrypt(plaintext: string, key: Buffer, tweak: Buffer): string {
  // delegate to a NIST SP 800-38G compliant implementation
  return ff1Encrypt({ key, tweak, radix: 10, plaintext });
}

FPE tokens are reversible with the key, which means the security model is closer to encryption than to a token vault. Compromising the FPE key compromises all tokens derived from it. This does not eliminate the vault, but it changes what the vault stores: you can store the FPE token alongside metadata without storing the original, and detokenize by decrypting rather than by vault lookup.

The tweak parameter is important. FF1 and FF3-1 accept a tweak (additional data that influences the ciphertext without being a key). Using a per-merchant or per-tenant tweak means the same card number produces different FPE tokens in different contexts, preventing cross-context correlation even if an attacker obtains tokens from multiple tenants.

Tradeoffs

Property	Random Token	Deterministic Token	FPE (FF1/FF3)
Format preservation	No	No	Yes
Vault required	Yes (mapping table)	No (or small)	Optional (store FPE token)
Reversibility	Vault lookup only	PRF key	AES key
Key rotation	Easy (re-tokenize gradually)	Hard (re-derive all)	Hard (re-encrypt all)
Deduplication	Vault lookup required	Implicit	Possible via consistent tweak
Legacy system compatibility	Low	Low	High
Brute-force resistance	High	Medium (key-dependent)	High (key-dependent)
PCI Council guidance	Preferred for new systems	Acceptable with controls	Accepted with AES-256 + NIST-compliant algorithm

For new systems, random tokens are the most defensible choice. For legacy systems where a schema change is not feasible, FPE lets you drop in a tokenization layer without touching the surrounding infrastructure.

Token Vault Architecture

The vault is a database that maps tokens to their original values. Its design determines your security posture, your compliance surface, and your detokenization latency.

Vault Database Schema

// vault-schema.ts — the core mapping table
interface VaultEntry {
  token: string;          // primary key, indexed
  ciphertext: string;     // encrypted card number (envelope-encrypted, see below)
  keyVersion: number;     // which KMS key version encrypted this entry
  tenantId: string;       // tenant isolation
  createdAt: Date;
  lastAccessedAt: Date;   // for token expiry and audit
  metadata: {
    bin: string;          // first 6 digits (not sensitive, useful for routing)
    lastFour: string;     // last 4 digits (not sensitive, used for display)
    expiryMonth: number;
    expiryYear: number;
    cardBrand: string;    // VISA, MC, AMEX — derived at tokenization time
  };
}

Notice what is stored in plaintext: the BIN (first six digits), the last four digits, the expiry, and the card brand. PCI DSS allows these in scope systems when stored without the full PAN. Storing them alongside the token means detokenization is not required to display “Visa ending in 4242” on a UI. That detail matters for latency: a checkout page that needs to show saved cards does not need to call the detokenization endpoint.

Envelope Encryption with KMS and HSM

The ciphertext in the vault is not directly encrypted with a KMS key. That would mean one compromised KMS call exposes all vault entries. Instead, use envelope encryption.

import { KMSClient, GenerateDataKeyCommand, DecryptCommand } from "@aws-sdk/client-kms";

const kms = new KMSClient({ region: "us-east-1" });

async function encryptForVault(
  plaintext: string,
  tenantId: string
): Promise<{ ciphertext: string; encryptedDataKey: string; keyVersion: number }> {
  // 1. ask KMS for a data key — a fresh AES-256 key encrypted under the CMK
  const { Plaintext: dataKey, CiphertextBlob: encryptedDataKey } =
    await kms.send(new GenerateDataKeyCommand({
      KeyId: `alias/vault-cmk-${tenantId}`,  // per-tenant CMK for isolation
      KeySpec: "AES_256",
    }));

  // 2. encrypt the sensitive value with the plaintext data key (in-memory only)
  const iv = randomBytes(12);
  const cipher = createCipheriv("aes-256-gcm", dataKey!, iv);
  const encrypted = Buffer.concat([cipher.update(plaintext, "utf8"), cipher.final()]);
  const authTag = cipher.getAuthTag();

  // 3. store the encrypted data key alongside the ciphertext
  //    the plaintext data key never leaves memory and is never stored
  return {
    ciphertext: Buffer.concat([iv, authTag, encrypted]).toString("base64"),
    encryptedDataKey: Buffer.from(encryptedDataKey!).toString("base64"),
    keyVersion: CURRENT_KEY_VERSION,
  };
}

async function decryptFromVault(
  ciphertext: string,
  encryptedDataKey: string,
  tenantId: string
): Promise<string> {
  // 1. ask KMS to decrypt the data key — KMS enforces IAM authorization here
  const { Plaintext: dataKey } = await kms.send(new DecryptCommand({
    CiphertextBlob: Buffer.from(encryptedDataKey, "base64"),
    KeyId: `alias/vault-cmk-${tenantId}`,
  }));

  // 2. decrypt the ciphertext with the recovered data key
  const raw = Buffer.from(ciphertext, "base64");
  const iv = raw.subarray(0, 12);
  const authTag = raw.subarray(12, 28);
  const encrypted = raw.subarray(28);

  const decipher = createDecipheriv("aes-256-gcm", dataKey!, iv);
  decipher.setAuthTag(authTag);
  return decipher.update(encrypted) + decipher.final("utf8");
}

The pattern here: KMS holds the Customer Master Key (CMK) and never exposes it. Every encrypt call generates a fresh data key, encrypted under the CMK. The data key encrypts the actual card number. To decrypt, you give KMS the encrypted data key and get back the plaintext data key, which you use locally. The card number’s ciphertext and the encrypted data key are both stored in the vault. An attacker who steals the vault database still cannot decrypt anything without KMS access.

For the highest-assurance deployments, the KMS CMK should itself be backed by an HSM (AWS CloudHSM, Azure Dedicated HSM, or a physical appliance). HSM-backed keys never leave the hardware boundary. Key material cannot be exported, and cryptographic operations happen inside tamper-resistant hardware.

Detokenization Authorization

Tokenization is only half the system. Detokenization (turning a token back into a card number) is where authorization matters.

The rule is: only systems in PCI scope should detokenize, and only for specific, auditable reasons.

interface DetokenizeRequest {
  token: string;
  tenantId: string;
  requestorId: string;      // service or user making the request
  purpose: DetokenizePurpose;
  contextId: string;        // order ID, charge attempt ID — for audit correlation
}

type DetokenizePurpose =
  | "charge_attempt"        // payment processor call
  | "card_update"           // customer updating their saved card
  | "compliance_export"     // PCI-authorized data export
  | "fraud_investigation";  // authorized fraud team request

async function detokenize(req: DetokenizeRequest): Promise<string> {
  // 1. validate the requestor is authorized for this purpose
  await assertAuthorization(req.requestorId, req.purpose, req.tenantId);

  // 2. look up the vault entry
  const entry = await vaultDb.findByToken(req.token, req.tenantId);
  if (!entry) throw new TokenNotFoundError(req.token);

  // 3. decrypt
  const cardNumber = await decryptFromVault(
    entry.ciphertext,
    entry.encryptedDataKey,
    req.tenantId
  );

  // 4. write an immutable audit record before returning
  await auditLog.record({
    event: "detokenize",
    token: req.token,
    tenantId: req.tenantId,
    requestorId: req.requestorId,
    purpose: req.purpose,
    contextId: req.contextId,
    timestamp: new Date(),
    // never log the card number itself
  });

  await vaultDb.updateLastAccessed(req.token);

  return cardNumber;
}

A few things worth calling out here. The authorization check happens before any vault lookup. A failed authorization should not tell the caller whether the token exists. Return the same error regardless.

The purpose field is not just documentation. It drives authorization rules: a fraud investigator can detokenize any token in their tenant, but a charge service can only detokenize tokens associated with an in-flight charge attempt (verifiable via the contextId). These rules are enforced in assertAuthorization, not trusted from the caller.

The audit record is written before the card number is returned. If the write fails, the function throws and the caller does not receive the card number. This ordering is important for compliance: every detokenization must be auditable.

Tenant Isolation

Multi-tenant tokenization services need hard isolation at every layer.

At the data layer, every vault entry is scoped to a tenantId. Queries that cross tenant boundaries are impossible at the query level, not just at the application level. A row-level security policy in PostgreSQL (or equivalent) enforces this:

-- row-level security policy on the vault table
CREATE POLICY vault_tenant_isolation ON vault_entries
  USING (tenant_id = current_setting('app.current_tenant_id'));

-- each connection sets this at session start, before any query
SET LOCAL app.current_tenant_id = '{{tenantId}}';

At the encryption layer, per-tenant CMKs in KMS mean that a misconfigured IAM policy granting access to one tenant’s CMK does not expose another tenant’s data. The KMS key ARN is a security boundary, not just a naming convention.

At the network layer, the tokenization service should not be reachable from general application infrastructure. Route only services with legitimate detokenization needs. Everything else should receive a 403 before the request is processed.

Performance and Caching

Vault lookups add latency. A tokenize call involves a KMS GenerateDataKey round trip (roughly 5-15ms) plus a database write. A detokenize call involves a KMS Decrypt call plus a database read.

For tokenization-heavy workloads (card-on-file storage during signup), the KMS call dominates. Options:

Data key caching: cache a plaintext data key in memory for a short window (30-60 seconds) to batch multiple tokenizations under one KMS call. AWS Encryption SDK provides this out of the box. The risk is that a process crash loses a cache of plaintext data keys, which is acceptable for a short window.

Async vault writes: accept the card number, return a token immediately, and write the vault entry asynchronously. The token is generated before the vault write completes. This works only if token generation is deterministic (FPE or HMAC-based), which introduces the key-rotation tradeoffs described earlier.

For detokenization, the right answer is usually not to cache the plaintext. Caching decrypted card numbers in application memory or Redis is precisely the attack surface tokenization is designed to eliminate. The one exception is a short-lived, in-process cache during a transaction that requires multiple operations on the same card, cleared after the transaction completes.

class DetokenizationSession {
  private cache = new Map<string, { value: string; expiresAt: number }>();
  private readonly ttlMs = 30_000; // 30 seconds max lifetime for any cached value

  async get(token: string, tenantId: string, request: DetokenizeRequest): Promise<string> {
    const cached = this.cache.get(token);
    if (cached && cached.expiresAt > Date.now()) {
      return cached.value;
    }

    const value = await detokenize(request);
    this.cache.set(token, { value, expiresAt: Date.now() + this.ttlMs });
    return value;
  }

  clear(): void {
    this.cache.clear(); // call this at the end of every transaction
  }
}

This session object lives in the request scope, not in a shared cache. Its lifetime is the transaction. After the payment attempt completes (success or failure), clear() removes the card number from memory.

Compliance Boundary Reduction

The tangible output of a well-designed tokenization service is a smaller PCI scope. Here is what that means in practice.

Before tokenization, your PCI scope includes everything that touches card numbers: your application servers, your database, your logging pipeline, your monitoring tools, your data warehouse, your customer support portal. Auditors examine all of it.

After tokenization, your PCI scope includes: the tokenization service, the vault database, the KMS/HSM, and any system that calls the detokenize endpoint. Your application servers store only tokens. Your logging pipeline captures only tokens. Your analytics platform queries only tokens.

The remaining PCI scope is smaller, better-defined, and easier to harden. You can apply stricter network controls, more aggressive audit logging, and more frequent penetration testing to a smaller surface area.

One architectural mistake undermines this: logging the card number anywhere in the tokenization request path. Even if the log entry is short-lived, it puts your logging infrastructure in scope. Use structured logging with explicit field allowlists, and validate that card number fields are never passed to log functions.

function sanitizeForLogging(req: TokenizeRequest): Record<string, unknown> {
  return {
    tenantId: req.tenantId,
    requestId: req.requestId,
    cardBrand: detectBrand(req.cardNumber),  // derived, not raw
    binPrefix: req.cardNumber.slice(0, 6),   // first 6 are not sensitive
    // cardNumber is explicitly excluded
  };
}

Production Considerations

Token expiry: tokens that are never used should expire. An unused token that exists for five years is an unnecessary liability. Implement expiry based on last-access date, with a grace period for dormant accounts. For card-on-file, expire tokens that have not been used in 18-24 months.

Key rotation: envelope encryption makes key rotation feasible without downtime. Generate a new CMK version in KMS. Re-encrypt vault entries lazily on access, or in a background job, writing the new encryptedDataKey and keyVersion back to the vault. The old CMK version remains available for decryption until all entries are migrated. This is why keyVersion is stored on every vault entry.

Vault high availability: the vault is in the critical path of every payment. A vault database failure means no charges can be processed. Use a synchronously replicated database (PostgreSQL with streaming replication and a read replica, Aurora with Multi-AZ, or equivalent) with a tested failover procedure. Your RTO for the vault should match your RTO for the payment system itself.

Detokenization rate limits: apply rate limits to the detokenize endpoint per requestor and per tenant. Unusual spikes in detokenization volume are an early indicator of a compromised service account attempting bulk data extraction. Alert when a single requestor detokenizes more than N tokens outside of a known bulk operation pattern.

HSM performance: HSM-backed KMS operations are slower than software-backed operations, sometimes by an order of magnitude. Benchmark your expected tokenization and detokenization volume against HSM throughput before committing to HSM-backed keys. For most use cases, software-backed CMKs with appropriate IAM controls are sufficient; HSM backing is required only when your compliance posture or threat model demands it.

Closing Thoughts

Tokenization is not a product you buy. It is an architectural decision you make once, with consequences that propagate through every downstream system. The token type you choose determines your legacy compatibility, your key rotation story, and your deduplication capabilities. The vault design determines your PCI scope. The authorization model determines whether detokenization is auditable or a free-for-all.

Get the fundamentals right: random tokens for new systems, FPE for legacy compatibility, envelope encryption everywhere, per-tenant key isolation, audit before return. Everything else is operational detail that you can iterate on. The decisions that are hard to change later are the token type and the encryption scheme. Make those deliberately.