System Design ·

Designing a Tenant Provisioning System: Automated Onboarding, Resource Allocation, and Namespace Isolation for Multi-Tenant SaaS

Manual tenant setup is a ticking clock for SaaS teams. This article covers the full system design of automated tenant provisioning: schema isolation, resource quotas, async workflows, rollback, and observability.

Designing a Tenant Provisioning System: Automated Onboarding, Resource Allocation, and Namespace Isolation for Multi-Tenant SaaS

When a new customer signs up for your SaaS product, a lot has to happen before they can do anything useful. A database schema or dedicated database needs to exist. Configuration has to be injected. Resource limits have to be set. Secrets have to be generated and stored. Auth roles have to be created. Maybe a dedicated subdomain or a Kubernetes namespace gets spun up.

If you are doing any of this manually, you already know the pain. Someone on your team gets paged or Slack-messaged every time a customer upgrades to a plan that requires dedicated resources. Your staging environment has drifted from production because the checklist got followed inconsistently. A customer waited 45 minutes after completing payment because a human had to run a setup script.

This article walks through the system design of a tenant provisioning service that handles all of this automatically, correctly, and with enough observability that you can tell when it goes wrong.

The Core Decision: Shared vs. Dedicated Resources

Before writing any code, you need to decide how much isolation each tenant tier gets. This shapes almost everything else.

There are three common models:

Shared schema, shared tables. All tenants live in the same tables, discriminated by a tenant_id column. Simplest to operate, hardest to keep isolated. Row-level security (RLS) in Postgres handles the access control, but you are one missed WHERE clause from a data leak. Quotas are enforced in application code, not at the infrastructure layer.

Schema-per-tenant. Each tenant gets a dedicated schema within a shared database server. Postgres makes this straightforward. You get real isolation at the query level, easier backup and restore per tenant, and a natural namespace for migrations. The main cost is that schema-level migrations now fan out across N schemas.

Database-per-tenant. Full isolation. Each tenant gets their own database instance, potentially their own compute. This unlocks regulatory compliance scenarios (HIPAA, SOC2 with strict data residency requirements) and lets you offer genuine SLA guarantees. The operational overhead is significant.

Most SaaS products use a hybrid: shared schema for free and starter tiers, schema-per-tenant or database-per-tenant for enterprise. The provisioning system needs to handle all three.

type IsolationLevel = "shared" | "schema" | "dedicated";

interface TenantPlan {
  planId: string;
  isolationLevel: IsolationLevel;
  resourceQuota: ResourceQuota;
  features: string[];
}

interface ResourceQuota {
  maxUsers: number;
  maxStorageGb: number;
  maxApiCallsPerMonth: number;
  maxComputeUnits: number; // normalized CPU/memory budget
}

Provisioning as a Workflow, Not a Function

The mistake most teams make early on is writing tenant provisioning as a single synchronous function: create schema, seed data, inject config, done. This breaks the moment any step is slow or fallible, which is always.

A provisioning workflow has multiple steps that can fail independently:

  1. Validate the request (plan exists, customer payment confirmed)
  2. Reserve a namespace or schema name
  3. Create database schema or database instance
  4. Run schema migrations
  5. Seed required baseline data
  6. Generate and store tenant secrets
  7. Create auth roles and permissions
  8. Inject environment-specific configuration
  9. Allocate and register resource quotas
  10. Register tenant in the control plane
  11. Send welcome notification
  12. Mark tenant as active

Each of these steps can fail. Steps 3 through 6 involve external systems. Steps 5 through 9 may need to be retried independently without re-running earlier steps. If step 8 fails, you should not create a new schema on retry: you should resume from where you left off.

This is a textbook case for a durable execution model. You can implement this with a workflow engine like Temporal, with a simple state machine persisted to your own database, or with a queue-based approach where each step emits an event consumed by the next step.

For most teams, a state machine approach with a jobs table is sufficient and avoids introducing a new infrastructure dependency:

type ProvisioningStatus =
  | "pending"
  | "validating"
  | "schema_creation"
  | "migration"
  | "seeding"
  | "secrets"
  | "auth_setup"
  | "config_injection"
  | "quota_allocation"
  | "control_plane_registration"
  | "notification"
  | "active"
  | "failed"
  | "rolling_back";

interface TenantProvisioningJob {
  id: string;
  tenantId: string;
  planId: string;
  status: ProvisioningStatus;
  isolationLevel: IsolationLevel;
  schemaName?: string;
  databaseId?: string;
  checkpoints: ProvisioningCheckpoint[];
  failedStep?: string;
  failureReason?: string;
  createdAt: Date;
  updatedAt: Date;
  completedAt?: Date;
}

interface ProvisioningCheckpoint {
  step: ProvisioningStatus;
  completedAt: Date;
  metadata?: Record<string, unknown>;
}

The provisioning runner reads the current status, executes the next step, and updates the checkpoint before moving on. If it crashes mid-step, the next pick-up resumes from the last completed checkpoint.

async function runProvisioningStep(
  job: TenantProvisioningJob
): Promise<TenantProvisioningJob> {
  const nextStep = getNextStep(job.status);

  try {
    const result = await executeStep(nextStep, job);
    return await advanceJob(job, nextStep, result.metadata);
  } catch (err) {
    const isRetryable = classifyError(err);
    if (isRetryable && job.retryCount < MAX_RETRIES) {
      return await scheduleRetry(job, err);
    }
    return await failJob(job, nextStep, err);
  }
}

Schema Creation and Migration

For schema-per-tenant isolation, schema creation is the first real infrastructure operation. This needs to be idempotent.

async function createTenantSchema(
  db: DatabaseConnection,
  schemaName: string
): Promise<void> {
  // Idempotent: safe to call multiple times
  await db.query(`CREATE SCHEMA IF NOT EXISTS ${sanitizeIdentifier(schemaName)}`);

  // Grant schema access to the application role
  await db.query(
    `GRANT USAGE ON SCHEMA ${sanitizeIdentifier(schemaName)} TO app_role`
  );
  await db.query(
    `ALTER DEFAULT PRIVILEGES IN SCHEMA ${sanitizeIdentifier(schemaName)}
     GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_role`
  );
}

function sanitizeIdentifier(name: string): string {
  // Only allow alphanumeric and underscores; prefix with tenant_ to avoid collisions
  const safe = name.replace(/[^a-z0-9_]/gi, "_").toLowerCase();
  if (!/^[a-z_]/.test(safe)) {
    throw new Error(`Invalid schema name: ${name}`);
  }
  return `"tenant_${safe}"`;
}

After schema creation, you need to run migrations scoped to that schema. If you are using a tool like Flyway or Liquibase, you can parameterize the schema name. If you are managing migrations yourself, the pattern is to set the search_path before running migration SQL:

async function migrateSchema(
  db: DatabaseConnection,
  schemaName: string,
  migrations: Migration[]
): Promise<void> {
  await db.query(`SET search_path TO ${sanitizeIdentifier(schemaName)}, public`);

  for (const migration of migrations) {
    const alreadyApplied = await checkMigrationApplied(db, migration.version);
    if (alreadyApplied) continue;

    await db.query("BEGIN");
    try {
      await db.query(migration.sql);
      await recordMigration(db, migration.version);
      await db.query("COMMIT");
    } catch (err) {
      await db.query("ROLLBACK");
      throw err;
    }
  }
}

One thing to get right early: the migration tracking table (schema_migrations or equivalent) needs to live inside the tenant schema, not in a global schema, or you will end up with a messy mixed-ownership situation.

Resource Quota Enforcement

Setting quotas is not the same as enforcing them. You can record a quota of 1,000 API calls per month, but if nothing checks that quota at request time, it is decorative.

There are two common enforcement points:

In the API gateway. Before the request reaches your application, a middleware or plugin checks the tenant’s current usage against their quota. This is the most reliable pattern because it prevents overuse from reaching the database at all. If you are running on something like Cloudflare Workers, you can check a KV or Durable Object for the tenant’s current usage counter.

In the application layer. The application itself checks usage before executing expensive operations. This is easier to implement but adds latency and can be bypassed by application bugs.

For most SaaS products, a hybrid approach works well: the API gateway enforces hard limits on request rate, while the application enforces soft limits on resource consumption (storage, compute units, user count).

interface QuotaStore {
  getTenantUsage(tenantId: string, metric: string): Promise<number>;
  incrementUsage(tenantId: string, metric: string, delta: number): Promise<void>;
  getQuotaLimit(tenantId: string, metric: string): Promise<number>;
}

async function enforceQuota(
  store: QuotaStore,
  tenantId: string,
  metric: string,
  required: number
): Promise<void> {
  const [usage, limit] = await Promise.all([
    store.getTenantUsage(tenantId, metric),
    store.getQuotaLimit(tenantId, metric),
  ]);

  if (usage + required > limit) {
    throw new QuotaExceededError(tenantId, metric, usage, limit);
  }
}

Quota state needs to be stored somewhere fast (Redis or an equivalent) and reset on a billing cycle. The provisioning system is responsible for writing the initial quota record when the tenant is created. On plan upgrade, the provisioning system also updates the quota limits, which is a simpler workflow than initial provisioning.

Namespace and Environment Isolation

Beyond database schemas, tenants often need isolation at other layers: Kubernetes namespaces (for tenants who run their own compute), S3 bucket prefixes or dedicated buckets, separate log streams, and separate secret namespaces in something like AWS Secrets Manager or Vault.

For Kubernetes-based workloads, a namespace-per-tenant model gives you resource quotas enforced by the scheduler, network policies scoped to the namespace, and RBAC isolation. Provisioning a Kubernetes namespace as part of tenant onboarding looks like this:

import { KubeConfig, CoreV1Api, V1Namespace } from "@kubernetes/client-node";

async function provisionKubernetesNamespace(
  tenantId: string,
  quotas: ResourceQuota
): Promise<void> {
  const kc = new KubeConfig();
  kc.loadFromDefault();
  const k8sApi = kc.makeApiClient(CoreV1Api);

  const namespace: V1Namespace = {
    apiVersion: "v1",
    kind: "Namespace",
    metadata: {
      name: `tenant-${tenantId}`,
      labels: {
        "app.kubernetes.io/managed-by": "provisioning-service",
        "tenant-id": tenantId,
      },
    },
  };

  await k8sApi.createNamespace(namespace);

  // Apply resource quotas
  const resourceQuota = {
    apiVersion: "v1",
    kind: "ResourceQuota",
    metadata: {
      name: "tenant-quota",
      namespace: `tenant-${tenantId}`,
    },
    spec: {
      hard: {
        "requests.cpu": `${quotas.maxComputeUnits * 100}m`,
        "limits.memory": `${quotas.maxComputeUnits * 256}Mi`,
      },
    },
  };

  await k8sApi.createNamespacedResourceQuota(
    `tenant-${tenantId}`,
    resourceQuota
  );
}

For S3, the isolation is at the prefix or bucket level. Prefix-level isolation is cheaper but requires careful IAM policy design to prevent cross-tenant reads. Bucket-per-tenant is costier but simpler from a policy standpoint. The provisioning system should write the IAM policy or bucket policy as part of setup and store the resulting ARN or path in the tenant configuration record.

Configuration Injection

Every tenant has configuration: their database connection string (or schema name), their S3 prefix, their quota limits, their feature flags, their custom domain, their branding settings. This configuration needs to be available to every service that handles requests for that tenant.

The naive approach is to pass configuration via environment variables. This breaks for multi-tenant systems because you cannot have N sets of environment variables for N tenants in a single process.

The better approach is a configuration service or a structured config lookup at request time. When a request arrives with a tenant ID (from a JWT, a subdomain, or a header), your middleware looks up the tenant’s configuration from a fast store (Redis, a CDN edge KV, or a local cache with TTL).

interface TenantConfig {
  tenantId: string;
  schemaName: string;
  databaseUrl: string;
  s3BucketPrefix: string;
  featureFlags: Record<string, boolean>;
  plan: string;
  quotas: ResourceQuota;
  customDomain?: string;
}

async function getTenantConfig(
  cache: RedisClient,
  db: DatabaseConnection,
  tenantId: string
): Promise<TenantConfig> {
  const cacheKey = `tenant:config:${tenantId}`;
  const cached = await cache.get(cacheKey);
  if (cached) return JSON.parse(cached) as TenantConfig;

  const config = await db.queryOne<TenantConfig>(
    "SELECT * FROM tenants WHERE id = $1",
    [tenantId]
  );

  if (!config) throw new TenantNotFoundError(tenantId);

  await cache.setex(cacheKey, 300, JSON.stringify(config)); // 5-minute TTL
  return config;
}

The provisioning system writes this config record as one of its final steps, before marking the tenant as active. Until that record exists, the tenant cannot be served.

Rollback on Failure

What happens when provisioning fails at step 8 of 11? You have created a schema, run migrations, seeded data, generated secrets, created auth roles, and injected config. Then secret storage fails. Now you have a partially provisioned tenant: database resources exist but the tenant cannot be served.

Rollback is the right answer, but it has to be explicit. There is no magic here. Each provisioning step needs a corresponding cleanup operation, and the rollback sequence needs to run those cleanups in reverse order.

const ROLLBACK_HANDLERS: Partial<Record<ProvisioningStatus, RollbackFn>> = {
  schema_creation: async (job) => {
    await dropTenantSchema(job.schemaName!);
  },
  auth_setup: async (job) => {
    await deleteAuthRoles(job.tenantId);
  },
  secrets: async (job) => {
    await purgeSecrets(job.tenantId);
  },
  config_injection: async (job) => {
    await deleteTenantConfig(job.tenantId);
  },
  quota_allocation: async (job) => {
    await deleteQuotaRecord(job.tenantId);
  },
};

async function rollbackProvisioning(
  job: TenantProvisioningJob
): Promise<void> {
  const completedSteps = job.checkpoints.map((c) => c.step).reverse();

  for (const step of completedSteps) {
    const handler = ROLLBACK_HANDLERS[step];
    if (!handler) continue;

    try {
      await handler(job);
    } catch (err) {
      // Log but continue rollback; partial rollback is better than no rollback
      logger.error({ err, step, tenantId: job.tenantId }, "Rollback step failed");
    }
  }
}

Rollback failures are worth special handling. If rollback itself fails, you have leaked resources: a schema that exists but is not connected to any tenant, or secrets that were generated but never registered. You need to alert on this and have a human process for cleaning up orphaned resources. A daily job that scans for schemas with no corresponding tenant record is a reasonable safety net.

Tradeoffs at Scale

ConcernShared SchemaSchema-per-TenantDedicated DB
IsolationRLS only, app-enforcedSchema boundary, per-tenant migrationsFull OS and network isolation
Migration complexitySingle migrationFan-out to N schemasFan-out to N databases
Provisioning speedSub-second1-5 seconds30s to 5 minutes
Compliance fitLimitedModerateStrong (HIPAA, SOC2)
Cost per tenantNear-zero marginalLow marginalHigh marginal
Rollback on failureConfig onlyDrop schemaDrop database

The provisioning system architecture should be the same regardless of which isolation model you choose: a durable workflow, idempotent steps, explicit rollback handlers, and quota records written at the end. What changes is the implementation of each step.

Monitoring Provisioning Health

Provisioning is not a background job you can ignore. A failed provisioning means a customer who paid and cannot log in. You need:

A current-state dashboard. How many tenants are in each status? How many have been pending for more than 10 minutes? Surface these as metrics, not just as database counts. A Grafana panel over a Prometheus counter works. So does a simple query that your on-call rotation can run.

Alerting on stuck provisioning. A job that has not advanced in 15 minutes in a non-terminal state is stuck. This should page someone.

Duration histograms per step. Which step is slow? If schema_creation consistently takes 8 seconds on large Postgres instances, you learn this from a histogram, not from random sampling.

Rollback rate as a signal. If 5% of provisioning attempts are rolling back, something in your infrastructure is flaky. The rollback rate is a leading indicator of a systemic problem.

function recordProvisioningMetrics(
  job: TenantProvisioningJob,
  step: ProvisioningStatus,
  durationMs: number,
  outcome: "success" | "failure" | "retry"
): void {
  metrics.histogram("provisioning.step.duration_ms", durationMs, {
    step,
    isolation_level: job.isolationLevel,
    plan: job.planId,
  });

  metrics.increment("provisioning.step.outcomes", {
    step,
    outcome,
  });

  if (job.status === "active") {
    const totalMs = Date.now() - job.createdAt.getTime();
    metrics.histogram("provisioning.total.duration_ms", totalMs, {
      isolation_level: job.isolationLevel,
      plan: job.planId,
    });
  }

  if (job.status === "rolling_back") {
    metrics.increment("provisioning.rollback.initiated", {
      failed_step: job.failedStep ?? "unknown",
    });
  }
}

The Control Plane Record

Every tenant needs a canonical record in a control plane: a master registry that other services query to know whether a tenant exists, what their plan is, and what configuration they should receive.

This is distinct from the tenant’s own data schema. The control plane lives in a global schema or a dedicated database that every service has read access to. It is the source of truth for tenant existence and should be the last thing written during provisioning and the first thing removed during deprovisioning.

interface ControlPlaneTenantRecord {
  id: string;
  name: string;
  slug: string; // used for subdomain, e.g., acme.yourapp.com
  status: "provisioning" | "active" | "suspended" | "deprovisioned";
  plan: string;
  isolationLevel: IsolationLevel;
  provisioningJobId: string;
  createdAt: Date;
  activatedAt?: Date;
}

By writing this record at the end of provisioning (with status active), you get a clean activation gate: nothing in your system will serve a tenant until provisioning has fully completed. Reads of this record are cheap if you cache aggressively (the record changes rarely once activated).

What to Build First

If you are starting from scratch with a small team and a handful of tenants, build the state machine and the control plane record first. Get the durable execution model right before you optimize the individual steps. A provisioning system that can resume after a crash is more valuable than one that is fast but leaves orphaned resources when something goes wrong.

Add rollback handlers step by step, starting with the most expensive resources to leak (databases, then schemas, then secrets). Add metrics from day one: provisioning duration and rollback rate cost almost nothing to instrument and save significant debugging time later.

The namespace isolation model can start simple and be upgraded per plan tier. You do not need Kubernetes namespaces on day one if your free tier is running in a shared schema. What you do need is the provisioning workflow to be aware of isolation levels from the start, so that switching a tenant from shared to dedicated is a provisioning operation, not a bespoke migration script written at 2am when your first enterprise customer signs.

More in System Design

How Amazon Aurora Works Internally: The Log-Is-the-Database Architecture, Quorum Writes, and Storage-Compute Separation Behind Cloud-Native SQL
System Design ·

How Amazon Aurora Works Internally: The Log-Is-the-Database Architecture, Quorum Writes, and Storage-Compute Separation Behind Cloud-Native SQL

A deep dive into Aurora's storage-compute separation, the log-is-the-database design that pushes redo log processing to storage nodes, quorum-based replication across 6 copies in 3 AZs, protection group architecture, fast cloning, Serverless v2 scaling mechanics, and honest tradeoffs versus RDS, self-managed PostgreSQL, CockroachDB, and Cloud Spanner.

How CockroachDB Works Internally: Distributed SQL, Raft Consensus Per Range, and the Architecture Behind Serializable Transactions at Global Scale
System Design ·

How CockroachDB Works Internally: Distributed SQL, Raft Consensus Per Range, and the Architecture Behind Serializable Transactions at Global Scale

A deep dive into CockroachDB's internals: the 512MB range-based data model with automatic splitting, per-range Raft replication, MVCC timestamp ordering with hybrid logical clocks, DistSQL physical planning, serializable transactions with timestamp refreshes, closed timestamps for follower reads, online schema changes using multi-version schema descriptors, and production considerations for hotspots and write amplification.

How Container Runtimes Work Internally: Namespaces, Cgroups, and the OCI Stack From Docker Run to Process Isolation
System Design ·

How Container Runtimes Work Internally: Namespaces, Cgroups, and the OCI Stack From Docker Run to Process Isolation

Containers are not virtual machines and they are not magic. They are a thin composition of Linux kernel primitives: namespaces, cgroups, and a layered filesystem. This article traces exactly what happens between docker run and a running process, covering the OCI spec, containerd, runc, overlay filesystems, and container networking at the kernel level.

How YugabyteDB Works Internally: DocDB Storage, Tablet Splitting, and the Dual API Architecture That Scales PostgreSQL Horizontally
System Design ·

How YugabyteDB Works Internally: DocDB Storage, Tablet Splitting, and the Dual API Architecture That Scales PostgreSQL Horizontally

A deep dive into YugabyteDB internals covering the DocDB storage engine built on RocksDB LSM trees, per-tablet Raft consensus, YSQL and YCQL dual query layers, distributed MVCC with hybrid logical clocks, automatic tablet splitting and rebalancing, xCluster multi-region replication, and production considerations for hotspots, connection pooling, and schema design.