Designing a Multi-Tenant Data Export Pipeline: Bulk Exports, Streaming Downloads, and Rate-Limited Delivery for SaaS Products

Data export is one of those features that looks trivial on the surface and turns out to be one of the more interesting engineering problems in a SaaS product. The request is simple: “let users download their data.” The implementation is not.

The complexity comes from the intersection of several hard constraints: tenant isolation means you must never leak one tenant’s data into another’s export; large datasets mean you cannot hold everything in memory; format negotiation means your pipeline needs to support CSV, JSON, and sometimes Parquet; and rate limiting means a single large tenant should not starve others. Add GDPR’s data portability requirements and you have a compliance surface to manage on top of all of it.

This article walks through the full architecture: request queuing, chunked processing, streaming responses, presigned URL delivery, tenant-level fair scheduling, background job design with progress tracking, and the compliance layer.

Why Synchronous Export Breaks at Scale

The naive implementation is a synchronous HTTP endpoint that runs a database query, serializes the result, and streams it back. This works fine for small datasets. It fails in several predictable ways as data grows.

First, the database query time exceeds typical HTTP timeout thresholds. Most load balancers and API gateways cut connections after 30-60 seconds. A tenant with 500,000 records will hit this ceiling regularly.

Second, serializing large payloads into memory before streaming them causes RSS to spike per request. If three large tenants trigger exports simultaneously, your API servers OOM.

Third, there is no retry boundary. If the connection drops at 80% complete, the user gets nothing and has no way to resume.

The correct model is asynchronous: accept the export request immediately, return a job ID, process in the background, and notify the user when the file is ready. This separates the user-facing latency from the actual data processing time.

Request Queue Architecture

The export request lifecycle has four stages: submission, scheduling, processing, and delivery.

// Export request submission
interface ExportRequest {
  id: string;
  tenantId: string;
  requestedBy: string;
  format: 'csv' | 'json' | 'parquet';
  filters: ExportFilters;
  columns: string[];
  createdAt: Date;
  priority: 'normal' | 'high';
}

interface ExportFilters {
  dateRange?: { from: Date; to: Date };
  entityType: string;
  conditions: Record<string, unknown>;
}

async function submitExportRequest(
  tenantId: string,
  userId: string,
  options: Pick<ExportRequest, 'format' | 'filters' | 'columns'>
): Promise<{ jobId: string }> {
  // Enforce per-tenant rate limit before enqueuing
  await checkExportRateLimit(tenantId);

  const job: ExportRequest = {
    id: crypto.randomUUID(),
    tenantId,
    requestedBy: userId,
    ...options,
    createdAt: new Date(),
    priority: 'normal',
  };

  await db.exportJobs.create({
    data: { ...job, status: 'pending', progress: 0 },
  });

  await exportQueue.add('process-export', { jobId: job.id }, {
    jobId: job.id,
    attempts: 3,
    backoff: { type: 'exponential', delay: 5000 },
  });

  await auditLog.record({
    tenantId,
    userId,
    action: 'export.requested',
    metadata: { jobId: job.id, format: job.format, entityType: job.filters.entityType },
  });

  return { jobId: job.id };
}

Persisting the job to the database before enqueuing is important. The queue message is ephemeral. The database record is the source of truth for job status, which the user polls or which drives webhook notifications.

Tenant Isolation in the Processing Layer

The biggest correctness risk in a multi-tenant export pipeline is data leakage. Every database query in the processing layer must be scoped to the requesting tenant. This sounds obvious, but in practice it requires discipline at the query layer, not just the controller layer.

async function buildExportQuery(
  job: ExportRequest,
  offset: number,
  limit: number
): Promise<QueryResult[]> {
  // tenantId is always injected from the job record,
  // never from user-supplied input in the current request context
  return db.query(
    `SELECT ${sanitizeColumns(job.columns)}
     FROM ${job.filters.entityType}
     WHERE tenant_id = $1
       AND ${buildFilterClause(job.filters.conditions)}
     ORDER BY id
     LIMIT $2 OFFSET $3`,
    [job.tenantId, limit, offset]
  );
}

function sanitizeColumns(requested: string[]): string {
  const allowed = EXPORT_COLUMN_ALLOWLIST[job.filters.entityType] ?? [];
  const safe = requested.filter(col => allowed.includes(col));
  if (safe.length === 0) return allowed.join(', ');
  return safe.join(', ');
}

Column selection has its own isolation concern: users should only be able to export columns that are explicitly allowed for their entity type. Passing raw column names into SQL is a SQL injection vector. Use an allowlist, not a denylist.

Chunked Processing to Avoid Memory Pressure

Chunking is the core technique for keeping worker memory bounded regardless of dataset size. The worker fetches records in pages, serializes each page incrementally, and writes to a temp file or streams directly to object storage.

const CHUNK_SIZE = 1000; // rows per chunk

async function processExportJob(jobId: string): Promise<void> {
  const job = await db.exportJobs.findUniqueOrThrow({ where: { id: jobId } });

  await db.exportJobs.update({
    where: { id: jobId },
    data: { status: 'processing', startedAt: new Date() },
  });

  const totalCount = await getExportCount(job);
  const tempPath = `/tmp/export-${jobId}.${job.format}`;
  const writeStream = createWriteStream(tempPath);
  const serializer = createSerializer(job.format, writeStream);

  await serializer.writeHeader(job.columns);

  let offset = 0;
  let processed = 0;

  while (offset < totalCount) {
    const rows = await buildExportQuery(job, offset, CHUNK_SIZE);

    if (rows.length === 0) break;

    await serializer.writeRows(rows);

    processed += rows.length;
    offset += CHUNK_SIZE;

    // Progress update throttled to avoid DB write storms
    if (processed % (CHUNK_SIZE * 5) === 0) {
      const progress = Math.round((processed / totalCount) * 100);
      await db.exportJobs.update({
        where: { id: jobId },
        data: { progress, processedRows: processed, totalRows: totalCount },
      });
    }
  }

  await serializer.finalize();
  writeStream.end();

  const fileUrl = await uploadToObjectStorage(jobId, tempPath, job.format);
  await fs.unlink(tempPath); // clean up temp file

  await db.exportJobs.update({
    where: { id: jobId },
    data: { status: 'completed', progress: 100, fileUrl, completedAt: new Date() },
  });

  await notifyUser(job);
}

The serializer abstraction hides format-specific logic. CSV uses a streaming writer that appends rows incrementally. JSON line-delimits records (NDJSON) so the file is appendable without holding the full structure in memory. Parquet is more involved because it requires a columnar layout; for Parquet, use a library like parquetjs-lite that supports row-group-based writing.

Uploading to object storage (S3, GCS, R2) rather than serving from the worker process is the correct delivery model. Workers are ephemeral. Object storage is durable, scalable, and already has the primitives for secure delivery.

Streaming vs. Presigned URL Delivery

For small exports (under ~50MB), streaming the response directly is acceptable: the worker generates the file and the API streams it back with appropriate headers. For large exports, presigned URLs are the right model.

Delivery method	Max practical size	Resumable	Requires background job	Server bandwidth cost
Synchronous streaming	~50MB	No	No	High (API servers)
Background job + presigned URL	Unlimited	Yes (range requests)	Yes	Low (object storage)
Background job + streaming proxy	~500MB	No	Yes	Medium
Chunked multipart download	Unlimited	Yes	Yes	Low

Presigned URLs let you sign a time-limited, scoped object storage URL that the client downloads directly from S3/GCS. This offloads bandwidth entirely and supports HTTP range requests, which means failed downloads can be resumed without re-processing the export.

import { S3Client, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

const s3 = new S3Client({ region: process.env.AWS_REGION });

async function generatePresignedDownloadUrl(
  jobId: string,
  tenantId: string
): Promise<string> {
  const job = await db.exportJobs.findUniqueOrThrow({
    where: { id: jobId },
  });

  // Verify the requesting tenant owns this job
  if (job.tenantId !== tenantId) {
    throw new Error('Forbidden');
  }

  if (job.status !== 'completed') {
    throw new Error('Export not ready');
  }

  const command = new GetObjectCommand({
    Bucket: process.env.EXPORT_BUCKET,
    Key: `exports/${tenantId}/${jobId}.${job.format}`,
    ResponseContentDisposition: `attachment; filename="export-${jobId}.${job.format}"`,
  });

  // URLs expire in 1 hour; use shorter TTLs for sensitive data
  return getSignedUrl(s3, command, { expiresIn: 3600 });
}

The tenant ID validation in the presigned URL generator is critical. The job ID alone should not be sufficient to download any export. The request must prove it belongs to the right tenant.

Tenant-Level Rate Limiting and Fair Scheduling

Without rate limiting, a single tenant can submit 50 export requests simultaneously and saturate the worker pool, starving everyone else. The right model is token bucket rate limiting at the tenant level combined with a fair queue scheduler.

import { RateLimiterRedis } from 'rate-limiter-flexible';
import Redis from 'ioredis';

const redis = new Redis(process.env.REDIS_URL);

// Per-tenant: 5 export requests per hour, burst of 2
const exportRateLimiter = new RateLimiterRedis({
  storeClient: redis,
  keyPrefix: 'export_rl',
  points: 5,
  duration: 3600,
  blockDuration: 0,
});

async function checkExportRateLimit(tenantId: string): Promise<void> {
  try {
    await exportRateLimiter.consume(tenantId);
  } catch {
    throw new Error('Export rate limit exceeded. Try again later.');
  }
}

// Fair scheduling: cap concurrent active jobs per tenant
async function canStartJob(tenantId: string): Promise<boolean> {
  const activeCount = await db.exportJobs.count({
    where: { tenantId, status: { in: ['processing'] } },
  });
  return activeCount < MAX_CONCURRENT_EXPORTS_PER_TENANT; // typically 2-3
}

For the queue itself, use per-tenant queues with weighted scheduling if you have high-volume tenants on paid plans versus free-tier tenants. A simpler approach that works for most SaaS products: a single queue with a priority field and a worker that checks canStartJob before picking up a job, re-queuing it with a short delay if the tenant is already at capacity.

Background Job Design and Progress Tracking

The job record in the database is the single source of truth for export state. The client polls a status endpoint, and when the job completes, the response includes the presigned URL.

interface ExportJobStatus {
  id: string;
  status: 'pending' | 'processing' | 'completed' | 'failed';
  progress: number; // 0-100
  processedRows?: number;
  totalRows?: number;
  fileUrl?: string; // presigned URL, only present when completed
  error?: string;
  createdAt: Date;
  startedAt?: Date;
  completedAt?: Date;
  expiresAt?: Date; // when the file will be deleted from object storage
}

// Status polling endpoint
async function getExportStatus(
  jobId: string,
  tenantId: string
): Promise<ExportJobStatus> {
  const job = await db.exportJobs.findUnique({ where: { id: jobId } });

  if (!job || job.tenantId !== tenantId) {
    throw new Error('Not found');
  }

  const result: ExportJobStatus = {
    id: job.id,
    status: job.status,
    progress: job.progress,
    processedRows: job.processedRows ?? undefined,
    totalRows: job.totalRows ?? undefined,
    createdAt: job.createdAt,
    startedAt: job.startedAt ?? undefined,
    completedAt: job.completedAt ?? undefined,
    expiresAt: job.expiresAt ?? undefined,
  };

  if (job.status === 'completed' && job.fileUrl) {
    // Regenerate a fresh presigned URL on each poll
    // to avoid serving an expired URL if the user polled late
    result.fileUrl = await generatePresignedDownloadUrl(jobId, tenantId);
  }

  if (job.status === 'failed') {
    result.error = job.errorMessage ?? 'Export failed. Please try again.';
  }

  return result;
}

Regenerating the presigned URL on each status poll is a small detail with real user impact. If a user submits an export, walks away, and comes back two hours later to check the status, a URL generated at job completion time may have already expired.

For error handling: distinguish between retriable errors (database timeouts, transient network issues) and permanent failures (invalid filter configuration, schema mismatch). Use the queue retry mechanism for retriable errors. Mark the job as permanently failed for configuration errors, and surface a specific message to the user.

Data Filtering and Column Selection APIs

Exposing a filtering and column selection API for exports requires the same input validation discipline as any other query parameter surface.

const EXPORT_SCHEMA: Record<string, ExportEntityConfig> = {
  orders: {
    columns: ['id', 'status', 'total', 'currency', 'created_at', 'customer_id'],
    sensitiveColumns: ['customer_email', 'customer_phone'],
    filters: {
      dateRange: true,
      status: ['pending', 'completed', 'refunded', 'cancelled'],
    },
    estimatedRowsPerDay: 1000,
  },
  customers: {
    columns: ['id', 'created_at', 'plan', 'country'],
    sensitiveColumns: ['email', 'name', 'phone'],
    filters: {
      dateRange: true,
      plan: ['free', 'pro', 'enterprise'],
    },
    estimatedRowsPerDay: 50,
  },
};

function validateExportRequest(
  entityType: string,
  requestedColumns: string[],
  filters: unknown,
  userPermissions: Set<string>
): { columns: string[]; filters: ExportFilters } {
  const config = EXPORT_SCHEMA[entityType];
  if (!config) throw new Error(`Unknown entity type: ${entityType}`);

  // Filter columns to what the user is allowed to see
  const allowedColumns = userPermissions.has('export:sensitive')
    ? [...config.columns, ...config.sensitiveColumns]
    : config.columns;

  const columns = requestedColumns.length > 0
    ? requestedColumns.filter(c => allowedColumns.includes(c))
    : allowedColumns;

  if (columns.length === 0) {
    throw new Error('No valid columns selected');
  }

  return { columns, filters: parseFilters(filters, config) };
}

Separating columns from sensitiveColumns and gating sensitive fields behind a permission scope (rather than hiding them entirely) gives you the flexibility to let admin users export PII while restricting standard users from doing so. This matters for GDPR compliance: you need to be able to produce a full data portability export when a data subject requests it, but you also need to prevent unauthorized bulk extraction of personal data.

Compliance: GDPR Data Portability and Audit Logging

GDPR Article 20 grants data subjects the right to receive their personal data in a structured, commonly used, machine-readable format. For SaaS products with EU users, your export pipeline is part of your compliance surface.

Three concrete requirements follow from this:

1. Subject Access Request (SAR) exports. When a user requests their own data, you must produce a complete export of all personal data you hold about them. This is a different query shape from a tenant bulk export: it is scoped to a single data subject (user ID), not a tenant, and must traverse all tables that contain personal data for that subject.

2. Audit logging of every export. You must be able to answer: who requested an export, when, what data was included, and whether the export was delivered. The audit record should be append-only and protected from modification.

async function recordExportAuditEvent(
  event: 'requested' | 'started' | 'completed' | 'downloaded' | 'failed',
  job: ExportRequest,
  metadata?: Record<string, unknown>
): Promise<void> {
  await db.auditLog.create({
    data: {
      id: crypto.randomUUID(),
      tenantId: job.tenantId,
      actorId: job.requestedBy,
      action: `export.${event}`,
      resourceType: 'export_job',
      resourceId: job.id,
      metadata: {
        format: job.format,
        entityType: job.filters.entityType,
        columnCount: job.columns.length,
        includesSensitiveData: job.columns.some(c =>
          EXPORT_SCHEMA[job.filters.entityType]?.sensitiveColumns.includes(c)
        ),
        ...metadata,
      },
      createdAt: new Date(),
    },
  });
}

3. File retention limits. Generated export files should not persist indefinitely in object storage. Set a TTL of 24-72 hours, enforce it with an object storage lifecycle policy, and reflect the expiresAt timestamp in the job record so users know when the download link will stop working.

Tradeoffs Summary

Concern	Simple approach	Robust approach	When to upgrade
Export delivery	Synchronous HTTP streaming	Background job + presigned URL	Dataset exceeds 10MB or processing time exceeds 10s
Tenant isolation	Controller-level tenant ID check	Tenant ID injected at query layer, column allowlist	Before first multi-tenant customer
Memory management	Load all rows, then serialize	Chunk + stream to object storage	Dataset exceeds available worker RAM
Rate limiting	Global concurrent job limit	Per-tenant token bucket + fair scheduler	First tenant with high export volume
Progress tracking	Polling on job status	Polling with progress percentage and row counts	Users complain about uncertainty on large exports
Compliance logging	Application logs	Append-only audit table with sensitive field flags	Before handling EU personal data
File retention	No TTL	Lifecycle policy + expiresAt in job record	Before object storage costs become noticeable

Production Considerations

Estimate export size before queuing. A quick COUNT and rough bytes-per-row estimate lets you warn users about large exports, enforce per-tenant storage quotas, and set appropriate job timeouts. A 10-million-row export will take different resources than a 10,000-row export.

Handle schema evolution. If you add a column to a table after a job is queued but before it is processed, the column selection may be stale. Validate columns against the live schema at processing time, not at submission time, and drop unknown columns gracefully.

Worker idempotency. If a worker crashes mid-export and the job is retried, the temp file may be partially written. Clear the temp file at the start of processing, not just at the end. An alternative is to upload chunks to object storage as they are processed using multipart upload, which gives you a natural restart point at the chunk boundary.

Compression. Compress exports at rest and in transit. Gzip adds negligible CPU cost and typically reduces file size by 60-80% for structured tabular data. S3 presigned URLs can serve gzip-compressed files with the correct Content-Encoding header so the browser decompresses transparently.

Notify via webhook, not just polling. Polling works but creates unnecessary load when the queue is long. Send a webhook to the tenant’s configured endpoint when the job completes. The payload should include the job ID, status, and the presigned URL. Let tenants poll as a fallback, not the primary notification path.

The data export feature is a forcing function for getting several foundational pieces right: tenant isolation at the query layer, background job infrastructure with durable state, fair multi-tenant scheduling, object storage for file delivery, and an audit trail that satisfies compliance requirements. Build each layer independently and the complexity becomes manageable.