Secrets Management in Production: HashiCorp Vault, AWS Secrets Manager, and Secure Injection Patterns for Kubernetes and Serverless

Environment variables became the default secrets transport because they are simple and universal. Every runtime exposes them. Every framework reads them. And they work right up until they do not.

The failure modes are not theoretical. A .env file checked into a feature branch. A process dump that includes the environment. An ECS task definition stored in plaintext in an S3 bucket because someone exported the Terraform state wrong. A secret that has not rotated in 18 months because nobody knows what will break if it does.

At production scale, the problem is not just that secrets can leak. It is that you cannot tell whether they already have. This article covers the full lifecycle: centralized storage with Vault and AWS Secrets Manager, injection patterns for Kubernetes and serverless, and what to do when something goes wrong.

Why Environment Variables Break at Scale

Environment variables have three structural problems that compound as a system grows.

No access control at the value level. The moment a secret is in the environment, every subprocess, every library, and every crash reporter with access to the process inherits it. There is no way to grant read access to one key without granting access to all of them. In a monolith this is tolerable. In a microservices architecture it means every service sees every secret it has ever been given, across every version of every deployment.

No audit trail. When a database credential is compromised, the first question is who had access and when. Environment variables give you nothing. You know the credential existed. You do not know who read it, when, from which service, or whether it was forwarded to a third-party library.

No rotation mechanism. Rotating a value stored in an environment variable requires redeploying every service that uses it. In practice this means rotation happens rarely, if at all, which means compromised credentials often stay active long after the compromise.

A centralized secrets store solves all three: fine-grained access control, full audit logging, and rotation with lease-based expiry.

HashiCorp Vault Architecture

Vault is a secrets management server with a pluggable backend. Understanding its architecture is necessary to operate it correctly, not just use it.

Seal and Unseal

Vault encrypts its storage backend with a master key. On startup, that key is not in memory. Vault starts in a sealed state and refuses all requests. To unseal it, an operator must provide key shares (Shamir’s Secret Sharing by default) sufficient to reconstruct the master key. Only then does Vault start serving traffic.

In production, manual unseal is a liability. Vault supports auto-unseal via AWS KMS, Azure Key Vault, GCP KMS, and HashiCorp’s managed service. Auto-unseal means Vault restarts after a crash without human intervention, which is necessary for any HA deployment.

// Health check that distinguishes sealed from unhealthy
async function checkVaultHealth(vaultAddr: string): Promise<{
  initialized: boolean;
  sealed: boolean;
  standby: boolean;
}> {
  const response = await fetch(`${vaultAddr}/v1/sys/health`, {
    // 200 = active, 429 = standby, 472 = recovery, 501 = not init, 503 = sealed
    redirect: "follow",
  });

  if (response.status === 200 || response.status === 429) {
    return await response.json();
  }

  throw new Error(`Vault unhealthy: HTTP ${response.status}`);
}

Auth Methods

Vault does not manage passwords. It manages tokens. To get a token, a client authenticates using an auth method that maps to the client’s identity. The token carries policies that define what the client can do.

For Kubernetes workloads, the Kubernetes auth method is the right choice. The pod presents its service account JWT, Vault validates it against the Kubernetes API, and Vault issues a token scoped to the pod’s policies. No pre-shared credentials required.

// Kubernetes auth: exchange service account JWT for a Vault token
async function vaultKubernetesAuth(
  vaultAddr: string,
  role: string
): Promise<string> {
  const jwt = await fs.readFile(
    "/var/run/secrets/kubernetes.io/serviceaccount/token",
    "utf8"
  );

  const response = await fetch(`${vaultAddr}/v1/auth/kubernetes/login`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ role, jwt }),
  });

  if (!response.ok) {
    throw new Error(`Vault auth failed: ${response.status}`);
  }

  const data = await response.json();
  return data.auth.client_token;
}

For Lambda, the AWS auth method validates the IAM identity of the function’s execution role. No extra credentials needed because Lambda already has an identity.

Dynamic Secrets and Lease Rotation

Static secrets are credentials you store in Vault and retrieve. Dynamic secrets are credentials Vault generates on demand, with a TTL. When the lease expires, Vault revokes them.

For PostgreSQL:

# Vault configuration for dynamic PostgreSQL credentials
path "database/config/prod-pg" {
  plugin_name    = "postgresql-database-plugin"
  connection_url = "postgresql://{{username}}:{{password}}@db.prod.internal:5432/app"
  allowed_roles  = ["app-readonly", "app-readwrite"]
  username       = "vault-root"
  password       = "{{ lookup_from_environment }}"
}

path "database/roles/app-readwrite" {
  db_name = "prod-pg"
  creation_statements = [
    "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';",
    "GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"
  ]
  default_ttl = "1h"
  max_ttl     = "24h"
}

Each service that reads database/creds/app-readwrite gets a unique username and password. When the lease expires, Vault drops the role. A compromised credential is automatically invalid after one hour. Rotation without deployment.

The client is responsible for renewing leases before expiry. Vault’s agent sidecar handles this automatically, which is one of the reasons the sidecar pattern is worth the overhead in Kubernetes.

AWS Secrets Manager

AWS Secrets Manager is simpler to operate than Vault if you are already AWS-native. It has no server to run, no seal/unseal ceremony, and it integrates with IAM for access control. It also costs money per secret per month ($0.40/secret/month as of 2026), which matters at scale.

Automatic Rotation

Secrets Manager can rotate secrets automatically using a Lambda function. AWS provides rotation lambdas for RDS, Redshift, and DocumentDB. For custom backends, you write your own.

import {
  SecretsManagerClient,
  GetSecretValueCommand,
  RotateSecretCommand,
} from "@aws-sdk/client-secrets-manager";

const client = new SecretsManagerClient({ region: "us-east-1" });

// Fetch and parse a secret
async function getSecret<T>(secretId: string): Promise<T> {
  const command = new GetSecretValueCommand({ SecretId: secretId });
  const response = await client.send(command);

  if (!response.SecretString) {
    throw new Error(`Secret ${secretId} has no string value`);
  }

  return JSON.parse(response.SecretString) as T;
}

// Trigger immediate rotation
async function rotateNow(secretId: string): Promise<void> {
  const command = new RotateSecretCommand({
    SecretId: secretId,
    RotateImmediately: true,
  });
  await client.send(command);
}

During rotation, Secrets Manager maintains two versions: AWSCURRENT (the live credential) and AWSPENDING (the new one being tested). Once the rotation lambda confirms the new credential works, it promotes AWSPENDING to AWSCURRENT. This prevents a rotation failure from taking down your application.

Cross-Account Access

A common production pattern is a central secrets account with resource-based policies that allow access from application accounts. This avoids replicating secrets across accounts while keeping credentials centralized.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/app-prod-execution-role"
      },
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "arn:aws:secretsmanager:us-east-1:999988887777:secret:prod/db/credentials-*"
    }
  ]
}

The application account role also needs an IAM policy allowing secretsmanager:GetSecretValue on the cross-account resource, and the secrets must be encrypted with a KMS key that allows cross-account decrypt.

Vault vs. AWS Secrets Manager: Direct Comparison

Dimension	HashiCorp Vault	AWS Secrets Manager
Operational overhead	High (cluster management, HA, seal/unseal)	None (managed service)
Dynamic secrets	Native (database, PKI, SSH, AWS)	No (static secrets only)
Multi-cloud	Yes	AWS-only
Cost	License or Vault HCP cost	$0.40/secret/month + $0.05/10K API calls
Audit logging	Built-in, structured	CloudTrail integration
Access control	Fine-grained policies + namespaces	IAM resource policies
Rotation	Lease-based automatic revocation	Lambda-backed, AWS-managed for RDS
Secret versioning	Yes	Yes (100 version limit)
Kubernetes integration	Vault Agent, CSI driver, sidecar	Secrets Store CSI with AWS provider

The short version: if you are running a single-cloud AWS architecture, Secrets Manager is easier to operate and sufficient for most teams. If you need dynamic secrets, multi-cloud, or strict certificate lifecycle management, Vault is worth the operational cost. The two are not mutually exclusive: Vault can use AWS KMS for auto-unseal and Secrets Manager as a backend.

Kubernetes Injection Patterns

Kubernetes has a native Secret resource, but mounting it as an environment variable or a volume does not encrypt it at rest by default and does not integrate with external secret stores. There are four production patterns worth knowing.

Pattern 1: Native Kubernetes Secrets (the baseline)

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: app-prod
type: Opaque
data:
  # base64-encoded values — NOT encrypted
  username: YXBwdXNlcg==
  password: c3VwZXJzZWNyZXQ=
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: app
          envFrom:
            - secretRef:
                name: db-credentials

This works but has known limitations: etcd stores secrets base64-encoded (not encrypted) unless you configure encryption at rest, secrets are replicated to all nodes where pods run, and you have no integration with rotation. Use it only when your threat model accepts these tradeoffs and you have etcd encryption enabled.

Pattern 2: CSI Secrets Store Driver

The Secrets Store CSI Driver mounts secrets from external providers (Vault, AWS, Azure, GCP) as volumes. The secret never becomes a Kubernetes Secret object. It is fetched at pod startup from the external store and mounted directly into the pod’s filesystem.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
  namespace: app-prod
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.prod.internal:8200"
    roleName: "app-prod"
    objects: |
      - objectName: "db-password"
        secretPath: "database/creds/app-readwrite"
        secretKey: "password"
      - objectName: "db-username"
        secretPath: "database/creds/app-readwrite"
        secretKey: "username"
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      volumes:
        - name: secrets-vol
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "vault-db-creds"
      containers:
        - name: app
          volumeMounts:
            - name: secrets-vol
              mountPath: "/mnt/secrets"
              readOnly: true

The application reads /mnt/secrets/db-password as a file. When the Vault lease expires and the CSI driver refreshes the mount, the file updates without a pod restart.

Pattern 3: Init Container

For applications that expect environment variables and cannot be modified to read files, an init container can fetch secrets from Vault and write them to a shared volume before the main container starts.

// init-secrets.ts — runs as an init container
import { writeFileSync } from "fs";

async function fetchAndWriteSecrets(): Promise<void> {
  const token = await vaultKubernetesAuth(
    process.env.VAULT_ADDR!,
    process.env.VAULT_ROLE!
  );

  const response = await fetch(
    `${process.env.VAULT_ADDR}/v1/database/creds/app-readwrite`,
    { headers: { "X-Vault-Token": token } }
  );

  if (!response.ok) {
    throw new Error(`Vault fetch failed: ${response.status}`);
  }

  const { data } = await response.json();

  // Write to shared volume as shell-sourceable file
  const envContent = [
    `DB_USERNAME=${data.username}`,
    `DB_PASSWORD=${data.password}`,
  ].join("\n");

  writeFileSync("/shared/secrets.env", envContent, { mode: 0o600 });
  console.log("Secrets written to /shared/secrets.env");
}

fetchAndWriteSecrets().catch((err) => {
  console.error(err);
  process.exit(1);
});

The init container pattern does not handle lease renewal. Once the lease expires, the credential becomes invalid. Use this only for short-lived workloads or pair it with the sidecar pattern.

Pattern 4: Vault Agent Sidecar

The Vault Agent sidecar runs alongside your application container, authenticates to Vault, renders secrets into template files, and renews leases automatically. This is the most complete pattern for long-running services.

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        # Vault Agent injector annotations (requires vault-agent-injector installed)
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-prod"
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/app-readwrite"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/app-readwrite" -}}
          DB_USERNAME={{ .Data.data.username }}
          DB_PASSWORD={{ .Data.data.password }}
          {{- end }}
    spec:
      serviceAccountName: app-prod-sa
      containers:
        - name: app
          command: ["/bin/sh", "-c"]
          args:
            - |
              source /vault/secrets/db-creds
              exec node server.js

The injector mutates the pod spec at admission time, adding the Vault Agent container and the shared volume. The application sources the rendered file at startup. When the lease is about to expire, the agent rewrites the file. If your application watches for file changes and reconnects on credential refresh, you get seamless rotation with no downtime.

Serverless Injection Patterns

Serverless functions have no persistent filesystem and no long-running sidecar. The injection strategies differ.

Lambda: AWS Secrets Manager with In-Process Caching

Fetching a secret on every invocation is slow (20-50ms round-trip) and expensive. The right pattern is to fetch once per Lambda instance lifecycle and cache with a TTL.

import {
  SecretsManagerClient,
  GetSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";

const client = new SecretsManagerClient({ region: process.env.AWS_REGION! });

interface CachedSecret {
  value: unknown;
  fetchedAt: number;
  ttlMs: number;
}

const cache = new Map<string, CachedSecret>();

async function getSecretCached<T>(
  secretId: string,
  ttlMs = 300_000 // 5 minutes
): Promise<T> {
  const cached = cache.get(secretId);
  const now = Date.now();

  if (cached && now - cached.fetchedAt < cached.ttlMs) {
    return cached.value as T;
  }

  const command = new GetSecretValueCommand({ SecretId: secretId });
  const response = await client.send(command);

  if (!response.SecretString) {
    throw new Error(`No string value for secret: ${secretId}`);
  }

  const value = JSON.parse(response.SecretString) as T;
  cache.set(secretId, { value, fetchedAt: now, ttlMs });
  return value;
}

// Lambda handler
export const handler = async (event: unknown) => {
  const dbCreds = await getSecretCached<{ username: string; password: string }>(
    "prod/db/credentials"
  );

  // Use dbCreds.username, dbCreds.password
};

The cache lives in the Lambda instance’s module scope. It persists across warm invocations. When the TTL expires, the next invocation fetches fresh credentials. Size the TTL to be shorter than your rotation period.

For Lambda calling Vault, use the AWS IAM auth method. The function’s execution role authenticates with Vault using sts:GetCallerIdentity signed by the IAM credentials Lambda provides automatically.

Cloudflare Workers: Secrets via Wrangler

Cloudflare Workers does not have a filesystem or environment-variable-style secret store at the infrastructure level. Secrets are bound to a Worker at deployment time and accessed via the environment parameter.

// wrangler.toml
// [vars] for non-sensitive config
// secrets added via: wrangler secret put DB_PASSWORD

interface Env {
  DB_PASSWORD: string;
  DB_USERNAME: string;
  API_KEY: string;
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    // Secrets are available on env, never in process.env
    const db = await connectToDatabase({
      username: env.DB_USERNAME,
      password: env.DB_PASSWORD,
    });

    return new Response("ok");
  },
};

Workers secrets are encrypted at rest and in transit and are not visible in the dashboard after creation. The limitation is that rotation requires a wrangler secret put call and a new deployment. For frequently rotated secrets, integrate this into your CI/CD pipeline with a secrets manager as the source of truth:

// rotate-worker-secret.ts — runs in CI after Vault rotates a credential
import { execSync } from "child_process";

async function syncVaultSecretToWorker(
  vaultPath: string,
  secretKey: string,
  workerName: string,
  workerSecretName: string
): Promise<void> {
  const token = process.env.VAULT_TOKEN!;
  const response = await fetch(
    `${process.env.VAULT_ADDR}/v1/${vaultPath}`,
    { headers: { "X-Vault-Token": token } }
  );

  const { data } = await response.json();
  const secretValue = data[secretKey];

  // Pipe the value into wrangler secret put
  execSync(
    `echo "${secretValue}" | wrangler secret put ${workerSecretName} --name ${workerName}`,
    { stdio: "inherit" }
  );
}

Rotation Strategies

Rotation is not a discrete event. It is a protocol that must handle the window where both old and new credentials are valid, different service instances may hold different generations of a secret, and failure at any step must not cause an outage.

Pre-rotation: Before changing the credential in the backend, notify dependent services if possible. For database credentials, this means creating the new role before removing the old one.

Dual validity window: Keep both the old and new credentials valid for a defined window (typically 15-30 minutes). During this window, services can reconnect using the new credential without a forced restart. Vault’s dynamic secrets handle this automatically through lease overlap. Secrets Manager’s AWSPENDING/AWSCURRENT versioning handles it for managed rotation.

Application-side reconnection: Applications must handle database connection errors by refetching the credential and reconnecting rather than crashing. A simple retry wrapper:

async function queryWithRetry<T>(
  queryFn: (client: DatabaseClient) => Promise<T>,
  maxRetries = 1
): Promise<T> {
  try {
    return await queryFn(getConnection());
  } catch (err: unknown) {
    const isAuthError =
      err instanceof Error &&
      (err.message.includes("authentication failed") ||
        err.message.includes("password authentication failed"));

    if (isAuthError && maxRetries > 0) {
      await refreshDatabaseCredentials();
      return queryWithRetry(queryFn, maxRetries - 1);
    }

    throw err;
  }
}

Emergency Revocation

When a secret is confirmed compromised, revocation must happen faster than any rotation schedule.

In Vault, revoke by lease or by accessor:

# Revoke a specific lease immediately
vault lease revoke database/creds/app-readwrite/xxxxxx

# Revoke all leases under a path (nuclear option)
vault lease revoke -prefix database/creds/app-readwrite

# Revoke a token by accessor without knowing the token value
vault token revoke -accessor <accessor-id>

In AWS Secrets Manager, there is no lease-based revocation. You must rotate the secret immediately, which triggers the rotation Lambda to replace the underlying credential:

async function emergencyRotate(secretId: string): Promise<void> {
  const command = new RotateSecretCommand({
    SecretId: secretId,
    RotateImmediately: true,
  });
  await client.send(command);
  console.log(`Emergency rotation triggered for ${secretId}`);
}

For database credentials not managed by either tool, have a documented runbook that specifies exactly which queries to run and which services to bounce. The runbook should be executable by anyone on the team, not just the person who originally set up the database.

After revocation, the audit log tells you what accessed the compromised credential, from which service, and at what time. This is what justifies the operational overhead of a centralized secrets store. Without it, you are guessing at blast radius.

---

The pattern that works at scale is not the most elegant one. It is the one that handles rotation without downtime, logs every access, and gives you a path to revocation in under five minutes. Whether that is Vault or Secrets Manager depends on your cloud strategy and your willingness to run infrastructure. The injection pattern depends on your runtime. The rotation and revocation protocol depends on your threat model. All three decisions compound.