Immutable Infrastructure in Practice: Golden Images, Container Registries, and Deployment Patterns That Eliminate Configuration Drift

Every team eventually produces a snowflake server. It starts innocuously: a hotfix applied directly via SSH during an incident at 2 AM, a dependency installed by hand to unblock a deadline, a config file edited “just to test something” that never got reverted. Six months later, nobody knows why that server runs differently from the others. Reproducing the environment from scratch is a multi-day archaeology project. Deployments to it are nerve-wracking because behavior is unpredictable.

This is configuration drift. It is not a team discipline problem. It is an architecture problem. Mutable infrastructure, where you update running servers in place, makes drift structurally inevitable. Immutable infrastructure eliminates the mechanism by which drift occurs: you never modify a running instance. Instead, you build a new image, deploy it, and discard the old one.

That sounds simple. The operational reality involves image pipelines, registry promotion workflows, deployment patterns that handle traffic cutover, runtime secret injection, and knowing when to not use immutability at all. This article works through each of those concerns concretely.

Why Mutable Infrastructure Drifts

The root cause is that SSH (or any interactive access to production) creates an out-of-band change channel. Infrastructure-as-code tools like Terraform and Ansible model what a server should look like, but they do not prevent someone from logging in and changing it. Reconciliation runs on a schedule or a trigger. The gap between runs is where drift lives.

Configuration management tools (Chef, Puppet, Ansible) were supposed to solve this with idempotent runs. In practice, they reduced drift but did not eliminate it. Idempotency assumes your playbook covers every possible state. It does not cover the state where someone installed a package that your playbook does not know to remove.

Immutable infrastructure sidesteps the problem entirely. If you cannot modify a running server, it cannot drift. The only way to change the environment is to ship a new image. That forces every change through your build pipeline where it can be reviewed, tested, and recorded.

Golden Image Pipelines

A golden image is a base machine image or container image that encodes everything needed to run a service: OS packages, runtime, application code, and configuration (except secrets, more on that shortly). The pipeline that produces it is deterministic: given the same inputs, it produces the same output.

For VMs, Packer is the standard tool. Here is a minimal Packer HCL template that builds an Amazon Machine Image (AMI) for a Node.js service:

packer {
  required_plugins {
    amazon = {
      version = ">= 1.2.0"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

variable "node_version" {
  type    = string
  default = "20.11.1"
}

variable "app_version" {
  type    = string
  default = "1.0.0"
}

source "amazon-ebs" "app" {
  ami_name      = "my-service-${var.app_version}-{{timestamp}}"
  instance_type = "t3.medium"
  region        = "us-east-1"

  source_ami_filter {
    filters = {
      name                = "al2023-ami-2023.*-x86_64"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = ["amazon"]
  }

  ssh_username = "ec2-user"

  tags = {
    Name        = "my-service"
    Version     = var.app_version
    NodeVersion = var.node_version
    BuildDate   = "{{timestamp}}"
    ManagedBy   = "packer"
  }
}

build {
  sources = ["source.amazon-ebs.app"]

  provisioner "shell" {
    inline = [
      "sudo dnf update -y",
      "sudo dnf install -y nodejs-${var.node_version}",
      "sudo useradd -r -s /bin/false appuser",
    ]
  }

  provisioner "file" {
    source      = "dist/"
    destination = "/opt/app/"
  }

  provisioner "shell" {
    inline = [
      "cd /opt/app && sudo npm ci --omit=dev",
      "sudo chown -R appuser:appuser /opt/app",
    ]
  }

  provisioner "file" {
    source      = "deploy/app.service"
    destination = "/etc/systemd/system/app.service"
  }

  provisioner "shell" {
    inline = [
      "sudo systemctl daemon-reload",
      "sudo systemctl enable app.service",
    ]
  }

  post-processor "manifest" {
    output = "packer-manifest.json"
  }
}

The output is a versioned AMI tagged with the application version and build timestamp. Terraform then references this AMI by version tag when provisioning instances, not by a mutable “latest” pointer.

For containers, Docker multi-stage builds are the golden image equivalent. The staging layer holds build tooling that never makes it to production:

# Stage 1: build
FROM node:20.11.1-alpine AS builder

WORKDIR /build

COPY package.json package-lock.json ./
RUN npm ci

COPY tsconfig.json ./
COPY src/ ./src/
RUN npm run build

# Prune dev dependencies
RUN npm ci --omit=dev

# Stage 2: production image (the golden image)
FROM node:20.11.1-alpine AS production

# Set a non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup

WORKDIR /app

# Copy only what production needs
COPY --from=builder --chown=appuser:appgroup /build/dist ./dist
COPY --from=builder --chown=appuser:appgroup /build/node_modules ./node_modules
COPY --from=builder --chown=appuser:appgroup /build/package.json ./

USER appuser

ENV NODE_ENV=production
ENV PORT=3000

EXPOSE 3000

ENTRYPOINT ["node", "dist/index.js"]

The resulting image has no compiler, no TypeScript toolchain, no dev dependencies. It is smaller, has a smaller attack surface, and its contents are exactly what you put in.

Container Registry Management and Image Promotion

An image built from a commit is not automatically production-ready. It needs to pass tests, security scans, and integration validation in staging before it goes to production. This is image promotion: the same image digest moves through environments, not a rebuilt image per environment.

The workflow looks like this:

1. CI builds the image on every pull request merge to main
2. The image is tagged with the commit SHA and pushed to a registry namespace for staging
3. Automated tests and scans run against the staging-tagged image
4. On success, the image is re-tagged (or logically promoted) to a production-eligible tag
5. Deployment to production references the promoted digest, not a rebuilt image

The key constraint is that the image digest must be identical across environments. If you rebuild for production, you have broken the immutability guarantee: the artifact running in production is not the artifact that was tested.

Here is a GitHub Actions workflow that implements this pattern:

name: Build and Promote

on:
  push:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    outputs:
      digest: ${{ steps.build.outputs.digest }}
      sha_tag: ${{ steps.meta.outputs.version }}

    steps:
      - uses: actions/checkout@v4

      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Generate metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha,prefix=sha-,format=short

      - name: Build and push (staging tag)
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  scan:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Scan image for vulnerabilities
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.digest }}"
          severity: "CRITICAL,HIGH"
          exit-code: "1"

  promote-to-staging:
    needs: [build, scan]
    runs-on: ubuntu-latest
    permissions:
      packages: write
    steps:
      - name: Retag image as staging-latest
        run: |
          docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.digest }}
          docker tag ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.digest }} \
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:staging
          docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:staging

      # Trigger staging deployment (update GitOps repo image tag)
      - name: Update staging kustomization
        run: |
          git clone https://x-access-token:${{ secrets.GITOPS_TOKEN }}@github.com/org/gitops-config.git
          cd gitops-config
          kustomize edit set image \
            ghcr.io/org/my-service@${{ needs.build.outputs.digest }}
          git commit -am "chore: promote ${{ needs.build.outputs.sha_tag }} to staging"
          git push
        working-directory: gitops-config/overlays/staging

The promotion to production is a separate job, gated on staging smoke tests passing. The digest referenced is identical to what ran in staging. The container registry is the source of truth, not a build script.

Deployment Patterns That Leverage Immutability

Immutable images make blue-green and canary deployments operationally straightforward. You have two known-good artifacts and a traffic routing layer. Rollback is switching the router, not rebuilding.

Blue-green maintains two identical production environments. One is live (blue), one is idle (green). You deploy the new version to green, run validation, then shift all traffic. If something is wrong, traffic switches back to blue in seconds. The cost is running two production environments simultaneously, though for most services the overlap is minutes, not hours.

A TypeScript health check gate before traffic cutover:

interface HealthCheckConfig {
  endpoint: string;
  expectedStatus: number;
  timeoutMs: number;
  retries: number;
  retryDelayMs: number;
}

async function waitForHealthy(config: HealthCheckConfig): Promise<void> {
  for (let attempt = 0; attempt <= config.retries; attempt++) {
    try {
      const response = await fetch(config.endpoint, {
        signal: AbortSignal.timeout(config.timeoutMs),
      });

      if (response.status === config.expectedStatus) {
        console.log(`Health check passed on attempt ${attempt + 1}`);
        return;
      }

      console.warn(
        `Health check returned ${response.status}, expected ${config.expectedStatus}`
      );
    } catch (err) {
      console.warn(`Health check attempt ${attempt + 1} failed: ${err}`);
    }

    if (attempt < config.retries) {
      await new Promise((resolve) => setTimeout(resolve, config.retryDelayMs));
    }
  }

  throw new Error(
    `Health check failed after ${config.retries + 1} attempts at ${config.endpoint}`
  );
}

// Gate before traffic cutover
await waitForHealthy({
  endpoint: "https://green.internal/health",
  expectedStatus: 200,
  timeoutMs: 5_000,
  retries: 10,
  retryDelayMs: 3_000,
});

// Shift traffic
await updateLoadBalancerTarget("green");

Canary shifts a small percentage of traffic (5-10%) to the new version while the old version handles the rest. You observe error rates and latency in the canary cohort before promoting fully. This is the right pattern when you cannot afford full rollback risk (high-traffic services, large state dependencies) or when the change’s correctness depends on production traffic patterns you cannot reproduce in staging.

Kubernetes makes this concrete with weighted routing via an Ingress or service mesh:

# Canary at 10% via Nginx Ingress annotations
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-service-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service-v2
                port:
                  number: 3000

The existing Ingress without the canary annotation continues serving 90% of traffic to the stable version. You bump the weight incrementally as confidence grows, then delete the canary Ingress and update the stable deployment.

Secrets Injection Without Baking Them In

The most common mistake in golden image pipelines is treating secrets like configuration. They are not. A secret baked into an image is a secret that lives forever in your registry history, accessible to anyone who can pull the image. Environment variables injected at build time in a RUN layer are visible in the image manifest.

The correct model: images are configuration-free for secrets. At runtime, the container fetches or receives secrets from an external source. Three patterns cover most cases:

Environment variable injection via orchestrator: Kubernetes injects secrets as environment variables from Secret objects. The secret value never touches the image. The application reads process.env.DATABASE_URL at startup. This is the lowest-friction approach. The tradeoff: environment variables are visible in /proc/1/environ and show up in process listings if you are not careful.

Volume-mounted secrets files: Kubernetes mounts the secret as a file in the container’s filesystem. The application reads the file. This avoids environment variable exposure and supports secrets that rotate (Kubernetes updates the mounted file when the Secret object changes). The tradeoff: the application must handle file reads and rotation (re-reading the file periodically or on SIGHUP).

Runtime secret fetch via SDK: The application itself calls AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager at startup using an identity attached to the pod (IRSA, Workload Identity, Vault agent sidecar). This is the most secure model because secrets are fetched with minimal scope, rotation is handled server-side, and there is an audit trail per access. The tradeoff: startup latency and dependency on the secrets store being available.

A runtime fetch pattern in TypeScript using AWS Secrets Manager:

import {
  SecretsManagerClient,
  GetSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";

interface AppSecrets {
  databaseUrl: string;
  jwtSecret: string;
}

async function loadSecrets(secretArn: string): Promise<AppSecrets> {
  const client = new SecretsManagerClient({ region: "us-east-1" });

  const response = await client.send(
    new GetSecretValueCommand({ SecretId: secretArn })
  );

  if (!response.SecretString) {
    throw new Error(`Secret ${secretArn} has no string value`);
  }

  const raw = JSON.parse(response.SecretString);

  // Validate required fields before returning
  if (!raw.database_url || !raw.jwt_secret) {
    throw new Error("Secret is missing required fields");
  }

  return {
    databaseUrl: raw.database_url,
    jwtSecret: raw.jwt_secret,
  };
}

// Application startup
const secrets = await loadSecrets(process.env.SECRET_ARN!);
const app = createApp(secrets);
await app.listen(3000);

The image contains no credentials. The SECRET_ARN environment variable (not a secret itself) tells the application where to look. IAM policy controls which pods can read which secrets.

Immutable Infrastructure and GitOps

These two practices reinforce each other naturally. GitOps declares the desired state in Git and continuously reconciles the running environment toward it. Immutable infrastructure ensures that the running environment is exactly what was declared: a specific, known image digest.

Without immutability, a GitOps reconciler can declare the correct image but have that image’s runtime behavior diverge because someone modified the running container’s filesystem or environment. With immutability, the reconciler’s declaration is the complete description: image digest plus environment configuration.

The combination gives you:

• Full audit trail: every change to production is a Git commit with author, timestamp, and diff
• Reproducible rollback: previous state is a previous image digest plus previous Git state
• Drift elimination: the reconciler continuously enforces the declared digest; any deviation is corrected automatically

The practical integration point is the image tag update pattern. When CI produces a new image, it commits the new digest to the GitOps repository. The GitOps operator (Flux or ArgoCD) detects the commit and triggers a deployment. The deployment replaces running pods with the new image. If the health check fails, the operator can be configured to revert the commit automatically.

Tradeoffs

Dimension	Immutable infrastructure	Mutable infrastructure
Configuration drift	Structurally impossible after deploy	Accumulates over time
Deployment speed	Slower (build + push + deploy)	Faster (in-place update)
Rollback	Switch to previous image (fast, reliable)	Reverse the change (slow, error-prone)
Build pipeline complexity	Higher (image pipelines, registry management)	Lower (deploy scripts)
Debugging running issues	Harder (no SSH-and-fix; requires reproduce in image)	Easier (SSH in, test changes live)
Storage cost	Higher (registry retains many image versions)	Negligible
Stateful workloads	Poor fit (databases, queues)	Natural fit
Secret management	Requires runtime injection discipline	Can rely on in-place config changes
Incident response	Requires pre-built runbooks; no live edits	Live edits possible
Audit trail	Comprehensive (every change is a build)	Sparse (SSH sessions are often unlogged)

Production Considerations

Image retention policy: A CI pipeline building on every commit to main produces many images. Without a retention policy, registry storage costs grow unboundedly. Tag images with both the commit SHA and a latest pointer. Keep the last 30 SHA-tagged images for production rollback. Archive older ones to cheaper cold storage or delete them. Never delete an image that is currently referenced by a running deployment.

Base image updates: Your golden image depends on a base image (Node 20 Alpine, Amazon Linux 2023). Base images receive security patches on their own schedule. Automating base image updates is non-trivial: you need a process that periodically rebuilds from an updated base, runs your full test and scan pipeline, and promotes the result. Dependabot or Renovate can open pull requests for base image version bumps, but the promotion pipeline still needs to run.

Image scan failures: A critical CVE in a dependency does not always mean you cannot ship. Your team needs a process for triaging scan failures: Is the vulnerability exploitable in your context? Is a patched version available? What is the remediation timeline? A blanket “block on any critical CVE” policy will stop your pipeline for vulnerabilities in packages that are not even loaded at runtime. Configure your scanner to ignore non-applicable CVEs with documented justifications in a .trivyignore or equivalent file checked into the repository.

Cold start and image pull time: Large images slow deployments. A 2 GB image pulling across a fleet of nodes before traffic shifts is a meaningful delay. Use multi-stage builds to keep production images under 200 MB where possible. Configure your container runtime to pre-pull images before traffic shift, not during. Enable registry mirrors inside your VPC or data center to avoid repeated egress costs and latency on the same image.

Handling stateful dependencies: Immutable infrastructure applies cleanly to stateless services. For stateful services (databases, message queues, caches with persistent data), the model is different. You manage them with configuration management tools, managed services, or operators that understand state. Do not force immutability on workloads where state is the point. The goal is eliminating configuration drift, not achieving immutability as a philosophical ideal.

Emergency access during incidents: When a service is misbehaving and you cannot reproduce the issue outside production, the immutable model creates friction. Your team cannot SSH in and run a debug binary. The mitigation is to build observability in: structured logs, distributed traces, remote profiling endpoints that can be enabled via feature flag. Treat the inability to SSH as a forcing function for better instrumentation, not as a blocker to fix after the incident.

When Mutable Infrastructure Is the Right Choice

Immutable infrastructure is not always the correct answer. Situations where mutable is genuinely better:

• Developer environments and local boxes: Rebuilding an image for every dependency change during development is friction with no reliability benefit.
• Long-lived stateful services with frequent config changes: A PostgreSQL instance with dozens of configuration parameters that need tuning is better managed with a configuration management tool that applies changes in-place.
• Low-traffic internal tools with slow build pipelines: If your image pipeline takes 20 minutes and the tool has two users, the operational overhead is not worth it.
• Cost-constrained environments: Running two production environments for blue-green has a real cost. For non-critical services in resource-constrained organizations, in-place deployments with a tested rollback procedure are a reasonable tradeoff.

The principle behind immutable infrastructure is eliminating uncontrolled change. If you can achieve that with a simpler model for a given workload, use the simpler model.

Closing

Configuration drift is a mechanism problem, not a discipline problem. Immutable infrastructure eliminates the mechanism by making in-place modification architecturally impossible. The operational cost is real: you need a build pipeline, a registry with promotion workflows, deployment patterns that handle traffic cutover, and runtime secret injection. For stateless production services where reliability matters, that cost is worth it. The rollbacks are reliable, the audit trail is complete, and the gap between “what we think is running” and “what is actually running” closes to zero.