Build Caching Strategies for CI/CD: Docker Layer Caching, Dependency Caching, and Reducing Pipeline Times by 80%

A pipeline that takes 18 minutes to run will be skipped. Not intentionally, but developers stop waiting for it, they stop fixing it when it breaks, and eventually they push directly to main because running locally was faster anyway. Build time is not a comfort metric. It is a proxy for how much the team trusts the pipeline.

The good news: most pipelines have 60-80% of their runtime spent re-doing work that did not change. Downloading the same 847 npm packages. Rebuilding Docker layers that have not been touched in two weeks. Re-running tests against code that was not modified. Every one of these is recoverable with caching.

This article covers the four layers where caching has the highest impact, with concrete configuration for GitHub Actions and GitLab CI, and real numbers from before/after measurements.

Where Time Goes in a Typical Pipeline

Before writing any YAML, instrument your pipeline. Most CI platforms show per-step timing. A representative unoptimized Node.js monorepo pipeline on GitHub Actions looks like this:

Step	Time (before)	Time (after caching)
Checkout	12s	12s
npm install	4m 20s	18s
Docker build (app)	6m 45s	1m 10s
Docker build (worker)	5m 30s	45s
Run tests	3m 15s	1m 40s
Push images	1m 20s	1m 20s
Total	21m 22s	5m 25s

The 75% reduction comes from three changes: lockfile-based dependency caching, proper Docker layer ordering, and selective test execution. None of these require new infrastructure. They require understanding what is actually being cached and why cache hits fail.

Docker Layer Caching

Docker builds are deterministic given the same inputs, but CI runners start with a cold daemon by default. Cache misses cascade: one changed layer invalidates every layer below it.

Layer Ordering is the Entire Game

The most common mistake is copying application code before installing dependencies. This invalidates the dependency installation layer on every commit:

# BAD: code copy before install invalidates node_modules on every change
FROM node:20-alpine
WORKDIR /app
COPY . .
RUN npm ci
RUN npm run build

The fix is to separate the dependency installation layer from the code:

# GOOD: dependency layer only rebuilds when lockfile changes
FROM node:20-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --only=production

FROM node:20-alpine AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/package.json ./
EXPOSE 3000
CMD ["node", "dist/index.js"]

The multi-stage build also reduces the final image size: the runner stage has no devDependencies and no source files.

GitHub Actions: Registry Cache

GitHub Actions runners have no persistent Docker layer cache between jobs. You need to use a registry as the cache backend:

# .github/workflows/build.yml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}/app:${{ github.sha }}
          cache-from: type=registry,ref=ghcr.io/${{ github.repository }}/app:cache
          cache-to: type=registry,ref=ghcr.io/${{ github.repository }}/app:cache,mode=max

The mode=max flag caches all layers, including intermediate stages. Without it, only the final stage is cached, which misses most of the benefit from multi-stage builds.

GitLab CI: Built-in Docker Layer Cache

GitLab CI with Docker-in-Docker has a simpler path using DOCKER_BUILDKIT and the local daemon:

# .gitlab-ci.yml
build:
  image: docker:24
  services:
    - docker:24-dind
  variables:
    DOCKER_BUILDKIT: "1"
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - |
      docker buildx build \
        --cache-from type=registry,ref=$CI_REGISTRY_IMAGE:cache \
        --cache-to type=registry,ref=$CI_REGISTRY_IMAGE:cache,mode=max \
        --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA \
        --push \
        .

If you use Kaniko instead of Docker-in-Docker (common in Kubernetes-native GitLab runners), replace the docker build step with a Kaniko executor job that uses --cache=true and --cache-repo.

Dependency Caching

npm install is a network operation that fetches the same packages on every pipeline run unless you cache the result. The cache key must be based on the lockfile, not the package.json, because only the lockfile captures exact resolved versions.

GitHub Actions: actions/cache

jobs:
  install:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
          # cache: 'pnpm' also works if you use pnpm

      - name: Install dependencies
        run: npm ci

The cache parameter on actions/setup-node handles everything: it generates a cache key from package-lock.json, restores the cache before install, and saves it after. On a cache hit, npm ci still runs but it finds the node_modules already populated.

For pnpm, the setup is slightly different because pnpm uses a global store rather than a per-project node_modules:

      - name: Setup pnpm
        uses: pnpm/action-setup@v4
        with:
          version: 9

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "pnpm"

      - name: Install dependencies
        run: pnpm install --frozen-lockfile

The distinction matters: pnpm’s global store means the cache is shared across all packages in a monorepo. One cache restore instead of one per workspace.

GitLab CI: cache key from lockfile

install:
  image: node:20-alpine
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
      - .npm/
  script:
    - npm ci --cache .npm --prefer-offline

The key.files directive in GitLab CI computes a hash of the listed files and uses it as the cache key. If package-lock.json does not change, the cache is restored and npm ci --prefer-offline skips the registry fetch entirely.

For pnpm in GitLab:

install:
  image: node:20-alpine
  cache:
    key:
      files:
        - pnpm-lock.yaml
    paths:
      - .pnpm-store/
  before_script:
    - npm install -g pnpm
    - pnpm config set store-dir .pnpm-store
  script:
    - pnpm install --frozen-lockfile

Turborepo Remote Caching for Monorepos

In a monorepo, you often have 12 packages but only 2 of them changed. Without task-level caching, you rebuild all 12. Turborepo solves this by hashing the inputs (source files, environment variables, build config) and storing outputs. A cache hit means the task result is restored from cache without re-running.

Remote caching extends this to CI: the cache is shared across all runners and all branches.

Setup

// turbo.json
{
  "$schema": "https://turbo.build/schema.json",
  "remoteCache": {
    "enabled": true
  },
  "tasks": {
    "build": {
      "dependsOn": ["^build"],
      "inputs": ["src/**", "tsconfig.json", "package.json"],
      "outputs": ["dist/**"]
    },
    "test": {
      "dependsOn": ["build"],
      "inputs": ["src/**", "test/**", "jest.config.*"],
      "outputs": []
    },
    "lint": {
      "inputs": ["src/**", ".eslintrc.*"],
      "outputs": []
    }
  }
}

The inputs field is critical. It determines what gets hashed. If your build depends on environment variables, add them to the env field rather than inputs. Missing an input means stale cache hits on changes that should have triggered a rebuild.

GitHub Actions with Turborepo Remote Cache

Turborepo’s remote cache integrates with Vercel (their hosted offering) or a self-hosted server. For self-hosting, ducktape and turborepo-remote-cache are common open-source options.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2  # Turborepo needs HEAD and HEAD~1 for affected packages

      - uses: pnpm/action-setup@v4
        with:
          version: 9

      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "pnpm"

      - name: Install dependencies
        run: pnpm install --frozen-lockfile

      - name: Build
        run: pnpm turbo build
        env:
          TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
          TURBO_TEAM: ${{ vars.TURBO_TEAM }}

      - name: Test
        run: pnpm turbo test
        env:
          TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
          TURBO_TEAM: ${{ vars.TURBO_TEAM }}

With remote caching active, a second run on the same commit hash completes in under 30 seconds for a 12-package monorepo because every task is a cache hit.

Affected Package Detection Without Remote Cache

If remote caching is not an option, Turborepo’s --filter flag combined with git diff can scope execution to affected packages:

      - name: Get base SHA
        id: base
        run: echo "sha=$(git rev-parse HEAD~1)" >> $GITHUB_OUTPUT

      - name: Build affected packages
        run: |
          pnpm turbo build \
            --filter="...[origin/main]" \
            --concurrency=4

The [origin/main] syntax tells Turborepo to build only packages with changes compared to main. This is less reliable than remote caching because it does not account for packages whose dependencies changed but the package itself did not, but it is useful as a fallback.

Test Result Caching and Selective Execution

Rebuilding and rerunning tests is one thing. Rerunning tests that cover untouched code is waste. Two approaches exist: hash-based test skipping and file-based test selection.

Jest with --changedSince

Jest has a built-in flag to run only tests related to changed files:

      - name: Run tests
        run: |
          npx jest \
            --changedSince=origin/main \
            --passWithNoTests \
            --ci \
            --coverage

The --passWithNoTests flag prevents a false failure when no tests are selected (common on infrastructure-only changes).

Caching Jest’s Transform Cache

Jest compiles TypeScript on every run by default. The transform cache persists the compiled output, and it is safe to cache between runs:

      - name: Restore Jest cache
        uses: actions/cache@v4
        with:
          path: /tmp/jest-cache
          key: jest-${{ runner.os }}-${{ hashFiles('**/tsconfig.json', 'jest.config.*') }}-${{ github.sha }}
          restore-keys: |
            jest-${{ runner.os }}-${{ hashFiles('**/tsconfig.json', 'jest.config.*') }}-

      - name: Run tests
        run: npx jest --cacheDirectory /tmp/jest-cache --ci

The restore key fallback (restore-keys) allows a partial cache hit: if no exact match exists for the current SHA, it restores the most recent cache from the same tsconfig hash.

Artifact Reuse Across Pipeline Stages

Building the same binary twice in a pipeline is common when each stage is isolated. The fix is to build once and pass the artifact forward.

GitHub Actions: Artifact Upload/Download

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci
      - run: npm run build
      - name: Upload build artifact
        uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/
          retention-days: 1

  test-integration:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist/
      - run: npm ci --only=production
      - run: npm run test:integration

  deploy:
    needs: [build, test-integration]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist/
      - run: ./deploy.sh

The build job runs once. Every downstream job restores the artifact. If the build takes 3 minutes and you have 4 downstream jobs, artifact reuse saves 9 minutes of redundant builds.

GitLab CI: Artifacts

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

test:integration:
  stage: test
  needs: [build]
  script:
    - npm ci --only=production
    - npm run test:integration

deploy:production:
  stage: deploy
  needs: [build, test:integration]
  script:
    - ./deploy.sh

The needs keyword enables DAG-style pipelines in GitLab: jobs run as soon as their dependencies are complete rather than waiting for the entire previous stage to finish.

Cache Invalidation Strategies

A cache that never expires is a stale cache waiting to cause a confusing failure. These are the failure modes to plan for:

Dependency cache with wrong key: Using package.json as the cache key instead of the lockfile means changes to version ranges hit the cache even when resolved versions changed. Always key on the lockfile.

Docker layer cache with secrets: If a build argument contains a secret (API key, private registry token), that layer will be cached and the secret will be stored in the cache. Use multi-stage builds to ensure secrets are not present in cached layers, and prefer --secret mounts over build args.

Test cache returning stale results: Jest’s transform cache is safe to cache. Jest’s result cache (which tracks which tests passed) is not safe to cache across code changes. Keep them separate.

Turborepo stale cache on environment variable changes: If your build reads process.env.API_URL, add it to the task’s env field. Otherwise, Turborepo will return cached output even when the variable changed.

// turbo.json - include env vars in the cache hash
{
  "tasks": {
    "build": {
      "dependsOn": ["^build"],
      "inputs": ["src/**", "tsconfig.json"],
      "outputs": ["dist/**"],
      "env": ["API_URL", "NODE_ENV", "NEXT_PUBLIC_APP_URL"]
    }
  }
}

Caching Approach Tradeoffs

Approach	Setup effort	Cache hit rate	Failure risk	Best for
Docker registry cache	Medium	High (stable deps)	Low	Any containerized app
npm/pnpm lockfile cache	Low	Very high	Very low	All Node.js projects
Turborepo remote cache	High	Very high (monorepo)	Medium (stale env)	Monorepos with 5+ packages
Jest transform cache	Low	High	Low	TypeScript projects
Artifact upload/download	Low	N/A (not a cache)	Low (size limit)	Multi-stage pipelines
Affected test selection	Low	Medium	Medium (detection gaps)	PRs on large test suites

Production Considerations

Cache size limits: GitHub Actions caches are limited to 10GB per repository. GitLab caches are configurable but typically have runner-level limits. Track cache sizes in CI logs and prune keys that are no longer in use. The actions/cache action includes eviction by LRU, but you should name cache keys so expired versions are not restored.

Cache poisoning: A compromised dependency in the cache can persist across runs. For security-sensitive builds, consider invalidating the dependency cache weekly by including the week number in the key: npm-${{ runner.os }}-${{ hashFiles('package-lock.json') }}-${{ env.CACHE_VERSION }} where CACHE_VERSION is a repo-level variable you increment on demand.

Branch cache isolation: By default, GitHub Actions caches created on a branch are accessible from the same branch and from the base branch. PRs can restore caches from main but cannot write to main’s cache. This is the correct behavior. In GitLab CI, configure cache scoping explicitly with key: $CI_COMMIT_REF_SLUG for branch isolation or key: $CI_DEFAULT_BRANCH to share a cache across all branches.

Parallelism and cache write contention: If two jobs write to the same cache key simultaneously, the last write wins. In GitHub Actions this is harmless because writes only happen after the job succeeds. In GitLab CI with runner-level caching, concurrent jobs can overwrite each other’s caches. Scope cache keys to include the job name if you run parallel test shards.

Runner warm-up: If you use self-hosted runners that persist between jobs, Docker layer caching is free: the daemon keeps layers on disk. Ephemeral runners (default on GitHub Actions and GitLab SaaS) require explicit cache-from/cache-to with a remote backend for Docker, but npm caches are handled by the cache action.

Putting It Together

The order of operations for implementing caching in an existing pipeline:

1. Add lockfile-based dependency caching. This is the lowest risk, highest reward change. It takes 15 minutes and typically saves 3-5 minutes per run.

2. Fix Docker layer ordering. Move COPY package*.json before COPY . . and add registry cache configuration. This is a code change and a CI config change, but the impact on build time is immediate.

3. Add artifact upload/download between stages if you have parallel jobs that each build the same thing.

4. Add Jest transform caching with a tsconfig-keyed cache key.

5. If you are on a monorepo, add Turborepo remote caching. This requires infrastructure setup but is the most significant change for repositories with many packages.

Every change should be measured. Most CI platforms show per-step duration in the job log. Note the before time, make one change, note the after time. Compounding five small improvements is how pipelines go from 20 minutes to 4 minutes without heroics.

The goal is not the fastest possible pipeline. The goal is a pipeline that developers trust because it finishes before they context-switch. That threshold is roughly 5 minutes for PR builds and 10 minutes for full builds. Caching is the most direct path to that number without rewriting the pipeline architecture.