CodeBuild Docker layer caching — from 6-minute builds to 90 seconds

The team’s main API image takes 6 minutes 12 seconds to build on every commit. The Dockerfile is 18 lines, the binary is small, and most builds change one Go file. Nothing should take that long. The culprit is straightforward — CodeBuild’s default Docker setup runs every build from cold, with no layer reuse and no BuildKit. Three configuration changes drop the same build to 88 seconds on a warm cache.

What’s happening

CodeBuild gives each build a fresh runtime container. The Docker daemon inside that container starts with no layer cache. Every RUN apt-get install, every COPY package.json && npm install, every base-image pull happens from zero on every build. For an image with one expensive RUN (an npm install, a go mod download, a pip install), that one step dominates the build.

Three caching mechanisms exist, and they solve different problems:

1. LOCAL_DOCKER_LAYER_CACHE — CodeBuild’s setting that pins builds to the same EC2 host between runs and keeps the Docker layer store warm on that host. Free, no Dockerfile changes. Caveat: works only on Linux Medium/Large/2XLarge compute and only when the host is reused (no guarantee, but usually yes for the same project within a few hours).

2. --cache-from against ECR — explicitly pull a previously-built image from ECR and tell BuildKit to use its layers. Works across hosts. Requires BuildKit enabled and the cache image tagged consistently. Pulls ~50-200 MB at the start of each build but reuses layers across cold hosts.

3. BuildKit inline cache — BuildKit-only feature that embeds cache metadata in the pushed image. Combined with --cache-from, it lets BuildKit fetch only the layer manifests, not the full layer blobs, to decide what to reuse.

The right setup combines all three. LOCAL_DOCKER_LAYER_CACHE is free insurance for warm hosts. --cache-from covers cold hosts. BuildKit makes both efficient.

Real timing on a Node app with a 280 MB node_modules:

• Cold build, no cache: 6 min 12 sec
• Warm LOCAL_DOCKER_LAYER_CACHE only: 1 min 48 sec
• ECR --cache-from only, cold host: 2 min 12 sec
• LOCAL_DOCKER_LAYER_CACHE + --cache-from, warm host: 88 sec
• BuildKit + both caches, warm host: 62 sec

The improvement is the difference between a developer pushing four commits in a morning and not being able to.

How to verify

First, enable LOCAL_DOCKER_LAYER_CACHE on the CodeBuild project. This is project configuration, not buildspec:

aws codebuild update-project \
  --name web-api \
  --cache type=LOCAL,modes=LOCAL_DOCKER_LAYER_CACHE,LOCAL_SOURCE_CACHE

Then enable BuildKit in the buildspec and configure --cache-from against ECR:

version: 0.2

env:
  variables:
    DOCKER_BUILDKIT: "1"
    ECR_REGISTRY: 123456789012.dkr.ecr.us-east-1.amazonaws.com
    IMAGE_NAME: web-api

phases:
  pre_build:
    commands:
      - aws ecr get-login-password --region us-east-1
          | docker login --username AWS --password-stdin $ECR_REGISTRY
      - docker pull $ECR_REGISTRY/$IMAGE_NAME:cache || true

  build:
    commands:
      - export IMAGE_TAG=${CODEBUILD_RESOLVED_SOURCE_VERSION:0:7}
      - |
        docker build \
          --cache-from $ECR_REGISTRY/$IMAGE_NAME:cache \
          --build-arg BUILDKIT_INLINE_CACHE=1 \
          -t $ECR_REGISTRY/$IMAGE_NAME:$IMAGE_TAG \
          -t $ECR_REGISTRY/$IMAGE_NAME:cache \
          .

  post_build:
    commands:
      - docker push $ECR_REGISTRY/$IMAGE_NAME:$IMAGE_TAG
      - docker push $ECR_REGISTRY/$IMAGE_NAME:cache

The :cache tag is the cache reference. Every successful build pushes a fresh :cache image with inline cache metadata; every build pulls it (|| true to not fail on the first build) and feeds it to --cache-from.

The Dockerfile needs to be cache-friendly. The classic mistake — COPY . . && RUN npm install — invalidates the install layer on every source change. The correct shape is to copy dependency manifests, install, then copy source:

FROM node:20-alpine AS build
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN corepack enable && pnpm install --frozen-lockfile
COPY . .
RUN pnpm build

FROM node:20-alpine
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
CMD ["node", "dist/index.js"]

The dependency install layer is cached whenever pnpm-lock.yaml doesn’t change — which is most commits.

For build-time secrets (an npm token for a private registry, a Sentry auth token), BuildKit’s --secret mount is the right pattern. The secret never lands in a layer:

pre_build:
  commands:
    - aws secretsmanager get-secret-value
        --secret-id /apps/web/npm-token
        --query SecretString --output text > /tmp/npm-token
build:
  commands:
    - DOCKER_BUILDKIT=1 docker build
        --secret id=npm-token,src=/tmp/npm-token
        --cache-from $ECR_REGISTRY/$IMAGE_NAME:cache
        -t $ECR_REGISTRY/$IMAGE_NAME:$IMAGE_TAG
        .
    - rm /tmp/npm-token

In the Dockerfile, the install step mounts the secret:

RUN --mount=type=secret,id=npm-token,target=/root/.npmrc \
    npm install --production

The secret is available during that one RUN and not stored in the image. The old pattern of ARG NPM_TOKEN plus --build-arg bakes the token into image history and is not safe.

To confirm the cache is working, the build log shows CACHED next to each reused step:

#7 [build 3/5] RUN corepack enable && pnpm install --frozen-lockfile
#7 CACHED

#8 [build 4/5] COPY . .
#8 DONE 0.4s

Without cache, every step says DONE with a duration. With cache, expensive steps say CACHED with zero duration.

The fix

The stack of changes that takes a CodeBuild Docker build from cold to warm-fast:

1. Enable LOCAL_DOCKER_LAYER_CACHE on the project. One-line update-project call. Free, no buildspec change. Catches the warm-host case which is the common case for active projects.

2. Set DOCKER_BUILDKIT: "1" in env. BuildKit is faster on every operation and is required for --cache-from inline cache and --mount=type=secret. The legacy builder is end-of-life.

3. Push a :cache tag, pull it in pre_build, pass to --cache-from. This handles the cold-host case. The :cache tag is the build’s gift to the next build.

4. Reorder the Dockerfile so dependency installs are stable layers. COPY package.json before COPY . .. COPY go.mod go.sum before COPY *.go .. The dependency layer must invalidate only on dependency changes.

5. Use --secret for build-time secrets, never --build-arg. --build-arg values are visible in image history (docker history); --mount=type=secret is not in the image at all. The token leaks if anyone pulls the image and runs docker inspect.

6. Choose compute by build profile, not default. The default BUILD_GENERAL1_SMALL is 3 GB RAM, 2 vCPU. For Docker builds with a big node_modules, the SMALL is memory-bound and slower than MEDIUM despite costing less per minute. The MEDIUM finishes the build faster and the total per-build cost drops. Measure both.

7. Don’t use multi-stage caching on --cache-from without inline cache. Without BUILDKIT_INLINE_CACHE=1, the pushed image only contains the final stage’s layers. Intermediate stages are not in the registry and not available to --cache-from. Set BUILDKIT_INLINE_CACHE=1 and BuildKit publishes the full graph.

Two patterns to watch for:

• The latest cache trap. Using :latest as the cache tag works for projects with a single branch, but breaks when feature branches push their own caches and overwrite each other. Use a branch-prefixed tag like :cache-main, :cache-feature-foo.

• The privileged-mode requirement. Building Docker images on CodeBuild requires the project’s privilegedMode: true in the environment block. Without it, the Docker daemon won’t start and every build fails on docker: command not found. This is project setup, not buildspec.

One last note on the math. A 6-minute build that runs 30 times a day costs 180 build-minutes daily. The same build at 90 seconds costs 45 build-minutes. Across a year of working days, the saved compute pays for the time you spent reading this article many times over — and the developer experience improvement is the bigger win.

Stack Harbor runs CodeBuild + CodePipeline at production scale as part of Managed AWS onboarding and ongoing Managed Cloud Ops. If your Docker builds run for 10 minutes when they should run for 90 seconds, start with an infrastructure consult.