Docker Build Cache Optimization for Local Development

A local image rebuild that should finish in five seconds takes five minutes because BuildKit re-runs a dependency install that nothing changed. This guide makes rebuild cost proportional to the edit: it configures BuildKit as the builder, orders Dockerfile layers by how often each input actually changes, mounts package-manager caches so a legitimate reinstall fetches from disk instead of the network, splits heavy work across build stages, and reads the build log to pinpoint exactly which layer missed. It sits within the broader containerized local environment patterns and complements the Compose-focused walkthrough in optimizing Docker Compose for fast local rebuilds; where that page fixes one service's Dockerfile end to end, this one is the reference for the cache mechanics that make the fix work and keep working.

The build cache is the single largest lever on inner-loop speed once a stack is containerized. Every unnecessary rebuild is paid not once but on every edit, by every engineer, on every branch — the cost compounds silently until a team accepts a multi-minute feedback loop as normal. The techniques below are ordered the way you should apply them: turn on BuildKit, fix ordering (which removes most of the waste for free), add cache mounts and multi-stage separation (which remove the residual cost), and finally learn the diagnostic commands so a future regression is a two-minute investigation instead of a mystery.

Prerequisites

Confirm the toolchain before changing any Dockerfile, because half of "the cache does not work" reports are a builder that silently fell back to the legacy engine or a Docker version too old to honour cache-mount syntax.

• Docker Engine 23.0 or newer (docker --version). BuildKit is the default builder from Engine 23.0; on older engines it must be opted into explicitly and several cache features are unavailable.
• Compose v2 (docker compose version) — the plugin, invoked as docker compose, not the legacy docker-compose Python script, which does not drive BuildKit consistently.
• Buildx present (docker buildx version). Buildx ships with Docker Desktop and modern Engine installs and provides the docker-container driver needed for exportable local caches.
• A Dockerfile that begins with # syntax=docker/dockerfile:1 so RUN --mount=type=cache and other BuildKit-only instructions parse. Without that directive, older frontends reject the syntax with a parse error.

Verify all four in one pass and fail loudly if any is missing, so the rest of the guide runs against a known-good baseline:

#!/usr/bin/env bash
set -euo pipefail
docker --version
docker compose version
docker buildx version
# Confirm BuildKit is the active builder, not the legacy engine
docker buildx inspect default --bootstrap | grep -E 'Driver|BuildKit'

If docker buildx inspect reports the docker driver, exportable type=local and type=registry caches are not available; create a container-driver builder with docker buildx create --use --name local-cache. Everything downstream assumes BuildKit is live and the syntax directive is present at the top of every Dockerfile you touch.

Section 1 - Make BuildKit the builder and prove it is active

The legacy build engine evaluates instructions strictly top to bottom and rebuilds sequentially; BuildKit models the Dockerfile as a directed acyclic graph, solves independent branches in parallel, skips stages that no target needs, and can import and export cache to external backends. On the same Dockerfile, that difference alone often halves a cold build — but only if BuildKit is genuinely the active builder, which is not guaranteed on inherited machines or CI images that pin an old engine.

1. Force BuildKit on for the current shell so a legacy fallback cannot mask a caching bug:

   #!/usr/bin/env bash
   set -euo pipefail
   export DOCKER_BUILDKIT=1
   export COMPOSE_DOCKER_CLI_BUILD=1
   docker compose build --progress=plain app 2>&1 | head -n 3

   A BuildKit build prints lines prefixed with #1 [internal] load build definition; a legacy build prints Step 1/8 : FROM …. If you see Step N/M, BuildKit is not engaged and no cache-mount instruction will work.

   The two environment variables above are the belt-and-braces switch for an older engine or a CI image that has not migrated. On Engine 23 and later BuildKit is already the default, but setting them explicitly costs nothing and removes a whole class of "works on my laptop" confusion where one machine builds with BuildKit and another silently falls back. If you want the setting to persist across shells rather than exporting it each session, add "features": { "buildkit": true } to /etc/docker/daemon.json (or the Docker Desktop settings) and restart the daemon, so every contributor's engine agrees.

2. Add the syntax directive as the literal first line of every Dockerfile, which selects the current stable frontend and unlocks cache mounts, bind mounts, and secrets:

   # syntax=docker/dockerfile:1
   FROM node:20.11.0-alpine@sha256:8f31d000000000000000000000000000000000000000000000000000000000ab
   WORKDIR /app

3. Pin the base image by digest, not a floating tag. A bare node:20-alpine can be republished upstream, which changes the FROM layer key and invalidates every layer beneath it even though your code is untouched — a cache miss with no local cause is almost always a moved tag.

The drift diagnostic for this section is a one-liner that fails when the wrong builder is selected, suitable for a make target or a shell profile guard:

#!/usr/bin/env bash
set -euo pipefail
driver="$(docker buildx inspect --bootstrap | awk -F': ' '/Driver/{print $2; exit}')"
test "$driver" = "docker-container" || { echo "expected docker-container driver, got: $driver"; exit 1; }
echo "BuildKit container driver active — exportable cache available"

Section 2 - Order layers by change frequency

Docker builds are content-addressed and evaluated in Dockerfile order. Each instruction produces a layer keyed by the instruction text plus the checksum of its inputs; the instant one layer's key changes, that layer and every instruction below it are invalidated and re-executed. Ordering is therefore not a stylistic choice — it is the primary determinant of how much work a one-line edit costs. The rule is mechanical: place the inputs that change least often at the top and the inputs that change most often at the bottom, so a frequent edit only invalidates the cheap tail of the build.

The canonical failure is COPY . . placed above the dependency install. That copy hashes the entire source tree, so it changes on every edit; sitting above RUN npm ci or RUN pip install, each edit invalidates the install below it and re-downloads packages that did not change. Invert the frequency instead — copy the manifests, install, then copy the source last:

1. Copy only the manifests first, so the layer that gates the expensive install changes only when dependencies actually change:

   # syntax=docker/dockerfile:1
   FROM python:3.12-slim@sha256:2c8f000000000000000000000000000000000000000000000000000000000abc
   WORKDIR /app
   COPY pyproject.toml poetry.lock ./
   RUN pip install --no-cache-dir poetry && poetry install --no-root
   COPY . .
   CMD ["python", "-m", "app"]

2. Keep the build context small with a tight .dockerignore. A bloated context ships node_modules, .git, and build output to the daemon, inflating the final COPY layer, slowing the context upload, and making the COPY . . checksum slower to compute:

   .git
   node_modules
   dist
   build
   *.log
   .env
   __pycache__

3. Split a single high-churn copy into staged copies when parts of your source change at different rates. Static assets, a generated API schema, or vendored code can be copied before the application source so an edit to one file does not invalidate the others.

The drift diagnostic here is to build twice with only a whitespace change and confirm the install layer reports CACHED. A blank (non-CACHED) install line after a no-op edit means the ordering is still wrong:

#!/usr/bin/env bash
set -euo pipefail
touch app/main.py
docker compose build --progress=plain app 2>&1 \
  | grep -E 'CACHED|RUN (pip|npm|poetry)'

Section 3 - Add cache mounts for package managers

Correct ordering keeps the install layer CACHED when nothing changed, but a legitimate dependency bump still re-runs it — and by default that means re-downloading every package from the network, because the package manager's cache lives inside the ephemeral layer and is discarded when the layer rebuilds. A BuildKit cache mount fixes exactly this: it attaches a persistent directory to a single RUN step, outside the image layers, that survives across builds. When the install layer must re-run, the package manager finds most artifacts already on disk and only fetches what actually changed.

The mount target is the manager's own cache directory: /root/.npm for npm, /root/.cache/pip for pip, /root/.cache/go-build for Go, ~/.cargo/registry for Cargo. The layer key does not include the mount contents, so the mount never causes a cache miss — it only accelerates the step when it does re-run.

1. Mount the download cache on the install step. For npm, disable the offline-first heuristic only if you need strict lockfile installs:

   # syntax=docker/dockerfile:1
   FROM node:20.11.0-alpine@sha256:8f31d000000000000000000000000000000000000000000000000000000000ab
   WORKDIR /app
   COPY package.json package-lock.json ./
   RUN --mount=type=cache,target=/root/.npm \
       npm ci --prefer-offline
   COPY . .

2. Give the mount a stable id per manager when several stages or services share one cache, so a monorepo's web and worker images reuse the same download directory instead of each maintaining its own:

   RUN --mount=type=cache,id=pip,target=/root/.cache/pip \
       pip install -r requirements.txt

3. Export the layer cache itself for fresh clones through Compose, so a new hire's first build imports warm layers rather than compiling from scratch. This is distinct from the download mount above — it caches the built layers, not the package tarballs:

   # docker-compose.yml
   services:
     app:
       build:
         context: .
         cache_from:
           - type=local,src=/tmp/.buildx-cache
         cache_to:
           - type=local,dest=/tmp/.buildx-cache,mode=max

One property of cache mounts trips people up: the mount is shared and mutable across concurrent builds, which is fine for read-mostly download caches but wrong for a directory a build writes to non-atomically. For the package managers above the shared model is exactly what you want — npm, pip, and Cargo treat their cache as an append-only content store keyed by hash — but if you ever mount a directory that a build step rewrites in place, add sharing=locked so BuildKit serializes access instead of letting two parallel builds corrupt it. The default sharing=shared is correct for every example in this section.

A cache mount also does not persist into the image, which is the point: the downloaded tarballs never inflate the layer you ship. That means a reader inspecting the final image with docker history will not see the cache at all, and a cold build on a machine that has never populated the mount pays full network cost once before the mount is warm. Prime it deliberately on a fresh clone by running the install stage once, and every subsequent reinstall on that machine draws from the local store.

The measured effect stacks: ordering removes the re-run for unrelated edits, and the cache mount cuts the residual cost of a real reinstall. The bar chart below records warm-rebuild times on one representative Node service — a no-cache baseline, then ordering alone, then ordering plus a cache mount.

Section 4 - Split heavy work across build stages

A single-stage image bakes the compiler, dev headers, and full toolchain into the artifact you run, which bloats the image and — more relevant here — couples build-time and run-time work into one cache lineage where a change to either invalidates the other. Multi-stage builds separate concerns: a deps stage resolves packages, a build stage compiles, and a lean runtime stage copies only the finished output. BuildKit caches and parallelizes each stage independently, so editing application source never invalidates the dependency stage, and the toolchain never ships to production.

The ordering discipline from Section 2 applies within every stage, and the cache mounts from Section 3 attach to the stage that installs. The gain unique to this section is cache scope: because stages are separate graph nodes, a change confined to one stage leaves the others CACHED.

1. Name each stage and copy forward only the artifacts the next stage needs, never the whole previous filesystem:

   # syntax=docker/dockerfile:1
   FROM node:20.11.0-alpine@sha256:8f31d000000000000000000000000000000000000000000000000000000000ab AS deps
   WORKDIR /app
   COPY package.json package-lock.json ./
   RUN --mount=type=cache,target=/root/.npm npm ci --prefer-offline
   
   FROM deps AS build
   COPY . .
   RUN npm run build
   
   FROM node:20.11.0-alpine@sha256:8f31d000000000000000000000000000000000000000000000000000000000ab AS runtime
   WORKDIR /app
   ENV NODE_ENV=production
   COPY --from=deps /app/node_modules ./node_modules
   COPY --from=build /app/dist ./dist
   CMD ["node", "dist/server.js"]

2. Build only the target you need locally with --target so an iteration on tests does not pay for the production runtime assembly:

   #!/usr/bin/env bash
   set -euo pipefail
   docker build --target build -t app:dev .

3. Point Compose at the right stage per environment so local development uses the fuller build stage and production images use runtime, both from one Dockerfile:

   # docker-compose.yml
   services:
     app:
       build:
         context: .
         target: build
       command: npm run dev

The drift diagnostic confirms stages are isolated: edit source, rebuild, and check that the deps stage is skipped entirely. BuildKit prints CACHED for the whole stage when nothing in it changed:

#!/usr/bin/env bash
set -euo pipefail
echo "// touch" >> src/index.js
docker compose build --progress=plain app 2>&1 \
  | grep -E '\[deps|\[build|\[runtime|CACHED'

Section 5 - Diagnose a cache miss

When a build that should be fast is slow, resist the urge to add more cache configuration — first find which layer missed and why. BuildKit annotates every step as CACHED or blank; the first blank step is where the cache broke, and everything below it is the cost you are paying. The investigation is a fixed sequence: read the plain-progress log, identify the first non-cached step, then attribute the miss to one of a small set of causes.

1. Run a plain-progress build and isolate the cache boundary. The first line without CACHED is the culprit; every line after it is collateral:

   #!/usr/bin/env bash
   set -euo pipefail
   docker compose build --no-cache=false --progress=plain app 2>&1 \
     | grep -E '=> \[|CACHED' | head -n 20

2. Attribute the miss. A miss on the FROM line means a moved base tag — pin the digest. A miss on an install step after a source-only edit means COPY . . sits above the install — reorder per Section 2. A miss with no code change at all often means the build context changed: a new untracked file, an edited .dockerignore, or a timestamp-sensitive input.

3. Inspect layer sizes newest-first to find a fat layer sitting below the dependency install, which is the ordering smell that guarantees expensive misses:

   #!/usr/bin/env bash
   set -euo pipefail
   docker history --no-trunc "$(docker compose images -q app)" \
     | awk 'NR<=8 {print $1"  "$2"  "$3}'

4. Check cache-store health when misses look random. A pruned or size-capped build cache silently evicts layers, so a build that was warm yesterday rebuilds cold today:

   #!/usr/bin/env bash
   set -euo pipefail
   docker system df -v | grep -A2 'Build Cache'
   docker buildx du --verbose | head -n 12

Reading the log is faster once you know what a healthy trace looks like. On a warm rebuild after a source-only edit, every step through the install and any compile that does not depend on the edited file should read CACHED, and only the trailing COPY and the steps genuinely downstream of it should execute. If a step you expected to be cached runs, the fix is always to make its inputs stable: pin the base, narrow the COPY so it hashes fewer files, or move a volatile value later. Treat a full-width rebuild after a one-character edit as a bug with a root cause, never as an unavoidable cost — the log always names the layer, and the layer always names the fix.

The most common non-obvious cause is a build argument or environment value embedded in a layer key. An ARG BUILD_DATE=$(date) or a --build-arg GIT_SHA=… that changes every commit invalidates every layer below the ARG on every build; move such values as late as possible, ideally into the final stage or into runtime environment, so they never gate the cache for the expensive install and compile steps.

Platform caveats

The build cache lives inside the Docker daemon, which on macOS and Windows is a Linux virtual machine rather than a native engine. Every technique above works across platforms, but the mechanics of where the cache lives and how fast the context reaches it differ, and those differences explain most "fast for me, slow for them" reports.

macOS (Docker Desktop): The build cache and cache mounts live inside the VirtioFS-backed VM, so prune with docker builder prune, never a host rm. A large build context is uploaded across the host-to-VM boundary before the build even starts, so a tight .dockerignore matters more here than on Linux; a bloated context can add seconds of upload before BuildKit runs a single step. Windows / WSL2: Keep the repository and any type=local cache directory on the Linux filesystem (~/code, not /mnt/c). A context or cache path under /mnt/c is proxied through the 9p bridge, which throttles the context read and cache writes badly enough to erase the layer savings. Run all build commands inside the WSL2 distro, not from PowerShell against a Windows-path project. Apple Silicon (ARM64): Cache keys embed the target platform, so a cache built for linux/amd64 will not satisfy an arm64 build and vice versa. Keep one cache per platform, or build a single platform consistently. Pin --platform linux/amd64 only for base images that lack an arm64 manifest, since emulation both slows the build and produces a cache that native builds cannot reuse.

Rollback and recovery

Every change in this guide is reversible, and because the cache is regenerable state, the safe recovery for any suspected corruption is to discard it and rebuild cold once. Restore the Dockerfile and Compose file from version control, clear the build cache, and force a clean rebuild to confirm the baseline still produces a working image:

#!/usr/bin/env bash
set -euo pipefail
# Revert config changes
git checkout HEAD -- Dockerfile docker-compose.yml .dockerignore
# Discard any local exportable cache
rm -rf /tmp/.buildx-cache
# Clear BuildKit's internal cache store
docker builder prune --all --force
# Prove the baseline rebuilds from nothing
docker compose build --no-cache app
docker compose up -d --wait app

If a multi-stage refactor produced a broken runtime image, roll back to a single-stage build temporarily by pointing Compose at the last known-good stage with target: or removing the target key entirely, then rebuild. Because the Dockerfile and Compose file are versioned, a git revert of the offending commit restores the previous build behaviour exactly, and the --no-cache rebuild above guarantees no stale layer survives the revert.

Frequently Asked Questions

Related

• Fix slow local rebuilds in a Compose service
• Coordinate multi-service startup and teardown
• Restore hot-reload after edits stop propagating
• Align local and CI build caches for parity