Debugging Env Variable Leakage in Multi-Stage Docker Builds

A token you passed with --build-arg to install private dependencies shows up in docker history and in the final image's environment, even though you only used it in an earlier build stage. This is part of CI/CD pipeline parity checks under the broader environment sync, secrets and CI parity baseline. The failure is subtle because the build succeeds, the application runs, and nothing in day-to-day usage hints that the credential is still embedded — it only surfaces when someone pulls the image and reads its metadata, by which point the token has been distributed to every registry mirror and every developer laptop that cached the layer.

Multi-stage builds are widely assumed to be a security boundary: "the secret only lives in the deps stage, and I don't ship that stage." That assumption is half true. The intermediate stage's filesystem is discarded, but the metadata recorded for every stage — the ARG declarations, the ENV assignments, and the verbatim text of each RUN command — travels with the layers that any later stage references. This page shows how to reproduce the leak, explains exactly which artifacts retain the value, and walks through converting the build to BuildKit secret mounts so the credential never touches a layer at all.

Diagnostic

Inspect what actually persisted in the published image. Build args and ENV instructions are recorded in image metadata and remain readable by anyone who can pull the image. The two commands below simulate the most common leak: a private-registry token handed to npm during dependency installation.

#!/usr/bin/env bash
set -euo pipefail
docker build --build-arg NPM_TOKEN=npm_secret123 -t app:leaky .

# Build args and ENV show up in the layer history
docker history --no-trunc app:leaky | grep -i "token\|secret\|NPM"

# ENV values leak into the running container's environment
docker run --rm app:leaky env | grep -i "token\|secret"

Expected BAD output:

  ARG NPM_TOKEN=npm_secret123
  /bin/sh -c npm config set //registry.npmjs.org/:_authToken=npm_secret123
NPM_TOKEN=npm_secret123

The token is visible in two places: the build history and the final container environment. For a more thorough audit, unpack the image with docker save and grep the raw layer tarballs and the config.json manifest — this catches values that were written into a file inside a layer, which docker history alone will not show:

#!/usr/bin/env bash
set -euo pipefail
docker save app:leaky -o /tmp/leaky.tar
mkdir -p /tmp/leaky && tar -xf /tmp/leaky.tar -C /tmp/leaky
# Search every layer's config and the image manifest for the secret shape
grep -rEi 'npm_secret|_authToken|token=' /tmp/leaky || echo "clean"
rm -rf /tmp/leaky /tmp/leaky.tar

If the grep prints matches, the secret is recoverable from a plain docker pull — no exploit required. Treat any hit as a confirmed disclosure, not a theoretical one.

One nuance worth checking during the diagnosis: docker history collapses secrets differently depending on how they were introduced. A value set via ARG and then referenced in a later RUN may appear both as an ARG NAME=value line and inside the expanded RUN string, so a single credential can produce several distinct matches. Conversely, a secret written into a file by an intermediate stage and then pulled forward with COPY --from will not appear in history at all — only the docker save layer grep catches it. Run both checks every time; neither is a superset of the other.

Root cause

ARG and ENV values are baked into image layer metadata. A multi-stage build does not automatically scrub them — only the filesystem of intermediate stages is discarded, not the metadata of any stage you COPY --from or build FROM. An ENV set in the final stage persists into every container started from the image, and an ARG is recorded in docker history even if it was "only" used in stage one. Echoing a secret into a RUN command writes it into that layer's command string permanently.

Three distinct mechanisms are at work, and conflating them is why the leak is hard to reason about. First, ARG values are stored as build metadata attached to the layer that follows the declaration; they are not filesystem content, so no amount of deleting files removes them. Second, RUN records the literal shell string it executed, so npm config set ...=$NPM_TOKEN is fine but npm config set ...=npm_secret123 — after the shell expands the variable in a way that ends up in the recorded command — bakes the plaintext in. Third, ENV is the worst offender because it is both metadata and injected into the process environment of every container, which is why docker run ... env prints it. Multi-stage isolation only helps with a fourth category — files written to the intermediate filesystem — and only when you never COPY --from those files forward. Because most credential leaks fall into the first three categories, the stage boundary provides no protection.

Resolution

1. Replace build-time secrets with BuildKit --secret mounts, which expose the value only to a single RUN and never write it to a layer.

# syntax=docker/dockerfile:1.7
FROM node:20-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN="$(cat /run/secrets/npm_token)" \
    npm config set //registry.npmjs.org/:_authToken="$NPM_TOKEN" && \
    npm ci && \
    npm config delete //registry.npmjs.org/:_authToken

FROM node:20-slim AS runtime
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
CMD ["node", "server.js"]

The secret is read from the tmpfs path /run/secrets/npm_token that BuildKit mounts only for the duration of that one RUN. Because the value lives in an in-memory mount rather than the layer filesystem, and because the recorded command string references the mount path — not the value — nothing about the credential is committed to metadata. The npm config delete at the end is defensive: it prevents a stray .npmrc from being written into node_modules and copied forward.

2. Build with the secret supplied from a file or env var, never as a build arg.

#!/usr/bin/env bash
set -euo pipefail
export DOCKER_BUILDKIT=1
printf '%s' "$NPM_TOKEN" | docker build \
  --secret id=npm_token,src=/dev/stdin \
  -t app:clean .

Piping through /dev/stdin avoids writing the token to a temporary file that another process could read or that could be captured in shell history. In CI, prefer --secret id=npm_token,env=NPM_TOKEN (supported by recent BuildKit) so the value is read straight from the masked pipeline variable and never materialises on disk.

3. Audit any remaining ENV lines. If a value is runtime configuration, inject it at docker run/Compose time instead of baking it in. Use ARG only for non-sensitive build inputs. A quick way to enumerate what a Dockerfile bakes is grep -nE '^(ARG|ENV)' Dockerfile — review every hit and ask whether it is a credential; anything that is belongs in a secret mount or in the runtime environment, never in the image.

4. For runtime credentials, pass them through Compose rather than the image. A compose.yaml that reads from the environment keeps the value out of every layer and lets each environment supply its own:

services:
  app:
    image: app:clean
    environment:
      - DATABASE_URL=${DATABASE_URL}
    env_file:
      - .env.runtime

This mirrors the pattern used for keeping local and CI environments aligned — the same variable resolves from .env.runtime on a laptop and from the pipeline's secret store in CI, with no image rebuild required. Keeping the image credential-free also means the same digest promotes cleanly from staging to production: because no environment-specific secret is baked in, the artifact you tested is byte-for-byte the artifact you ship, and only the surrounding runtime configuration changes between stages.

Expected output

$ docker history --no-trunc app:clean | grep -i "token\|secret"
$ docker run --rm app:clean env | grep -i "token\|secret"
$

Both greps return nothing — the secret never entered the image. Re-run the docker save unpack from the Diagnostic section against app:clean and confirm the raw-layer grep also prints clean; that final check proves the value is absent from filesystem content as well as from metadata.

Prevention

1. Add a CI step that fails if any secret-shaped string appears in image metadata:

#!/usr/bin/env bash
set -euo pipefail
if docker history --no-trunc "$IMAGE" | grep -Eiq 'token|secret|password|_key='; then
  echo "FAIL: potential secret in image history"; exit 1
fi

2. Add # syntax=docker/dockerfile:1.7 and a pre-commit grep that rejects ARG .*TOKEN / ENV .*SECRET patterns in Dockerfile. A minimal hook fails the commit before the leak is ever built:

#!/usr/bin/env bash
set -euo pipefail
if git diff --cached --name-only | grep -q 'Dockerfile'; then
  if grep -nEi 'ARG .*(TOKEN|SECRET|PASSWORD)|ENV .*(TOKEN|SECRET|PASSWORD)' Dockerfile; then
    echo "Refusing commit: sensitive ARG/ENV in Dockerfile — use --mount=type=secret"; exit 1
  fi
fi

3. Scan pushed images on a schedule, not just at build time. A leak that predates the CI check still sits in the registry; a nightly job that pulls each published tag and runs the same docker history grep catches regressions and images built outside the pipeline. Pair this with the consolidated parity checks so credential hygiene is verified alongside the rest of your environment invariants rather than as a separate, easily forgotten step.

Platform caveats

macOS (Docker Desktop): ensure BuildKit is the active builder (docker buildx ls); the legacy builder ignores --secret and silently falls back to no secret, so a Dockerfile that expects /run/secrets/npm_token fails with a missing-file error rather than leaking — but a Dockerfile that still reads $NPM_TOKEN from an ARG will build and leak. WSL2: the Docker Desktop WSL integration shares the daemon with Windows; secrets piped through /dev/stdin work, but avoid src= paths that point at a /mnt/c/... Windows drive, where permissions are relaxed and the file may be world-readable while the build runs. Apple Silicon (ARM64): pin --platform linux/amd64 when the published image targets amd64 runners, or the leaked-history check runs against a different architecture's image and can pass locally while the real artifact still leaks.

Rollback

If a leaked image was already pushed, treat the secret as compromised: rotate it, then docker rmi the local copies and delete the pushed tags. Layer metadata cannot be edited after the fact — rebuild clean and re-push. Rotation must come first and cannot be skipped: deleting the tag does not un-distribute layers that mirrors, pull-through caches, or developer machines have already fetched, so the only reliable remediation is to invalidate the credential itself and then remove the artifacts. If the leaked tag was ever used to deploy, assume the credential was scraped and audit the credential's own access logs for unexpected use before and after rotation.

#!/usr/bin/env bash
set -euo pipefail
# 1) rotate the credential at its source (registry, cloud console, secret store)
# 2) remove local copies
docker rmi app:leaky || true
# 3) delete the pushed tag (example for a v2 registry)
curl -fsS -X DELETE "https://$REGISTRY/v2/app/manifests/$DIGEST"

Frequently Asked Questions

Related

• Reproduce CI-only test failures locally with act
• Keep local secrets out of git
• Run one consolidated parity check