Fixing Time-to-First-PR Regressions After a Dependency Upgrade

New hires used to be productive in twenty minutes; after a dependency or toolchain upgrade, first-run setup now takes an hour — a measurable time-to-first-PR regression you can attribute to a specific commit. This guide is deliberately narrow: it treats cold setup time as a number, bisects that number back to the commit that moved it, and turns the fix into a gate so the same regression cannot silently return on the next upgrade.

Diagnostic

The setup-time regression is invisible on the machine that authored the upgrade, because that machine already has a warm dependency cache, a populated Docker layer cache, and compiled native modules sitting in node_modules. The only reliable measurement is a cold bootstrap that reproduces the new-hire path: a pristine checkout with every cache purged. Time it and compare against the recorded pre-upgrade baseline.

#!/usr/bin/env bash
set -euo pipefail
# cold-bootstrap-timer.sh — measure setup from a pristine state.
docker compose down -v >/dev/null 2>&1 || true
docker builder prune -af >/dev/null 2>&1 || true
rm -rf node_modules
start=$(date +%s)
make bootstrap >/dev/null
end=$(date +%s)
echo "cold_bootstrap_seconds=$((end - start))"

Expected BAD output — the cold setup time has roughly tripled versus the recorded baseline:

cold_bootstrap_seconds=3120
# baseline (pre-upgrade): cold_bootstrap_seconds=1080

A single wall-clock number tells you a regression exists but not where the time went. Before bisecting, break the bootstrap into phases so you know which stage to blame — an image pull, a dependency resolve, a native compile, or a database migration each fails differently and demands a different fix. Wrap each phase in the same date arithmetic and print a per-phase line.

#!/usr/bin/env bash
set -euo pipefail
# phase-timer.sh — attribute cold setup time to each stage.
phase() { local label=$1; shift; local s=$(date +%s); "$@" >/dev/null 2>&1; \
  echo "${label}_seconds=$(( $(date +%s) - s ))"; }
docker compose down -v >/dev/null 2>&1 || true
rm -rf node_modules
phase image_pull   docker compose pull
phase deps_install npm ci
phase image_build  docker compose build
phase db_migrate   make migrate

If deps_install_seconds or image_build_seconds is the line that ballooned, the cause is almost always a native module compiling from source or a cache key that stopped matching — the two failure modes this page resolves.

The diagnostic is only meaningful against a recorded baseline, so treat the pre-upgrade number as an artifact you keep. Store it next to the timer script — a one-line baseline.txt committed to the repo, or a value in the same telemetry system that already tracks time-to-first-PR — and re-measure it deliberately whenever you intentionally add setup work. Without a committed baseline you are comparing today's cold run against a half-remembered "it used to be fast," which is not a number you can bisect against or gate on. Run the timer three times and take the median: a single cold run picks up transient network variance during the image pull that can swing the total by tens of seconds and blur the signal you are trying to isolate.

Root cause

A dependency or toolchain upgrade slows onboarding when it changes how the dependency graph resolves or how caches are keyed. Common triggers: a major version bump pulls a transitive dependency that now compiles a native module from source (minutes of node-gyp/cffi build per cold install); a lockfile churn invalidates every entry so the package manager re-resolves the whole tree instead of replaying the lock; or a base-image/tool bump changes a cache key (hashFiles('**/package-lock.json'), a .tool-versions line, a Dockerfile layer) so every Docker layer and CI cache misses and rebuilds. The regression is invisible on a warm machine — the author's caches are already populated — and only appears on a cold clone, which is exactly the new-hire path. Bisecting cold-bootstrap time across the suspect commit range pinpoints the offending change.

The three triggers demand different fixes, so classify before you patch. A native-compile regression shows up as deps_install_seconds climbing and node-gyp / cc / rustc processes running during install; the fix is to pin a version that ships a prebuilt binary for your architectures. A lockfile-churn regression shows up when the lockfile changed but resolution is still non-deterministic — often because an install ran with a mutating command (npm install) instead of npm ci, leaving carets (^1.2.0) to float; the fix is an authoritative, committed lockfile. A cache-key regression shows up as a warm CI run that behaves like a cold one — the cache key embeds a value the upgrade changed (a tool version, a Dockerfile hash), so every restore misses. Docker layer caching is positional and content-addressed: a COPY package*.json layer only reuses its cached result while the copied files and every preceding instruction are byte-identical, so bumping a base image tag near the top of the Dockerfile invalidates every layer below it, dependency install included. Reading the diff of the suspect commit against these three shapes usually tells you which you have before bisect even finishes.

Resolution

1. Bisect the commit range with the cold-bootstrap timer as the test.
2. Inspect what the bad commit changed in the lockfile or cache key.
3. Restore deterministic resolution (commit the lockfile, pin the native dep to a prebuilt) and stable cache keys.
4. Re-time a cold bootstrap to confirm the regression is gone.

Use git bisect run with the timer wrapped to fail above a threshold. git bisect performs a binary search across the commit range, so it converges on the first offending commit in log2(N) builds rather than testing every revision — for a 200-commit range that is roughly eight cold bootstraps instead of two hundred.

#!/usr/bin/env bash
set -euo pipefail
# bisect-test.sh — exit non-zero when cold setup exceeds the SLA (seconds).
SLA=1500
secs=$(./cold-bootstrap-timer.sh | sed 's/cold_bootstrap_seconds=//')
[ "$secs" -le "$SLA" ]

#!/usr/bin/env bash
set -euo pipefail
git bisect start HEAD <last-good-tag>
git bisect run ./bisect-test.sh   # prints the first commit that blew the SLA
git bisect reset

Before pinning, confirm the target version actually publishes a prebuilt binary for your platforms — do not assume the changelog. For a native npm package you can check that the install skips compilation with npm install <pkg>@<version> --no-save --foreground-scripts and watching that no node-gyp rebuild line appears; for a Python wheel, pip download --only-binary=:all: <pkg>==<version> fails loudly if no wheel exists for your interpreter and architecture. A pin to a version that still compiles from source moves the regression, it does not fix it.

Once found, restore deterministic install and a prebuilt binary so cold installs stop compiling. Pinning to an exact version that publishes prebuilt binaries for both arm64 and x86_64 eliminates the node-gyp step entirely, and the overrides block prevents a transitive dependency from dragging in a build-from-source toolchain again.

// package.json — pin a version with a prebuilt arm64/x86_64 binary and keep the lock authoritative
{
  "dependencies": {
    "sharp": "0.33.4"
  },
  "overrides": {
    "node-gyp": "10.1.0"
  }
}

After editing the manifest, regenerate the lockfile with a deterministic install and commit it in the same change, so every future clone replays an identical tree instead of re-resolving:

#!/usr/bin/env bash
set -euo pipefail
npm ci                 # replay the lock exactly; fails if package.json and lock disagree
git add package.json package-lock.json
git commit -m "fix(deps): pin sharp to prebuilt 0.33.4 to restore cold-setup SLA"

Stabilize the Docker/CI cache key so a tool bump no longer busts every layer. Key the cache on the lockfile hash only — not on a tool version or a timestamp — and supply restore-keys so a partial match still seeds most of the cache on a near-miss:

# .github/workflows/ci.yml — key on the lockfile only, restore on partial match
steps:
  - uses: actions/cache@v4
    with:
      path: |
        ~/.npm
        node_modules
      key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}
      restore-keys: |
        ${{ runner.os }}-node-

Expected output

A cold bootstrap after the fix returns to (or below) the baseline:

$ ./cold-bootstrap-timer.sh
cold_bootstrap_seconds=1015

The bisect run names the exact regressing commit, making the cause auditable:

$ git bisect run ./bisect-test.sh
4f2a9c1 is the first bad commit
    chore: bump image-lib to 5.x (drops prebuilt binaries)

Plotting the three measured points — the pre-upgrade baseline, the regressed peak, and the post-fix run — makes the recovery legible in a code review and gives you the numbers to set an SLA threshold with margin.

Prevention

1. Add a CI job that runs cold-bootstrap-timer.sh and fails when setup exceeds an SLA, so a future upgrade that slows onboarding is blocked at PR time.
2. Commit lockfiles and prefer dependencies that ship prebuilt binaries for both architectures — coordinate with debugging works-on-my-machine runtime drift.
3. Watch the dependency graph for newly introduced heavy transitive deps with mapping microservice dependencies for local dev.

The gate itself is a few lines. Run the same timer that found the regression, on a clean runner, on every pull request. Set the SLA above the current baseline with margin (here 1500s against a 1080s baseline) so normal variance does not flap the check, but tightly enough that a tripling of setup time trips it.

# .github/workflows/onboarding-sla.yml
name: Onboarding Time Gate
on: [pull_request]
jobs:
  cold-setup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Time a cold bootstrap
        run: |
          chmod +x ./cold-bootstrap-timer.sh ./bisect-test.sh
          ./bisect-test.sh

Because a CI runner is warm-cache friendly by design, force the gate to measure a genuinely cold path — skip the actions/cache restore for this one job, or run it in a container without the mounted cache — otherwise it reports the author's happy path and never catches the new-hire regression it exists to prevent. The distinction between a gated and an ungated upgrade flow is exactly the difference between catching the regression at review and shipping it to every future clone.

Platform caveats

macOS (Docker Desktop): cold timings include VirtioFS sync overhead; record the baseline on the same OS you compare against, never mix host platforms. WSL2: measure on the Linux filesystem — node_modules extraction over /mnt/c inflates cold setup several-fold and pollutes the bisect signal. Apple Silicon (ARM64): a dep that dropped its arm64 prebuilt forces compilation locally but not on x86_64 CI; bisect on the developer architecture, not the runner's, to see the real regression.

Rollback

If the pin or cache-key change causes its own problem, revert the upgrade commit and restore the prior lockfile in one step. Because the lockfile was committed alongside package.json, npm ci after the revert reproduces the exact tree that was fast before:

#!/usr/bin/env bash
set -euo pipefail
git revert <bad-commit> && npm ci   # revert the upgrade and restore the prior lockfile state

Frequently Asked Questions

Related

• Measure onboarding time across distributed teams
• Debug works-on-my-machine runtime drift
• Detect circular dependencies in local builds
• Optimize Docker Compose for fast local rebuilds