Keeping Onboarding Docs in Sync with CI Checks

A new hire follows your README, types make setup, and gets make: *** No rule to make target 'setup' — because six weeks ago someone renamed the target to bootstrap and never touched the docs.

This page shows how to make that failure impossible to merge: a CI check that extracts every command the README tells a human to run and fails the build when any of them no longer exists as a real make target or a script on disk. It is the enforcement layer under README-driven automation, where the working assumption is that every documented step maps to something a machine actually runs. Documentation that is merely written down rots silently; documentation that CI executes against the tooling cannot. If you have not yet built the targets the README quotes, start with writing a make bootstrap target for one-command setup and come back here to guard them.

Diagnostic

Drift is invisible until someone trips over it, so the first job is a command that surfaces it on demand. The README below documents four commands; the Makefile has since diverged. Reproduce the mismatch by extracting both sets and diffing them.

#!/usr/bin/env bash
set -euo pipefail

# Commands the README tells a human to run.
grep -hoE 'make [a-z][a-z0-9_-]*' README.md \
  | awk '{print $2}' | sort -u > /tmp/documented.txt

# Targets the Makefile actually defines.
grep -oE '^[a-zA-Z0-9_-]+:' Makefile \
  | tr -d ':' | sort -u > /tmp/actual.txt

echo "== documented but missing from Makefile =="
comm -23 /tmp/documented.txt /tmp/actual.txt

On a drifted repository this prints the offending targets — the ones the README promises but the Makefile can no longer deliver:

== documented but missing from Makefile ==
migrate
setup

The README still says make setup and make migrate; the Makefile now spells them bootstrap and db-migrate. Nothing failed at commit time, nothing failed in the test suite, and the two files will keep drifting apart until a human hits the wall. That is exactly the class of failure to move left into CI.

Root cause

The command list exists in two places — prose the human reads and a Makefile the machine runs — and nothing binds them. A rename, a deleted helper script, or a new target added without a matching doc line moves one copy and leaves the other stale. Git does not object because both files are syntactically valid; the test suite does not object because it invokes targets by their real names, not by the names printed in the README. The drift is a cross-file invariant — "every command the docs mention must resolve to real tooling" — and no tool enforces cross-file invariants unless you write one. The fix is not better discipline; it is to encode that invariant as a script and run it on every pull request, so the machine checks the one thing humans reliably forget.

It is worth being precise about why discipline alone fails here. The author who renames a target is, at that moment, thinking about the code change, not the documentation two directories away; the reviewer reads the diff of the Makefile and the diff of the source, but the README is unchanged and therefore invisible in the review. Nobody is negligent — the information needed to catch the drift is simply not in front of anyone at the moment the drift is created. A mechanical check inverts that: it re-derives the invariant from the current state of both files on every run, so it does not depend on any human happening to hold both halves in their head at the same time.

Resolution

Build one check script that reads the documented commands out of the README, resolves each against the real Makefile targets and the scripts/ directory, and exits non-zero on any command that no longer resolves. Wire it into a make target so it is runnable locally, then run that target in CI.

1. Extract documented make commands and script paths from the README.
2. Enumerate the real targets from the Makefile and the real scripts on disk.
3. Diff the two sets in both directions and fail on any documented command with no backing tooling.

Save the following as scripts/check-doc-drift.sh and mark it executable with chmod +x scripts/check-doc-drift.sh.

#!/usr/bin/env bash
set -euo pipefail

README="${1:-README.md}"
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT

# 1. What the docs claim you can run.
grep -hoE 'make [a-z][a-z0-9_-]*' "$README" \
  | awk '{print $2}' | sort -u > "$tmp/doc_targets.txt"
grep -hoE '(\./)?scripts/[a-zA-Z0-9_./-]+\.sh' "$README" \
  | sed 's#^\./##' | sort -u > "$tmp/doc_scripts.txt"

# 2. What actually exists.
grep -oE '^[a-zA-Z0-9_-]+:' Makefile \
  | tr -d ':' | sort -u > "$tmp/real_targets.txt"
find scripts -type f -name '*.sh' 2>/dev/null \
  | sort -u > "$tmp/real_scripts.txt" || true

status=0

# 3a. Documented targets that no longer exist.
missing_targets="$(comm -23 "$tmp/doc_targets.txt" "$tmp/real_targets.txt")"
if [ -n "$missing_targets" ]; then
  echo "ERROR: README documents make targets that do not exist:"
  echo "$missing_targets" | sed 's/^/  make /'
  status=1
fi

# 3b. Documented scripts that are not on disk.
missing_scripts="$(comm -23 "$tmp/doc_scripts.txt" "$tmp/real_scripts.txt")"
if [ -n "$missing_scripts" ]; then
  echo "ERROR: README references scripts that do not exist:"
  echo "$missing_scripts" | sed 's/^/  /'
  status=1
fi

if [ "$status" -eq 0 ]; then
  echo "doc-drift: README commands match the Makefile and scripts/ directory."
fi
exit "$status"

Two decisions in this script matter. First, it resolves make targets against the Makefile itself rather than against make help, so a target that exists but lacks a ## help annotation still counts as real — this check is about existence, not documentation quality, and conflating the two produces confusing failures. Second, the script paths are normalized by stripping a leading ./, because a README writes ./scripts/doctor.sh while find prints scripts/doctor.sh; without that sed, every referenced script reports as missing and the check cries wolf.

Now expose it as a target so nobody has to remember the path:

.PHONY: check-docs
check-docs: ## Fail if README commands drift from Makefile targets or scripts
	@./scripts/check-doc-drift.sh README.md

Expected output

A clean run — README, Makefile, and scripts/ all agreeing — prints one line and exits zero, so the CI step goes green:

$ make check-docs
doc-drift: README commands match the Makefile and scripts/ directory.

A drifted run names every broken command and exits non-zero, which is what actually blocks the merge:

$ make check-docs
ERROR: README documents make targets that do not exist:
  make setup
  make migrate
ERROR: README references scripts that do not exist:
  scripts/reset-db.sh
make: *** [Makefile:14: check-docs] Error 1

The message points straight at the fix: either rename the target back, update the README to the new name, or restore the deleted script. Because the failure carries the exact command strings, the author does not have to reverse-engineer what CI is unhappy about — they read the two lists and reconcile them in a one-line edit.

Prevention

Run the check on every pull request so drift is caught on the change that introduced it, while the author still has the context to fix it. The workflow below checks out the branch and runs the same make check-docs a developer runs locally — no special CI-only logic, so a green local run predicts a green pipeline.

# .github/workflows/doc-drift.yml
name: Docs In Sync
on:
  pull_request:
    paths:
      - "README.md"
      - "Makefile"
      - "scripts/**"
jobs:
  doc-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Check README against tooling
        run: make check-docs

Scoping the trigger with paths keeps the job from running on pull requests that touch neither the docs nor the tooling, which trims noise without weakening the guard — the only way to introduce drift is to edit one of those three paths, and every one of them is listed. For a faster local feedback loop, run the same target from a pre-commit hook so an author who renames a target is told before the commit even lands:

#!/usr/bin/env bash
set -euo pipefail
# .git/hooks/pre-commit  (or a pre-commit framework entry)
if git diff --cached --name-only | grep -qE '^(README\.md|Makefile|scripts/)'; then
  make check-docs
fi

The CI job is the authority and the hook is the courtesy: hooks can be skipped with --no-verify, so the merge gate must live in CI where it cannot be bypassed. Together they close both ends — the hook catches the honest mistake early, and CI catches the one that slips past.

This check pairs naturally with the onboarding health-check script: doctor proves the documented environment still builds, while check-docs proves the documented commands still exist. Run both in the same pull-request pipeline and the README is held honest from two directions — the words it prints resolve to real targets, and those targets still produce a working stack.

Platform caveats

The check runs the same everywhere, but three of the tools it leans on differ by platform. Pin the behaviour so the gate does not pass on a developer laptop and fail on the CI runner, or vice versa.

macOS (Docker Desktop): stock macOS ships BSD grep, whose -E handling is fine here but whose -o prints matches slightly differently on multi-match lines; if you extend the patterns, test with ggrep from brew install grep to match the GNU behaviour the Ubuntu CI runner uses. WSL2: keep the repository on the Linux filesystem (~/code, not /mnt/c). A README saved from a Windows editor can carry \r\n line endings, and the trailing carriage return makes make setup\r never match the real setup target; run sed -i 's/\r$//' README.md or set .gitattributes to * text=auto so the check compares clean strings. Apple Silicon (ARM64): the check itself is architecture-neutral, but if scripts/ includes compiled helpers, find will list them while the README may document an amd64-only path; keep documented scripts as portable shell so the two sides resolve identically on every runner.

Rollback

The check adds one script, one target, and one workflow file, and mutates nothing else, so backing it out is a clean delete:

#!/usr/bin/env bash
set -euo pipefail
rm -f scripts/check-doc-drift.sh .github/workflows/doc-drift.yml
# remove the check-docs target block from the Makefile by hand, or:
git checkout -- Makefile
echo "Drift guard removed. README is no longer enforced against the tooling."

Because the guard is side-effect-free — it only reads files and returns an exit code — removing it cannot leave the repository in a broken state; the worst case is that documentation drift becomes silent again. If a single false positive is blocking an urgent merge, prefer narrowing the grep patterns over deleting the guard, so the invariant stays enforced for every command it can still resolve.

Frequently Asked Questions

Related

• Write a make bootstrap target for one-command setup
• Build a doctor health-check script for onboarding
• Turn onboarding docs into runnable automation