Resolving Service Startup Order and Healthcheck Races

Your API container crashes on boot with a connection-refused error against the database, even though depends_on lists db — because plain depends_on waits for the container to start, not for the application inside it to be ready. This is the classic startup race that surfaces once a stack grows past a single service, and it belongs to the broader problem of sequencing boot order covered in Multi-Service Orchestration with Compose. The tool at fault is not Docker Compose itself but the assumption that "started" means "accepting connections." It is fixed by gating dependents on a healthcheck rather than on container creation.

The distinction matters because the failure is non-deterministic. On a warm machine with a cached image the database may bind its socket in under a second and the API connects cleanly, so the stack looks correct. On a cold CI runner, an underprovisioned laptop, or the first up after a volume wipe, initialisation stretches to ten or twenty seconds and the same compose file crash-loops. A race that only fails sometimes is worse than one that always fails, because it survives review and reappears in front of a new hire on their first onboarding run.

Diagnostic

The dependent service exits early or restart-loops while the dependency is still initializing:

#!/usr/bin/env bash
# trace the boot order
set -euo pipefail
docker compose up -d
docker compose logs api | head -n 20
docker compose ps --format '{{.Name}}\t{{.Status}}'

# BAD: api tried to connect before postgres finished initdb
api-1  | Error: connect ECONNREFUSED 172.28.0.3:5432
api-1  | at TCPConnectWrap.afterConnect [as oncomplete]
api-1 exited with code 1
NAME    STATUS
db-1    Up 2 seconds
api-1   Restarting (1) Less than a second ago

The dependency container is Up, but its service has not finished starting — Postgres runs init scripts and a restart cycle before it accepts connections. The tell-tale signature is that db-1 reports Up with a small uptime while api-1 is Restarting: the ordering was honoured (the database started first) yet the dependent still failed, which rules out a missing depends_on and points squarely at readiness timing.

To confirm the race rather than a genuine configuration error, inspect the two events on the same timeline. If the API's first connection attempt timestamp precedes the database's "ready to accept connections" log line, you have a race; if it follows it, the problem is credentials, a wrong host, or a network alias.

#!/usr/bin/env bash
# correlate the two readiness moments
set -euo pipefail
docker compose logs --timestamps db  | grep -i "ready to accept connections" | head -n1
docker compose logs --timestamps api | grep -i "ECONNREFUSED" | head -n1

Root cause

depends_on: [db] only orders container creation and start; it returns as soon as the container process launches. The database daemon then takes several more seconds to run init scripts, bind its socket, and accept connections. During that window the API connects, is refused, and exits. Without a readiness condition, the order is correct but the timing is not — a race.

The reason Compose cannot do better on its own is that it has no application-level knowledge. From the orchestrator's point of view a container is a process tree; it cannot tell whether the process inside is a fully initialised Postgres accepting TCP or a shell script still copying seed data. The only way to give Compose that knowledge is to teach it a readiness signal, and the mechanism for that signal is the container's healthcheck. Once a healthcheck exists, Compose can wait on a service_healthy condition instead of the much weaker service_started, which is all a bare depends_on provides.

There are three readiness conditions Compose understands, and choosing the wrong one is a common second-order bug. service_started is the default and only guarantees the container exists. service_healthy waits for the healthcheck to report healthy and is what long-running dependencies need. service_completed_successfully waits for a container to exit with status zero and is what one-shot jobs — migrations, seed loaders, fixture importers — need. Reaching for service_healthy on a migration container that is designed to run once and exit will hang forever, because a container that has exited can never become healthy.

Resolution

1. Add a real healthcheck to the dependency so Compose can observe readiness, not just liveness.

# docker-compose.yml
services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: app_db
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres -d app_db"]
      interval: 3s
      timeout: 5s
      retries: 5
      start_period: 10s

The test here uses CMD-SHELL so the probe runs through the container's shell and can expand variables and use pipes; the CMD form runs an exec directly with no shell. Crucially, pg_isready checks that Postgres is accepting connections for a specific database, which is exactly the readiness the API cares about — a plain TCP connect to port 5432 would report healthy while initdb is still creating app_db, reintroducing the race one layer down.

2. Gate the dependent on condition: service_healthy so it does not start until the healthcheck passes.

# docker-compose.yml
services:
  api:
    build: .
    depends_on:
      db:
        condition: service_healthy
      migrate:
        condition: service_completed_successfully
    ports:
      - "${APP_PORT:-3000}:3000"

Note the long-form depends_on map: each dependency names its own condition. This is the only form that carries a condition; the short-form list (depends_on: [db, migrate]) silently degrades to service_started and is the single most common cause of the bug reappearing after someone "simplifies" the file.

3. For one-shot setup work (migrations, seeds) use a short-lived service and condition: service_completed_successfully so the API waits for it to exit cleanly.

# docker-compose.yml
services:
  migrate:
    build: .
    command: ["npm", "run", "db:migrate"]
    depends_on:
      db:
        condition: service_healthy
    restart: "no"

Set restart: "no" explicitly on one-shot jobs. If a global restart policy or an inherited restart: always is in play, a migration that exits zero will be restarted, never satisfy service_completed_successfully, and stall the whole graph. The migration itself depends on db being healthy, so the dependency chain reads: database ready → migration runs and exits → API starts.

4. Keep a defensive retry in the application or entrypoint for services that lack a native health endpoint, so a slow start degrades into a brief wait rather than a crash.

#!/usr/bin/env bash
# entrypoint.sh — wait for the API's own dependency, then exec
set -euo pipefail
until nc -z "${DB_HOST:-db}" "${DB_PORT:-5432}"; do
  echo "waiting for ${DB_HOST:-db}:${DB_PORT:-5432}..."
  sleep 1
done
exec "$@"

An entrypoint wait is belt-and-suspenders, not a replacement for service_healthy. It protects against readiness gaps the orchestrator cannot see — for example a dependency in a different compose project, or a managed service outside Docker — and it makes the container resilient if someone runs it with docker run directly, bypassing Compose conditions entirely.

5. Bring the stack up with --wait so up itself blocks until every healthcheck passes or fails.

#!/usr/bin/env bash
set -euo pipefail
docker compose up -d --wait

--wait turns readiness into an exit code, which is what makes it valuable in scripts and CI: the command returns non-zero if any service fails to become healthy within its start_period plus retries budget, so a broken stack fails the pipeline instead of racing on to a flaky integration test.

Expected output

[+] Running 3/3
 ✔ Container db-1       Healthy
 ✔ Container migrate-1  Exited (0)
 ✔ Container api-1      Healthy

docker compose ps --format '{{.Name}}\t{{.Status}}'
# db-1       Up 12 seconds (healthy)
# api-1      Up 4 seconds (healthy)

The order in the summary is meaningful: the database reports Healthy, the migration reports Exited (0), and only then does the API report Healthy. If you instead see the API come up in one or two seconds flat, its healthcheck is probably trivial (a bare exit 0) and is not actually gating anything downstream of it.

Verify the wiring rather than trusting the summary line. docker compose config renders the fully resolved file with all conditions expanded, which is the fastest way to catch a depends_on that quietly dropped back to the short-form list during an edit. Read the rendered condition for each dependency and confirm it matches the lifecycle you intended — service_healthy for daemons, service_completed_successfully for jobs.

#!/usr/bin/env bash
# confirm every depends_on carries an explicit condition
set -euo pipefail
docker compose config | grep -A3 'depends_on:'

Tuning start_period against real cold-start times

The start_period field is a grace window: failing probes during that window do not count toward retries and do not mark the container unhealthy. Set it to the dependency's realistic cold-start time and no larger, because an oversized start_period also delays the moment Compose is willing to declare a genuinely broken service unhealthy. Measure, don't guess — time the dependency from container start to its first successful probe on a cold volume, then add a modest margin.

#!/usr/bin/env bash
# measure real time-to-healthy on a cold volume
set -euo pipefail
docker compose down -v
start=$(date +%s)
docker compose up -d --wait db
end=$(date +%s)
echo "db reached healthy in $((end - start))s"

Prevention

1. Require a healthcheck on every stateful dependency (databases, brokers, caches) and reject depends_on lists that lack a condition.

#!/usr/bin/env bash
# bin/lint-depends.sh — fail if depends_on lacks a condition
set -euo pipefail
if docker compose config | grep -A2 'depends_on:' | grep -qE '^\s+-\s'; then
  echo "Found short-form depends_on without a condition; use condition: service_healthy." >&2
  exit 1
fi
echo "depends_on conditions OK"

2. Tune start_period to the dependency's real cold-start time so early failing probes during init do not count against retries.

3. Run the lint and a cold up --wait in CI, not just locally. The race hides on warm developer machines and only reliably reproduces on a fresh runner with no image or volume cache, so the pipeline is where you want the assertion to live.

#!/usr/bin/env bash
# ci: prove the stack converges from cold
set -euo pipefail
docker compose down -v --remove-orphans
docker compose up -d --wait --wait-timeout 90
docker compose ps
docker compose down -v

Platform caveats

macOS (Docker Desktop): healthcheck probes traverse the Linux VM, adding ~200ms latency; raise timeout slightly so probes do not flap on slower machines. Cold-start times measured on native Linux CI will read a few seconds higher on a laptop under load, so leave headroom in start_period. WSL2: enable systemd in /etc/wsl.conf or pg_isready may fail on a missing socket path and the dependency never reports healthy. Clock skew between the Windows host and the WSL2 VM can also make date +%s measurements inconsistent — measure inside the container if timings look wrong. Apple Silicon (ARM64): pull architecture-matched healthcheck binaries or wrap probes in CMD-SHELL to avoid exec format error. An emulated amd64 image runs its probe under QEMU and can take noticeably longer to go healthy, so a start_period tuned on native arm64 may be too tight.

Rollback

#!/usr/bin/env bash
set -euo pipefail
git checkout -- docker-compose.yml
docker compose down && docker compose up -d --wait

Frequently Asked Questions

Related

• Sequence multi-service boot order with healthchecks
• Speed up the rebuild loop while keeping startup deterministic
• Fix 'port is already allocated' on startup
• Diagnose containers that exit immediately on startup