Docker container unhealthy: how to read and fix a failing health check

A container marked (unhealthy) is running, but its HEALTHCHECK keeps failing. How to see the check's output, the five mistakes behind most failing checks, and how to write one that tells the truth.

· 5 min read · By the Spot Downtime team

docker ps shows Up 2 hours (unhealthy). The process is running, but the health check defined for the container has failed several times in a row (three by default). Sometimes the app really is broken: it's up but not answering. Just as often the check itself is wrong. The check's own output tells you which.

See what the check said

bash
docker inspect --format '{{json .State.Health}}' myapp | jq

You get the status, the number of failures in a row, and the last five results, each with its exit code and output. Then run the same command yourself, inside the container:

bash
docker inspect --format '{{json .Config.Healthcheck.Test}}' myapp
# ["CMD-SHELL","curl -f http://localhost:8080/health || exit 1"]

docker exec myapp sh -c 'curl -f http://localhost:8080/health; echo "exit=$?"'

Why health checks fail

1. The tool isn't in the image

The most common one. curl and wget aren't in slim, Alpine-based or distroless images, so the check fails with curl: not found (exit 127) while the app is fine. Install it in the image, use what's there (wget -q --spider on BusyBox), or write the check in the app's own language:

dockerfile
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
  CMD node -e "fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"

2. Wrong host or port

The check runs inside the container, so it must use the port the app listens on there, not the published host port. And localhost can resolve to the IPv6 address ::1 while the app only listens on IPv4: use 127.0.0.1.

3. No time to start

Apps that load data or run migrations at startup fail their first checks. --start-period gives them time: failures during it don't count. Set it a little longer than your slowest start.

4. A timeout that's too short

The default timeout is 30 seconds, but many examples set --timeout=3s. A health endpoint that queries the database can take longer under load, so the container turns unhealthy exactly when it's busy. Keep health endpoints cheap and give them room.

5. The endpoint checks too much

A health check that fails when a third-party API is slow marks your container unhealthy for someone else's problem, and with an orchestrator gets it restarted for nothing. Check what the container needs to do its job (it answers, it reaches its own database), not everything downstream.

Unhealthy doesn't mean restarted

Plain Docker and Compose only mark the container unhealthy and keep it running. Swarm replaces it; Kubernetes uses its own probes, not HEALTHCHECK. If you expected a restart, nothing will happen until someone acts, which is a good reason to be alerted.

When the app really is unhealthy

  • docker logs --tail 200 myapp around the time the check started failing.
  • docker stats --no-stream myapp: is it at its memory limit or pinned on CPU?
  • Thread or connection pools exhausted: the process is up but every worker is stuck. A restart buys time; the logs show why.

Be alerted when a container turns unhealthy

With the Spot Downtime agent installed with --docker, the server alerts as soon as a container is failing its health check, and again when it's healthy. A Docker container monitor goes down for it too, so it counts against uptime and can page you by text or call. Unhealthy containers are listed first in the server's Containers table. See the server agent to set it up, and containers that keep restarting if an orchestrator is restarting it.

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading