Kubernetes liveness and readiness probes vs external uptime monitoring

Probes keep pods healthy inside the cluster. They can't tell you users can't reach the site. How liveness, readiness and startup probes work, and what to add outside.

· 7 min read · By the Spot Downtime team

A common mistake on Kubernetes is to treat probes as monitoring. Liveness and readiness probes are control signals for the kubelet and the Service: they restart containers and remove pods from load balancing. They run from inside the cluster, against a single pod, and nobody is notified when they fail. A site can have green pods and still be unreachable.

This post explains what each probe does, how to configure them without causing outages of their own, and what an external check covers that probes cannot.

What each probe does

  • Liveness asks "is this container stuck?" On failure, the kubelet kills and restarts the container.
  • Readiness asks "can this pod take traffic right now?" On failure the pod is removed from the Service endpoints. It is not restarted, and it comes back when the probe passes again.
  • Startup asks "has the app finished booting?" While it runs, liveness and readiness probes are held off. Use it for slow-starting apps instead of a huge liveness delay.

A sensible configuration

yaml
containers:
  - name: web
    image: example/web:1.4.2
    ports:
      - containerPort: 8080
    startupProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 5
      failureThreshold: 30        # up to 150s to start
    livenessProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet: { path: /ready, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2

An HTTP probe succeeds on any status from 200 up to but not including 400. Anything else, or a timeout, counts as a failure. /healthz should check only that the process is working. /ready may check that the pod can do its job.

Don't put dependencies in the liveness probe

If the liveness endpoint checks the database and the database has a hiccup, every pod fails liveness at once and the kubelet restarts all of them. A shared dependency problem becomes a full restart loop, which is much worse than the original blip. Keep liveness local and cheap. Dependency checks, if you use them at all, belong in readiness, and even there they can remove every pod from the Service together.

Common probe mistakes

  • Liveness without a startup probe on a slow app: the container is killed before it finishes booting and ends up in CrashLoopBackOff.
  • Timeouts too tight. A timeoutSeconds of 1 under CPU throttling produces false failures.
  • Wrong port or path. The probe gets a 404 or connection refused and the pod never becomes ready. Check with kubectl describe pod, which shows probe failures in Events.
  • Same endpoint for both probes with dependency checks, so a transient failure restarts pods instead of just pausing traffic.
bash
kubectl describe pod web-7d9c6b5f4-abcde   # look at Events for probe failures
kubectl get pods -w                         # restarts and READY column
kubectl get endpoints web                   # which pod IPs receive traffic
kubectl logs web-7d9c6b5f4-abcde --previous # logs from the container that was restarted

What probes cannot see

Everything between the user and the pod is outside what a probe measures:

  • DNS records pointing at the wrong place or not resolving.
  • An expired or mismatched TLS certificate on the ingress or load balancer.
  • The ingress controller, cloud load balancer or CDN being down or misconfigured.
  • All pods unready: the cluster looks "working" while the Service has zero endpoints and users get 503s.
  • A bad deploy that returns 200 on /healthz but errors on real pages or API calls.
  • The whole cluster or region being unavailable, which also takes down anything inside it that would alert you.

Probes also don't alert anyone. A pod restarting every few minutes is only visible if someone looks at kubectl get pods.

What to add outside the cluster

A useful split: probes decide whether a pod should get traffic or be restarted, and external monitoring decides whether a human should be woken up.

Make the external check meaningful

Don't point the external monitor at /healthz, which is designed to be shallow. Point it at a page or endpoint that exercises the path users depend on, and give it a response-time threshold. For ideas on what to check, see the API monitoring guide.

Rule of thumb

If a failure would not trigger a probe but would make a customer unable to use the product, it needs an external check. If it would trigger a probe but needs a human, it needs an alert route as well, not just the restart.

Check the public side right now

Is it down?Check from outside your network whether the site answers, independent of what your cluster thinks of itself.

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading