How to configure gRPC health checking
Learn how to implement and configure gRPC health checking for reliable service discovery and load balancing: server status, Kubernetes probes, client-side health checks, Envoy, cloud load balancers and Consul.
· 7 min read · By the Spot Downtime team
gRPC has one health protocol, grpc.health.v1.Health, and almost everything that routes traffic can use it: Kubernetes, Envoy, cloud load balancers, service registries and gRPC clients themselves. The details of each one decide whether a sick server actually stops getting traffic. This guide walks through configuring each piece, starting with the server. If the health service itself is new to you, start with how gRPC health checks work.
1. Report a status that means something
Registering the health service is the easy part. What makes it useful is when you change the status. In Go, the server from google.golang.org/grpc/health keeps a status per service name:
hs := health.NewServer()
healthpb.RegisterHealthServer(s, hs)
// Not ready until dependencies are up.
hs.SetServingStatus("readiness", healthpb.HealthCheckResponse_NOT_SERVING)
go func() {
waitForDatabase()
hs.SetServingStatus("readiness", healthpb.HealthCheckResponse_SERVING)
}()
// On SIGTERM: stop taking new traffic first, then stop the server.
hs.Shutdown() // every service now reports NOT_SERVING
time.Sleep(10 * time.Second) // let load balancers notice
s.GracefulStop()A few rules keep the status honest:
- Keep liveness and readiness apart. Liveness answers “should this process be restarted?” and should only fail if the process is stuck. Readiness answers “should it get traffic?” and can depend on the database. Mixing them means a database outage restarts every pod, which makes things worse.
- Name services after what callers need. Use real service names (
shop.v1.Cart) when one server hosts several and they can fail separately, or role names likereadinessfor probes. The empty name is the server as a whole. - Don't flip on every blip. Set NOT_SERVING after a dependency has failed for a few seconds, not on the first error, or traffic will bounce between servers.
2. Kubernetes probes
Kubernetes calls the health service directly (built in since 1.24, stable since 1.27), so you don't need to add grpc_health_probe to the image:
containers:
- name: cart
ports:
- containerPort: 50051
startupProbe:
grpc: { port: 50051 }
failureThreshold: 30
periodSeconds: 2
livenessProbe:
grpc: { port: 50051 } # "" = the process as a whole
periodSeconds: 10
readinessProbe:
grpc: { port: 50051, service: readiness }
periodSeconds: 5
failureThreshold: 2The startup probe gives a slow-starting server time before liveness can restart it. The readiness probe takes the pod out of the Service's endpoints while it reports NOT_SERVING, without restarting it.
The kubelet's probe has no TLS or auth
/grpc.health.v1.Health/ from your auth interceptor, or run grpc_health_probe -tls as an exec probe instead.3. Client-side health checking
Kubernetes Services balance connections, not calls, and gRPC keeps one connection open for a long time. That's why many gRPC setups balance in the client instead: the client resolves every backend address (with a headless Service, for example) and spreads calls over them. gRPC clients can then watch each backend's health and skip the unhealthy ones. Turn it on in the service config:
import _ "google.golang.org/grpc/health" // registers the client-side health checker
conn, err := grpc.NewClient("dns:///cart.shop.svc.cluster.local:50051",
grpc.WithTransportCredentials(creds),
grpc.WithDefaultServiceConfig(`{
"loadBalancingConfig": [{"round_robin": {}}],
"healthCheckConfig": {"serviceName": "readiness"}
}`),
)The client opens a Watch stream to each backend and stops sending calls to any that report NOT_SERVING, without waiting for DNS or Kubernetes to catch up. It only applies to load-balancing policies that use it, like round_robin; the default pick_first ignores it. Java and C++ clients read the same healthCheckConfig.
4. Envoy and service meshes
Envoy can actively check each upstream host with the gRPC health protocol. The cluster must speak HTTP/2 to the upstream:
clusters:
- name: cart
type: STRICT_DNS
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config: { http2_protocol_options: {} }
health_checks:
- timeout: 1s
interval: 5s
unhealthy_threshold: 2
healthy_threshold: 2
grpc_health_check:
service_name: readiness
load_assignment:
cluster_name: cart
endpoints:
- lb_endpoints:
- endpoint: { address: { socket_address: { address: cart, port_value: 50051 } } }Only SERVING counts as healthy. In a mesh like Istio, the sidecar handles routing, but Kubernetes readiness still decides which pods are endpoints, so the probes in step 2 matter just as much.
5. Cloud load balancers
Google Cloud
Backend services support gRPC health checks natively, including a service name:
gcloud compute health-checks create grpc cart-health \
--port=50051 --grpc-service-name=readinessAWS Application Load Balancer
Create the target group with protocol version gRPC, set the health check path to /grpc.health.v1.Health/Check, and set the success codes to 0 (the gRPC OK code). The default path, /AWS.ALB/healthcheck, isn't implemented by your server, and the default success code, 12 (UNIMPLEMENTED), only proves that something gRPC-shaped answered.
ALB only reads the status code
UNAVAILABLE instead.6. Service registries
Consul checks gRPC services with the same protocol. In a service definition:
{
"service": {
"name": "cart",
"port": 50051,
"check": {
"grpc": "127.0.0.1:50051/readiness",
"grpc_use_tls": true,
"interval": "10s",
"timeout": "2s"
}
}
}Leave off the /readiness part to check the whole server. Consul only hands out healthy instances to callers that discover the service through it.
7. Check it from outside too
Everything above checks servers from inside your network. None of it tells you whether customers can reach the service: DNS, the public load balancer, the certificate and the network path all sit in front. Add one external check on the public endpoint with the same protocol. In Spot Downtime that's a gRPC monitor: enter the public host and port, optionally the service name and an authorization header as metadata, and it alerts your team when the answer isn't SERVING. It tracks the certificate's expiry from the same call.
Checklist
- The health service is registered, and readiness reflects real dependencies after a short grace period.
- Liveness only depends on the process itself.
- On shutdown, the server reports NOT_SERVING and waits before stopping.
- Kubernetes has startup, liveness and readiness probes, with a plaintext path for the kubelet if needed.
- Clients that balance calls themselves have
healthCheckConfigwithround_robin. - Load balancers use
/grpc.health.v1.Health/Checkwith success code 0, not their defaults. - One outside check watches the public endpoint.
Get new guides by email
Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.
Keep reading
- How-to · APIs · MonitoringHow to monitor an LLM API (OpenAI, Anthropic, Gemini) in Spot DowntimeA step-by-step guide to LLM API monitoring: use a separate key, pick the model, set the prompt and time limit, and keep the cost of checks to a fraction of a cent.November 2, 2026 · 5 min read
- How-to · APIs · MonitoringHow to monitor an API that needs a login, with multi-step checksChain requests in one monitor: log in, save the access token, and use it to call the endpoints that matter, with JSON assertions on every step.October 28, 2026 · 5 min read