How to configure gRPC health checking

Learn how to implement and configure gRPC health checking for reliable service discovery and load balancing: server status, Kubernetes probes, client-side health checks, Envoy, cloud load balancers and Consul.

· 7 min read · By the Spot Downtime team

gRPC has one health protocol, grpc.health.v1.Health, and almost everything that routes traffic can use it: Kubernetes, Envoy, cloud load balancers, service registries and gRPC clients themselves. The details of each one decide whether a sick server actually stops getting traffic. This guide walks through configuring each piece, starting with the server. If the health service itself is new to you, start with how gRPC health checks work.

1. Report a status that means something

Registering the health service is the easy part. What makes it useful is when you change the status. In Go, the server from google.golang.org/grpc/health keeps a status per service name:

hs := health.NewServer()
healthpb.RegisterHealthServer(s, hs)

// Not ready until dependencies are up.
hs.SetServingStatus("readiness", healthpb.HealthCheckResponse_NOT_SERVING)
go func() {
    waitForDatabase()
    hs.SetServingStatus("readiness", healthpb.HealthCheckResponse_SERVING)
}()

// On SIGTERM: stop taking new traffic first, then stop the server.
hs.Shutdown()          // every service now reports NOT_SERVING
time.Sleep(10 * time.Second) // let load balancers notice
s.GracefulStop()

A few rules keep the status honest:

  • Keep liveness and readiness apart. Liveness answers “should this process be restarted?” and should only fail if the process is stuck. Readiness answers “should it get traffic?” and can depend on the database. Mixing them means a database outage restarts every pod, which makes things worse.
  • Name services after what callers need. Use real service names (shop.v1.Cart) when one server hosts several and they can fail separately, or role names like readiness for probes. The empty name is the server as a whole.
  • Don't flip on every blip. Set NOT_SERVING after a dependency has failed for a few seconds, not on the first error, or traffic will bounce between servers.

2. Kubernetes probes

Kubernetes calls the health service directly (built in since 1.24, stable since 1.27), so you don't need to add grpc_health_probe to the image:

containers:
  - name: cart
    ports:
      - containerPort: 50051
    startupProbe:
      grpc: { port: 50051 }
      failureThreshold: 30
      periodSeconds: 2
    livenessProbe:
      grpc: { port: 50051 }          # "" = the process as a whole
      periodSeconds: 10
    readinessProbe:
      grpc: { port: 50051, service: readiness }
      periodSeconds: 5
      failureThreshold: 2

The startup probe gives a slow-starting server time before liveness can restart it. The readiness probe takes the pod out of the Service's endpoints while it reports NOT_SERVING, without restarting it.

The kubelet's probe has no TLS or auth

It connects in plaintext and sends no metadata. If your server only listens with TLS or requires a token on every method, open a separate plaintext port for health, exempt /grpc.health.v1.Health/ from your auth interceptor, or run grpc_health_probe -tls as an exec probe instead.

3. Client-side health checking

Kubernetes Services balance connections, not calls, and gRPC keeps one connection open for a long time. That's why many gRPC setups balance in the client instead: the client resolves every backend address (with a headless Service, for example) and spreads calls over them. gRPC clients can then watch each backend's health and skip the unhealthy ones. Turn it on in the service config:

import _ "google.golang.org/grpc/health" // registers the client-side health checker

conn, err := grpc.NewClient("dns:///cart.shop.svc.cluster.local:50051",
    grpc.WithTransportCredentials(creds),
    grpc.WithDefaultServiceConfig(`{
      "loadBalancingConfig": [{"round_robin": {}}],
      "healthCheckConfig": {"serviceName": "readiness"}
    }`),
)

The client opens a Watch stream to each backend and stops sending calls to any that report NOT_SERVING, without waiting for DNS or Kubernetes to catch up. It only applies to load-balancing policies that use it, like round_robin; the default pick_first ignores it. Java and C++ clients read the same healthCheckConfig.

4. Envoy and service meshes

Envoy can actively check each upstream host with the gRPC health protocol. The cluster must speak HTTP/2 to the upstream:

clusters:
  - name: cart
    type: STRICT_DNS
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config: { http2_protocol_options: {} }
    health_checks:
      - timeout: 1s
        interval: 5s
        unhealthy_threshold: 2
        healthy_threshold: 2
        grpc_health_check:
          service_name: readiness
    load_assignment:
      cluster_name: cart
      endpoints:
        - lb_endpoints:
            - endpoint: { address: { socket_address: { address: cart, port_value: 50051 } } }

Only SERVING counts as healthy. In a mesh like Istio, the sidecar handles routing, but Kubernetes readiness still decides which pods are endpoints, so the probes in step 2 matter just as much.

5. Cloud load balancers

Google Cloud

Backend services support gRPC health checks natively, including a service name:

gcloud compute health-checks create grpc cart-health \
    --port=50051 --grpc-service-name=readiness

AWS Application Load Balancer

Create the target group with protocol version gRPC, set the health check path to /grpc.health.v1.Health/Check, and set the success codes to 0 (the gRPC OK code). The default path, /AWS.ALB/healthcheck, isn't implemented by your server, and the default success code, 12 (UNIMPLEMENTED), only proves that something gRPC-shaped answered.

ALB only reads the status code

The ALB judges the call by its gRPC status code and sends an empty request, which asks about the whole server. A server reporting NOT_SERVING still returns code 0, so the ALB keeps sending it traffic. If you rely on ALB health checks, keep the whole-server status accurate, or wrap the health service so that NOT_SERVING returns UNAVAILABLE instead.

6. Service registries

Consul checks gRPC services with the same protocol. In a service definition:

{
  "service": {
    "name": "cart",
    "port": 50051,
    "check": {
      "grpc": "127.0.0.1:50051/readiness",
      "grpc_use_tls": true,
      "interval": "10s",
      "timeout": "2s"
    }
  }
}

Leave off the /readiness part to check the whole server. Consul only hands out healthy instances to callers that discover the service through it.

7. Check it from outside too

Everything above checks servers from inside your network. None of it tells you whether customers can reach the service: DNS, the public load balancer, the certificate and the network path all sit in front. Add one external check on the public endpoint with the same protocol. In Spot Downtime that's a gRPC monitor: enter the public host and port, optionally the service name and an authorization header as metadata, and it alerts your team when the answer isn't SERVING. It tracks the certificate's expiry from the same call.

Checklist

  • The health service is registered, and readiness reflects real dependencies after a short grace period.
  • Liveness only depends on the process itself.
  • On shutdown, the server reports NOT_SERVING and waits before stopping.
  • Kubernetes has startup, liveness and readiness probes, with a plaintext path for the kubelet if needed.
  • Clients that balance calls themselves have healthCheckConfig with round_robin.
  • Load balancers use /grpc.health.v1.Health/Check with success code 0, not their defaults.
  • One outside check watches the public endpoint.

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading