gRPC health checks: how they work and how to monitor a gRPC service

An open port doesn't mean a gRPC service works. How the standard grpc.health.v1 health service works, how to add it in Go, Python, Java and .NET, and how to read the errors when a check fails.

· 5 min read · By the Spot Downtime team

A TCP check on a gRPC port tells you something accepted the connection. It doesn't tell you the server can answer a call. A wedged process, a server that lost its database, or a proxy in front that doesn't speak HTTP/2 all leave the port open. gRPC has a standard way to ask the real question: the health service.

The standard health service

grpc.health.v1.Health is a small service defined by the gRPC project and shipped with every major gRPC library. Its Check method takes a service name and returns one of four statuses:

  • SERVING: ready for calls.
  • NOT_SERVING: running, but it says it can't do its job right now, usually because something it depends on is down.
  • UNKNOWN: the server hasn't decided yet, typically while it's starting.
  • SERVICE_UNKNOWN: only from the streaming Watch method, for a name it doesn't know. Check returns the error NOT_FOUND instead.

An empty service name asks about the server as a whole. A full name like shop.v1.Cart asks about one service, so a server that hosts several can report each one separately. The names are case-sensitive and include the package.

Adding it to your server

It's a few lines in most languages. In Go:

import (
    "google.golang.org/grpc/health"
    healthpb "google.golang.org/grpc/health/grpc_health_v1"
)

hs := health.NewServer() // reports the whole server ("") as SERVING
healthpb.RegisterHealthServer(grpcServer, hs)

// Later, when a dependency fails or recovers:
hs.SetServingStatus("shop.v1.Cart", healthpb.HealthCheckResponse_NOT_SERVING)

In Python, install grpcio-health-checking:

from grpc_health.v1 import health, health_pb2, health_pb2_grpc

hs = health.HealthServicer()
health_pb2_grpc.add_HealthServicer_to_server(hs, server)
hs.set("", health_pb2.HealthCheckResponse.SERVING)

In Java, add grpc-services and register new HealthStatusManager().getHealthService() with your server builder. In ASP.NET Core, call services.AddGrpcHealthChecks() and app.MapGrpcHealthChecksService(), which also reports your existing ASP.NET Core health checks through gRPC.

Make NOT_SERVING mean something

A health service that always says SERVING is only a slightly better port check. Set NOT_SERVING when the server truly can't handle calls, like when its database pool is exhausted. Don't set it for things a retry fixes, or every blip becomes an outage. During shutdown, set NOT_SERVING first so load balancers stop sending traffic before the server stops.

Testing it by hand

grpc_health_probe is a small command-line tool that makes the call and exits non-zero unless the answer is SERVING:

grpc_health_probe -addr=api.example.com:443 -tls
grpc_health_probe -addr=localhost:50051 -service=shop.v1.Cart

Kubernetes can call the same service for liveness and readiness probes (grpc: { port: 50051 }), with no extra binary in the image. The kubelet's probe doesn't use TLS or send credentials, so it only works on a plaintext port without auth. For probes, client-side health checking, Envoy and cloud load balancers, see how to configure gRPC health checking.

When the check fails

  • UNIMPLEMENTED: the server doesn't have the health service registered. Add it as above, or fall back to a TCP check.
  • NOT_FOUND: the service name isn't registered with the health service. Check the spelling and the package prefix, or leave the name empty to check the whole server.
  • A TLS handshake error, or the connection dropped straight away: the client and server disagree about TLS. A plaintext server hangs up on a TLS hello, and a TLS server on a plaintext HTTP/2 preface.
  • HTTP 404, 415, 502 or 503, or an HTML page: something in front isn't passing gRPC through. gRPC needs HTTP/2 all the way to the app: nginx needs grpc_pass rather than proxy_pass, and Cloudflare has a gRPC setting under Network that must be on.
  • UNAUTHENTICATED: an auth interceptor covers every method, including the health service. Either exempt /grpc.health.v1.Health/ from auth, or send a token as metadata.

Monitoring it from outside

Probes inside the cluster tell you a pod is healthy, not that your customers can reach it. An outside check covers DNS, the load balancer, TLS and the network path too. In Spot Downtime, add a monitor and choose gRPC, enter the host and port, and optionally a service name and metadata such as authorization: Bearer …. It calls Check on every interval and is up only when the answer is SERVING. The certificate's expiry is tracked from the same call, like on a website check.

gRPC checks are on every plan, including Free. Checks come from the public internet, so a server that only listens on a private network can't be reached: rely on the Kubernetes probe there, and have a job inside the network ping a heartbeat so you hear when it stops.

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading