Understanding a Health Check
A health check is a mechanism, typically an HTTP endpoint or a lightweight command, that a service exposes to report its own operational status. External systems, such as load balancers, container orchestrators, and monitoring tools, call this endpoint periodically and use the response to make decisions: whether to send traffic to an instance, whether to restart a failed container, or whether to page an on-call engineer. Health checks are one of the simplest and most foundational observability mechanisms because they provide a direct, binary-ish signal of service state rather than requiring interpretation of a metric or log pattern.
Shallow vs. Deep Health Checks
- Shallow health checks verify only that the process is running and able to respond to a request at all, often just returning HTTP 200 with no further logic. These are cheap and fast but can give a false sense of health, for example reporting “healthy” even when the service’s database connection is down and every real request would fail.
- Deep health checks verify that critical dependencies, such as a database connection, a downstream API, or a message queue, are actually reachable and responding before reporting healthy. These are more accurate but riskier: if a shared downstream dependency degrades, every instance’s deep health check can fail simultaneously, potentially causing an orchestrator to restart or remove every instance at once, worsening an outage instead of containing it.
Liveness vs. Readiness
In Kubernetes specifically, health checks are split into two distinct probe types that answer different questions: a liveness probe answers “is this process stuck or deadlocked and should it be restarted,” while a readiness probe answers “is this instance currently able to serve traffic and should it receive requests.” A pod can be alive but not ready, for example during startup while it’s still warming a cache or establishing database connections, in which case Kubernetes keeps the process running but stops routing traffic to it until the readiness probe passes. Getting this distinction right matters: using a deep dependency check as a liveness probe can cause Kubernetes to kill and restart healthy pods just because an unrelated downstream service is temporarily unavailable.
A Concrete Example
An API service exposes /healthz as a shallow liveness check that just confirms the process can respond, and /readyz as a deeper readiness check that verifies its database connection pool has at least one available connection. During a brief database failover, the readiness check starts failing across all instances, so the load balancer and Kubernetes stop routing new traffic to them, which is the correct behavior since they genuinely cannot serve requests. Because the liveness check remains shallow and unaffected by the database issue, Kubernetes does not restart the pods, avoiding the churn of terminating and recreating containers for a transient, external dependency issue that will resolve on its own.
Health Checks Beyond Kubernetes
Load balancers, such as an AWS Application Load Balancer or Nginx, use health checks to decide whether to route traffic to a given backend instance, removing unhealthy instances from rotation automatically. Monitoring systems also scrape health check endpoints directly as a basic uptime signal, often the simplest metric feeding into an SLO for availability. Health checks are also foundational to graceful deployment strategies like rolling updates, canary deployments, and blue-green deployments, since the deployment system relies on health check status to decide when it’s safe to shift traffic to a new version.
Trade-offs and Best Practices
- Avoid making liveness checks depend on external services; reserve deep dependency checks for readiness probes to prevent cascading restarts during unrelated outages.
- Keep health checks lightweight and fast, since they are called frequently, often every few seconds, and a slow or resource-intensive check can itself become a performance problem.
- Set appropriate timeout and failure threshold values so a single slow response doesn’t trigger an unnecessary restart or traffic removal, while still detecting genuine failures quickly.
- Treat health check endpoints as part of the service’s public contract, documenting what each one actually verifies so downstream teams configuring probes understand what “healthy” really means.
Frequently Asked Questions
What is Health Check?
A health check is an endpoint or probe reporting whether a service is running correctly, used by load balancers and orchestrators to decide whether to route traffic to an instance or restart it. Checks range from shallow process checks to deep dependency verification.
How does Health Check work?
Health Check works by combining the components described in the sections above. The main page walks through the architecture, the typical use cases, and the trade-offs to weigh before adopting it.
Why does Health Check matter?
Teams adopt Health Check to ship faster, run more reliably, and reduce the cognitive load on engineers. The benefits, limits, and adjacent tools are covered in the body above.
When should you use Health Check?
Use Health Check when the problems it solves match what your team is hitting today. The page above outlines the signals that mean you should adopt it now, and the cases where a simpler approach is fine.
