Load Balancing • Health Checks & Failure DetectionHard⏱️ ~3 min
Gray Failures and Deep Health Check Implementation
Gray failures represent the hardest class of problems in distributed health checking: instances that appear healthy to simple probes yet deliver degraded service to real users. A process might return HTTP 200 on its health endpoint while experiencing CPU soft lockups, network interface packet loss, hot shard contention, or one exhausted database connection in a pool of 50. These partial failures cause high tail latency and intermittent errors invisible to shallow checks.
Deep health checks attempt to verify actual readiness by exercising critical dependencies during the probe. For a read service, this might mean pinging the database with a requirement that round trip completes within 50ms, checking cache connectivity, and verifying queue depth stays under 100 queued requests. The challenge is that deep checks across thousands of instances become a self inflicted Distributed Denial of Service (DDoS). If 5000 instances each query the database every 10 seconds, that's 500 queries per second just for health checks. Rate limit deep checks, cache results for 5 to 30 seconds, and add 0 to 20% schedule jitter to prevent synchronized thundering herd.
The most effective approach combines shallow active probes with passive tail latency monitoring. Google production uses basic liveness checks for process wedging, then instruments actual request path to track p99 latency over rolling 60 second windows. If p99 exceeds budget (e.g., 300ms for an API with 200ms Service Level Objective), readiness reports degraded and the instance advertises reduced weight or 503 status. This catches gray failures that synthetic health endpoints miss because it measures real user impact.
Queue depth and concurrency are powerful signals for capacity aware health. If in-flight requests exceed 2 to 4 times CPU core count or queue wait time exceeds 50 to 100ms for a low latency API, the instance should reduce advertised weight even before violating latency SLOs. This provides early warning and gradual degradation instead of binary cliff edge failures. Envoy and HAProxy support dynamic weight updates that enable this progressive load shedding under stress.
💡 Key Takeaways
•Gray failures like CPU soft lockups or partial network interface packet loss cause high tail latency while health endpoints still return 200. Google mitigates this by tracking p99 latency on the actual request path over 60 second rolling windows, marking unhealthy when exceeding budget.
•Deep health checks querying database and cache from 5000 instances at 10 second intervals generate 500 queries per second just for probes. Cache health results for 5 to 30 seconds, add 0 to 20% jitter to schedules, and prefer lightweight signals like connection pool liveliness over full transactions.
•Queue depth over 2 to 4 times CPU core count or wait time over 50 to 100ms for low latency APIs should trigger weight reduction before latency SLO breach. This progressive degradation prevents binary cliff edge failures that cause cascading overload.
•Treating dependency failures as liveness failures causes cascading restarts that amplify outages. If database is down, restarting all application instances makes recovery harder. Readiness should reflect dependency health; liveness should only restart truly wedged processes.
•Health check paths must traverse critical code and dependencies to catch misconfigurations like missing credentials or expired certificates. A trivial HTTP 200 handler that doesn't touch the database won't detect database authentication failures until real traffic arrives.
📌 Examples
An Amazon team discovered instances returning health check 200 OK while serving user requests with 2 second p99 latency due to a single bad disk causing filesystem stalls. They added passive p99 tracking with 500ms threshold to catch these gray failures, reducing user facing errors by 80%.
Netflix implements queue aware health by measuring in-flight request count and pre-processing wait time. When queue wait exceeds 100ms, the instance reports reduced capacity to Eureka and client side load balancers shift traffic proactively before timeout based SLO violations occur.
A Google service had health checks querying a database table every 5 seconds from 10000 instances, creating 2000 queries per second load. They switched to checking connection pool borrowable status (a local operation) and had 1% of instances perform actual database pings, sharing cached results via shared memory for 30 seconds.