Production Implementation: Observability, Testing, and Capacity Planning

Implementing leader election in production requires comprehensive observability, rigorous testing, and capacity planning beyond the core algorithm. Key metrics include time since last leader heartbeat (alerts should fire when this exceeds 1.5 to 2 times the expected heartbeat interval), election duration percentiles (p50, p95, p99 to understand tail behavior), leader changes per day (excessive churn indicates false positives or instability), and quorum Remote Procedure Call (RPC) latencies (high tail latencies predict imminent false elections). Establish Service Level Objectives (SLOs) for control plane availability such as 99.9 percent of leader failovers complete within 2 seconds in single region clusters or 5 seconds across regions, and track compliance over time. Chaos testing is essential because leader election edge cases rarely appear in normal operation. Regularly test induced network partitions (disconnect the leader from 1 or more followers), simulate garbage collection pauses (using SIGSTOP to freeze a leader process for seconds), inject process crashes at various election phases, and create disk stalls on the leader to validate fencing and lack of split brain. For example, freeze the current leader for twice the election timeout, verify a new leader is elected and serving traffic, then unfreeze the old leader and confirm it recognizes its stale epoch and steps down without attempting writes. Automated chaos experiments should run continuously in staging and periodically in production to catch regressions. Capacity planning must account for quorum requirements and failure domains. For 3 nodes, a single failure eliminates spare capacity: you are running at 66 percent capacity during normal operation and 100 percent after one failure, with no tolerance for a second failure. For 5 nodes, you can lose two nodes and maintain quorum, but performance degrades significantly when running on 3 of 5 nodes. Spread nodes across failure domains (racks or Availability Zones) to tolerate correlated failures: for example, deploy 5 nodes across 3 AZs with a 2 plus 2 plus 1 layout so losing any single AZ leaves a majority. Monitor resource saturation on leaders because leaders become bottlenecks: if the leader Central Processing Unit (CPU) or network saturates, backpressure and timeouts trigger unnecessary elections. Mitigate with per shard leaders to distribute load, flow control to prevent overload, and offloading reads to follower replicas where consistency permits.

Google monitors Chubby session metrics and alerts when renewal latencies exceed p99 thresholds, indicating risk of false session expirations. Chubby SLOs target 99.9 percent of lock acquisition attempts succeeding within 5 seconds

Netflix Chaos Monkey for databases injects leader failures during peak traffic to validate sub 10 second failover SLOs and verify that client libraries correctly retry and discover new leaders without exposing errors to end users

A Kafka cluster with 1,000 partitions distributes leadership across 10 brokers (approximately 100 partitions per broker leader), limiting blast radius when one broker fails and preventing any single broker from becoming a CPU bottleneck. Per broker leader metrics track saturation and trigger rebalancing

etcd deployed in Kubernetes with 5 members across 3 AZs in 2 plus 2 plus 1 layout survives any single AZ failure with 3 remaining members forming quorum, maintaining control plane availability during infrastructure maintenance or outages