Failure Modes: Reconnect Storms, Slow Consumers, and TCP Head of Line Blocking

Reconnect storms occur when a region outage, deployment, or network partition causes millions of clients to simultaneously reconnect, creating a thundering herd that spikes TLS handshakes, authentication requests, broker subscriptions, and gateway CPU. A system handling 5 million connections that experiences a 30 second outage will see all 5 million clients attempt reconnection nearly simultaneously if they use naive fixed retry intervals. This can overwhelm authentication services, exhaust connection pools, and cascade into prolonged unavailability. Mitigation requires exponential backoff with jitter on the client side (randomizing retry intervals across a growing window), server side retry after hints in error responses to spread load, staggered token TTLs so not all tokens expire simultaneously, and admission control or rate limiting on connection acceptance to bound backend load during recovery. Slow or stuck consumers present another critical failure mode. Clients with constrained bandwidth, frozen browser tabs, or misbehaving code cause server send buffers to grow unboundedly, risking out of memory errors and increasing latency for other streams multiplexed on the same TCP connection due to head of line blocking. Production systems apply strict backpressure: per connection send buffers with hard caps (tens to hundreds of KB), drop policies when buffers fill (drop oldest updates, coalesce intermediate states like presence changes, or pause low priority subscriptions), and per topic rate limits to protect against abusive producers. Without these safeguards, a single slow client can exhaust server memory or degrade performance for thousands of other connections on the same gateway node. TCP head of line blocking within a single connection is an inherent protocol limitation: TCP delivers bytes in strict order, so a large message (for example, a multi megabyte image mistakenly sent over the same WebSocket channel as control messages) will delay all subsequent small messages until the large one fully transmits. This is particularly problematic for mixed workloads where latency sensitive control signals share a connection with bulk data. Solutions include segregating traffic classes onto separate connections or channels, chunking large payloads into smaller frames with interleaving, and prioritizing control frames. Additionally, intermediaries like NATs and corporate proxies often drop idle connections at 30 to 120 seconds without notification, causing silent connection death if heartbeats are missing or too infrequent. Production systems send ping/pong frames at intervals shorter than the shortest known idle timeout (commonly every 20 to 30 seconds) and disconnect clients that miss multiple consecutive heartbeats.

Discord shards clients by hash to isolate failures and prevent a single partition outage from triggering a full reconnect storm across all 5 million plus connections, using consistent hashing with virtual nodes for dynamic rebalancing

Slack colocates WebSocket edges with users and applies backpressure policies to prevent slow consumers in one region from affecting message delivery in others, maintaining sub 500 ms 99th percentile latencies under normal load