Production Implementation Patterns for L4/L7 Load Balancers

Effective L4 implementation starts with choosing between full proxy Network Address Translation (NAT) and Direct Server Return (DSR). Full proxy NAT provides stateful per flow tracking with precise control and better observability but requires egress capacity; budget 200 to 500 bytes per active Transmission Control Protocol (TCP) flow. DSR removes egress load by having servers reply directly to clients, achieving higher throughput (10 to 40 Gigabits per second (Gbps) per server with Meta Katran) but complicates source Internet Protocol (IP) preservation and Access Control Lists (ACLs). For global ingress, deploy anycast Virtual IP (VIPs) with Equal Cost Multipath (ECMP) routing to the nearest load balancer, then use consistent hashing (Maglev style lookup tables) from load balancer to backends to preserve flow stickiness. This minimizes connection remaps to under 5 to 10 percent when adding or removing nodes. Apply per backend weights for heterogeneous capacity and implement slow start when adding new backends to avoid immediate overload. L7 implementation requires full proxy architecture with separate client and upstream connection pools. Terminate Transport Layer Security (TLS) at the edge using modern ciphers (Elliptic Curve Diffie Hellman Ephemeral/Advanced Encryption Standard Galois Counter Mode (ECDHE/AES GCM) at 1 to 5 Gbps per core); re encrypt to backends with mutual TLS (mTLS) for zero trust networks. Normalize and validate HyperText Transfer Protocol (HTTP) strictly before routing: enforce header size limits (8 to 16 Kilobytes (KB) typical), reject malformed requests, and canonicalize header casing to prevent security bypasses. Route on host, path, header, or cookie with weighted splitting for canary releases (1 to 10 percent traffic) using request ID pinning to keep subsequent calls coherent. Implement power of two choices or Exponentially Weighted Moving Average (EWMA) latency balancing for per request load aware distribution, reducing tail latency by 10 to 30 percent compared to round robin. Health and resiliency differ by layer. L4 health checks use TCP connect or TLS handshake validation; mark backends down quickly on SYN timeouts (typically 3 to 5 second threshold). L7 requires HTTP health checks validating status codes and response body patterns. Implement outlier detection that ejects hosts with elevated 5xx rates (for example, 5 consecutive errors or 10 percent error rate over 30 seconds) with quarantine timers (30 to 60 second cool down before retry). Configure per route timeouts (connect 250 milliseconds to 1 second, header 5 to 30 seconds, idle 30 to 300 seconds) and retry policies with jittered backoff and budgets (limit fleet wide retries to 1.2x base load to prevent amplification). Connection management critically impacts performance. Prefer keepalives to amortize TCP and TLS handshakes; TLS 1.3 reduces handshake Round Trip Time (RTT) and session resumption or 0 RTT further improves latency where safe. Size upstream pools to cap concurrent connections on backends (typically 50 to 200 connections per backend); for HTTP/2, tune maximum concurrent streams (100 to 250 typical) to avoid head of line blocking. Segregate long lived connections (WebSockets, gRPC) into dedicated pools or L4 paths to prevent starving short requests. For capacity planning, maintain 20 to 30 percent headroom for failover; use consistent hashing to keep remaps under 5 to 10 percent during scale events; and monitor NAT table utilization at L4 (alert above 70 percent) and connection pool saturation at L7 (alert when queuing exceeds 10 milliseconds).

Google production: Anycast VIP to Maglev (L4) with consistent hashing, then GFE (L7) terminates TLS and routes; failover converges in seconds with under 5 percent connection remaps

AWS pattern: Network Load Balancer (L4 anycast) provides static IP and ultra low latency, forwards to Application Load Balancer (L7) for host and path routing with TLS termination

Service mesh config: Envoy sidecar with upstream pool of 100 connections per backend, HTTP/2 max 200 concurrent streams, outlier detection ejecting after 5 errors with 30 second quarantine