Scaling Decision Framework: When and How to Scale

The First Question: Should You Scale at All? Before adding infrastructure, ask whether optimization solves the problem cheaper. A single PostgreSQL instance handles 10,000 to 50,000 queries per second with proper indexing. Adding an index takes minutes and costs nothing. Adding read replicas takes days and costs hundreds per month. Many teams scale prematurely when a missing index or N+1 query is the real bottleneck. Metrics That Trigger Scaling: Scale when you see these sustained for 15+ minutes during peak hours: Central Processing Unit (CPU) utilization above 70% means requests are waiting for compute. Memory above 80% risks swap thrashing and garbage collection storms. Request queue depth growing means throughput capacity is exceeded. p99 latency exceeding 5x your p50 indicates resource contention. Error rate above 0.1% from timeouts or connection failures signals overload. The Cost Reality: Horizontal scaling has hidden costs beyond server bills. Ten small servers need configuration management, deployment pipelines hitting 10 targets, monitoring dashboards tracking 10 instances, and on call runbooks for distributed failure modes. A single large server costs more per unit of compute but less in operational overhead. For a startup with 2 engineers, operational simplicity often beats theoretical scalability. The break even point varies, but typically: below 5,000 requests per second, vertical scaling wins on total cost. Between 5,000 and 50,000 requests per second, either works depending on availability requirements. Above 50,000 requests per second, horizontal scaling becomes necessary regardless of preference.