What is Distributed SQL and How Does It Work?

Distributed SQL systems like Google Spanner and CockroachDB combine the familiar relational database model with a horizontally scalable, fault tolerant architecture. Instead of running on a single powerful server, these databases split data into small partitions called shards or ranges, then replicate each partition across multiple nodes using consensus protocols like Raft or Paxos.

The breakthrough is providing ACID transactions and strong consistency across this distributed architecture. A single node PostgreSQL can commit transactions in under 1 millisecond, but it cannot survive datacenter failures or scale beyond one machine's capacity. Distributed SQL trades that local speed for survivability and scale. Within a single region, consensus replicated writes complete in 2 to 5 milliseconds. Cross region writes take 50 to 150 milliseconds at the 95th percentile because they require majority agreement across geographically separated replicas.

At production scale, these systems handle massive workloads while appearing as a single logical database. Google's Spanner deployment for Ad platform metadata spans tens of thousands of nodes with over 2 exabytes of logical data, handling millions of operations per second. A typical CockroachDB deployment at a fintech like Coinbase might run hundreds of nodes with hundreds of terabytes of data, processing tens of thousands of transactions per second while automatically surviving full availability zone failures.

The key architectural components include Multi Version Concurrency Control (MVCC) for storing multiple timestamped versions of each key, consensus groups that replicate each data range across 3 to 7 nodes, and a global or logical clock that assigns timestamps to transactions. This timestamp based ordering ensures that even though data is scattered across thousands of nodes, reads see consistent snapshots and commits produce an externally consistent order, as if running on a single machine.

💡 Key Takeaways

•Distributed SQL provides ACID transactions and SQL across a horizontally scalable architecture using consensus replication (Raft or Paxos) for each data range

•Write latency increases from under 1ms (single node) to 2 to 5ms (single region) or 50 to 150ms (cross region) due to consensus coordination

•Production deployments scale to tens of thousands of nodes, petabytes of data, and millions of operations per second while surviving datacenter failures

•MVCC combined with timestamp based ordering ensures serializable transactions appear in a consistent order across all nodes

•Applications use standard SQL without manual shard routing, as the database automatically handles data placement and rebalancing

📌 Examples

Google Spanner deployment for Ad platform: tens of thousands of nodes, 2+ exabytes of logical data, millions of operations per second

Coinbase on CockroachDB: hundreds of nodes, hundreds of terabytes, tens of thousands of transactions per second with automatic zone failure recovery

Typical Spanner configuration: 3 replicas across Iowa, South Carolina, and Oregon with 30 to 60ms cross region latency

← Back to Distributed SQL (CockroachDB, Spanner) Overview