CAP Theorem: The Fundamental Trade-off in Distributed Systems

The Core Problem:

When your network partitions (and it will), you face an unavoidable choice: stay consistent or stay available. You cannot have both.

CAP Theorem (Consistency, Availability, Partition tolerance) states that in a distributed system, when a network partition occurs, you must choose between Consistency (every read sees the latest write, formally linearizability) or Availability (every request to a healthy node gets a response). Partition tolerance is not optional. Real networks fail constantly: switches drop packets, cross region links flap, and entire availability zones can become unreachable.

What This Actually Means:

Consistency in CAP means linearizability: every operation appears to take effect atomically at a single point in time, in a global order that all nodes agree on. If you write X=5, any subsequent read anywhere must see X=5 or a newer value, never X=4.

Availability means every request to a non failed node must receive a response. No timeouts, no "sorry, try again later" messages. The system stays responsive even when parts of the network are partitioned.

❗ Remember: CAP is NOT about ACID transactions. ACID consistency refers to invariants within a single database transaction. CAP consistency is about replica agreement and whether reads across different nodes see the same data.
The PACELC Extension:

CAP only describes behavior during partitions. But systems make consistency versus latency trade-offs even when the network is healthy. PACELC extends this: if Partition, choose A or C; ELSE (no partition), choose Latency or Consistency.

Strong consistency requires coordination, typically a majority quorum or leader lease. This adds at least one network round trip to your critical path. In a single datacenter, that might be 1 to 5 milliseconds. Across regions, it is 50 to 150 milliseconds. Weaker consistency avoids some coordination, delivers lower tail latency (often single digit milliseconds), and sustains higher throughput, but allows stale reads and potential write conflicts.

Making the Choice:

Decide which operations must be linearizable. Money transfers, inventory decrements, and unique constraint enforcement need consistency. Profile view counts, social feed updates, and shopping cart additions can tolerate eventual consistency. Design your data model and user experience around the guarantees you choose.

💡 Key Takeaways

•CAP forces a choice during network partitions: linearizable consistency means some nodes must refuse requests (sacrificing availability), while availability means accepting potentially stale or conflicting data (sacrificing consistency)

•Partition tolerance is mandatory in real systems because networks fail constantly through packet loss, link failures, and datacenter outages, making the practical choice between C or A

•PACELC extends CAP to normal operation: even without partitions, systems trade latency for consistency. Strong consistency adds at least one network round trip (1 to 5ms intra region, 50 to 150ms cross region) versus single digit milliseconds for eventual consistency

•Strong consistency requires coordination mechanisms like majority quorums or leader leases, which reduce throughput under contention and create unavailability if majority is lost

•CAP consistency (linearizability) differs fundamentally from ACID consistency (transaction invariants). CAP describes cross replica agreement; ACID describes single database correctness

📌 Examples

Money transfer requires CP: if network partitions, better to reject transfer than allow duplicate withdrawals or inconsistent balances across replicas

Shopping cart can use AP: if network partitions, both sides accept cart additions. When partition heals, merge cart contents. User might see temporary duplicate items but cart remains available

Google Spanner chooses CP with TrueTime: commit wait adds epsilon (typically 7ms) plus quorum round trip. Multi region writes take 50 to 150ms but guarantee global consistency

Amazon DynamoDB Global Tables choose AP: last writer wins conflict resolution with sub second to low second cross region propagation, prioritizing availability over strict consistency

← Back to CAP Theorem Overview