Block Storage Deep Dive: IOPS, Latency, and Consistency Guarantees

Block storage virtualizes raw disk sectors over the network, providing the foundation for databases and VM disks that demand precise control over write ordering and high IOPS random access. Understanding performance characteristics and consistency guarantees is critical for production deployments.

Amazon EBS io2 Block Express volumes deliver up to 256,000 IOPS and 4,000 MB/s per volume with target latencies under 1 ms for properly sized operations. In practice, databases achieve 50,000 to 200,000 IOPS for 8 to 16 KiB random operations with p50 latencies of 0.5 to 1.5 ms and p99 of 3 to 8 ms depending on queue depth tuning and network provisioning. Azure Ultra Disk provides similar specs with configurable IOPS and throughput per disk, targeting sub millisecond read latency. The key is matching filesystem block size to underlying storage page size (typically 4 to 16 KiB) to avoid read modify write amplification that can triple write latency.

Consistency guarantees center on crash recovery and write ordering. Block storage honors host flush barriers and FSYNC semantics, ensuring that when the filesystem issues a flush, writes become durable in order. This enables database Write Ahead Logging (WAL) where transaction commits must persist before data page updates. Power loss before flush can corrupt filesystems or databases, requiring journaling and WAL to recover. Some hypervisors or network paths can reorder writes if write through caching is not enforced; verify cache settings on critical paths. Snapshots are typically crash consistent, capturing the state as if power was lost, which can include torn writes. For application consistent backups, quiesce the application or use filesystem freeze to coordinate log checkpoints.

The major failure mode is multi attach without cluster awareness. Mounting a standard filesystem from multiple hosts simultaneously causes split brain and data corruption since the filesystem assumes exclusive access. Use cluster aware filesystems like Oracle Cluster File System (OCFS2) or Global File System 2 (GFS2) if multiple writers are required, or enforce strict one writer many readers patterns with read only mounts.

💡 Key Takeaways

•Amazon EBS io2 Block Express and Azure Ultra Disk deliver 256,000 IOPS and 4,000 MB/s per volume with p50 latencies of 0.5 to 1.5 ms for 8 to 16 KiB operations when queue depth and network are tuned correctly

•Align filesystem block size to storage page size (4 to 16 KiB) to avoid read modify write amplification; misalignment can triple write latency by forcing extra read and write operations per logical write

•Block storage honors FSYNC and flush barriers enabling database WAL semantics where transaction commits persist durably before data page updates, critical for Atomicity Consistency Isolation Durability (ACID) guarantees

•Snapshots are crash consistent not application consistent; they may capture torn writes or inconsistent state as if power was lost suddenly, requiring application quiesce or filesystem freeze for clean backups

•Multi attach hazard: mounting a non cluster aware filesystem from multiple hosts causes split brain corruption; use OCFS2, GFS2, or enforce strict one writer pattern with read only secondary mounts

📌 Examples

PostgreSQL WAL on dedicated EBS volume: separate WAL from data pages, provision 50,000 IOPS for WAL writes, achieve consistent 1 ms commit latency with synchronous_commit=on and proper fsync barriers

MySQL InnoDB tuning: set innodb_flush_log_at_trx_commit=1 for durability, align innodb_page_size to 16 KiB EBS page size, use io2 with 100,000 provisioned IOPS for write heavy workloads with p99 under 5 ms

Snapshot coordination: use filesystem freeze before EBS snapshot creation to flush buffers and pause writes, ensuring application consistent backup without torn writes or pending transactions

← Back to Block vs Object vs File Storage Overview