File Storage Trade-offs: Metadata Coordination and Shared POSIX Semantics

File storage provides a hierarchical namespace with POSIX like semantics including directories, atomic rename, file locking, and permissions. This enables familiar tooling and shared access patterns across multiple clients, but introduces metadata coordination overhead and scaling challenges that block and object storage avoid. Amazon EFS and Google Filestore High Scale separate metadata servers from data storage nodes. Metadata servers manage the directory tree, inodes, and lock state, often using consensus protocols like Raft or Paxos for consistency. Data is striped across many chunk servers for parallel throughput. Clients cache directory entries and file attributes to reduce metadata Round Trip Times (RTTs), but any operation that modifies directory structure like create, delete, or rename requires coordinating with the metadata service. This coordination introduces 10 to 20 ms typical latencies for metadata operations compared to sub millisecond block storage. Amazon EFS achieves 10+ GB/s aggregate throughput and hundreds of thousands of IOPS across many clients, but individual metadata heavy workloads like listing millions of files in a directory or heavy rename activity can bottleneck on the metadata tier. The shared access model relies on close to open consistency semantics and lease based locking. When a client writes to a file and closes it, other clients are guaranteed to see the new content when they open the file. Between open and close, visibility depends on explicit fsync or flush operations. Byte range locks or whole file locks coordinate concurrent writers, but lock manager failure or client crashes can leave stale locks that block progress until lease expiry or manual recovery. Network File System (NFS) lock state is traditionally not durable; a server reboot can lose lock information requiring client side retries and potential application level conflict resolution. The major scaling hazard is metadata hotspots. Millions of files in a single directory or heavy create/delete/rename activity in one path overload the metadata service causing tail latency spikes and throttling. The solution is to distribute load across directories by sharding or bucketing. For example, instead of placing 100 million files in one directory, use a two level hierarchy with 1,000 subdirectories each containing 100,000 files. Pre create directory trees during setup to amortize metadata costs. Batch small writes into larger files to reduce file count. Cost for managed file storage is $0.20 to $0.35 per GB month, significantly higher than block ($0.07 to $0.10) or object ($0.02 to $0.03), reflecting the complexity of distributed metadata coordination and multi client support.

ML feature store on EFS: 200 training instances reading shared Parquet files, 15 GB/s aggregate throughput, atomic feature updates via rename to tmp then rename to final path ensuring consistency

CI/CD artifact sharing: Jenkins with 50 build agents on EFS mount, atomic rename enables safe artifact publishing, but initial directory listing of 1 million artifacts takes 30 seconds due to metadata scan

Media rendering farm on Google Filestore High Scale: 100 render nodes accessing shared asset library, POSIX permissions for access control, but hot directory with 500,000 assets causes p99 latency spikes requiring sharding into project subdirectories