Document Database Failure Modes and Edge Cases

Even well designed document database systems encounter failure modes under production load. Understanding these edge cases prevents outages and SLA violations. Hot partitions, per-document write limits, index fan-out, and unbounded document growth are the most common culprits. Each has specific symptoms and mitigation patterns proven at scale.

Hot partitions occur when a shard key does not account for skew. A viral product or celebrity user can concentrate traffic on one shard even with a hashed key if the distribution is uneven. Symptoms include p99 latency jumping from 10ms to 500ms for queries hitting that shard while other shards remain healthy. Firestore enforces per-document write limits of approximately 1 sustained write per second per document. High-frequency counters (page views, likes) hitting one document will throttle. The standard solution is sharded counters: split the counter across 100 to 1,000 documents, each accepting increments independently, then sum on read. This trades read latency (aggregate N shards) for write throughput (N times higher capacity).

Index fan-out from large arrays amplifies write cost. A document with 1,000 tags indexed as multi-key generates 1,000 index entries per write. Updating that document takes 500ms instead of 5ms and consumes proportional storage. If many documents have large indexed arrays, write throughput collapses. Keep indexed arrays bounded (under 100 elements), move unbounded lists to separate child collections, or avoid indexing the array altogether and filter in application code.

Unbounded document growth hits hard size limits. MongoDB caps documents at 16 MiB, Firestore at approximately 1 MiB. Embedding unbounded arrays (comments, events, logs) eventually hits the limit, causing write failures. Symptom: writes succeed for months then suddenly fail with document size errors as arrays grow. Solution: split into parent document with pre-aggregated summary (comment count, top commenter) and child collection for individual items. This pattern is used at Meta for post comments: the post document stores commentCount and topComments (array of 3), full comment list lives in separate comments subcollection with pagination.

Replication lag and rollbacks in non-majority configurations can cause data loss. A write acknowledged by the primary but not yet replicated to secondaries is lost if the primary crashes and a secondary is elected. Use majority write concern for critical data. Monitor replication lag: if secondaries fall 10+ seconds behind, consider shedding read traffic from lagging replicas to avoid serving very stale data.

💡 Key Takeaways

•Hot partitions from skewed distribution: celebrity user or viral product concentrates traffic on one shard, p99 latency spikes from 10ms to 500ms while other shards idle, requires shard key redesign or secondary distribution key

•Firestore per-document write limit approximately 1 write/sec sustained: high-frequency counters throttle, sharded counter pattern splits across N documents for N times write capacity, trades read aggregation cost

•Index fan-out from large arrays: document with 1,000 indexed tags generates 1,000 index entries, single update takes 500ms instead of 5ms, multiply by write rate for total impact, bound arrays to under 100 elements

•Document size limits are strict: MongoDB 16 MiB, Firestore ~1 MiB, embedding unbounded arrays (comments, logs) eventually hits limit causing write failures, split into parent summary plus child collection pattern

•Non-majority write concern risks rollbacks: write acknowledged by primary but not replicated is lost on primary crash, use majority write concern for financial, booking, or critical user data despite latency cost

•Replication lag monitoring prevents stale reads: secondary 10+ seconds behind serves outdated data, users see inconsistent state, shed read traffic from lagging replicas or alert on lag exceeding threshold

📌 Examples

Hot partition: viral tweet with 1M likes, all like increments hit one document, Firestore throttles writes to 1/sec, users see "like failed" errors, solution is sharded counter across 1,000 documents for 1,000 writes/sec capacity

Index fan-out: product document with 5,000 search keywords indexed as array, single product update writes 5,000 index entries taking 800ms, reduce to 50 core keywords and move rest to separate search index

Unbounded growth: blog post document embeds comments array, after 2 years hits 1 MiB Firestore limit, writes fail, refactor to post document with commentCount field and /posts/{id}/comments subcollection for pagination

Rollback scenario: MongoDB w:1 write acknowledged, primary crashes before replication, secondary elected as new primary without the write, data lost, user sees order placed then disappears, majority write prevents this

← Back to Document Databases (MongoDB, Firestore) Overview