Failure Modes and Edge Cases in Production

When Things Go Wrong: In interviews, demonstrating awareness of failure modes shows production experience. Let's cover where Druid breaks and how to mitigate. Ingestion Backpressure Crisis: Your Kafka topic normally sees 1 million events per second. Suddenly it spikes to 10 million due to a traffic surge or upstream bug emitting duplicate events. Your 20 indexing tasks can't keep up. What happens? First, real time query latency increases. Queries that should return data from the last 5 minutes now only see data from 10 minutes ago because recent segments haven't been built yet. The gap between event time and queryable time grows from 30 seconds to 5 minutes or more. Second, indexing tasks start running out of memory as they buffer more events before persisting segments. Tasks crash and restart, further falling behind. In severe cases, this creates a death spiral: tasks repeatedly OOM (Out Of Memory), Kafka consumer lag grows to millions of messages, and recovery takes hours. High Cardinality Dimension Explosion: You decide to add user_id as a dimension on a clickstream dataset with 50 million daily active users. Each user generates 100 events per day. With rollup, you expect compression, right? Wrong. Rollup only works when multiple events share the same dimension combination. With user_id included, almost no events have identical dimensions (same user, same timestamp bucket, same country, same device). Your data goes from 1 million rolled up rows per hour to 500 million raw rows. Segment sizes explode from 200 MB to 50 GB compressed. Query performance degrades severely. Dictionary encoding for 50 million distinct user_id values per day consumes gigabytes of memory. Bitmap indexes become massive. Simple aggregations that took 80 milliseconds now take 5 seconds. Solution: don't dimension high cardinality identifiers. Store user_id as a metric (non indexed column) for rare drill downs, or hash it to reduce cardinality (user_id modulo 1000), or offload per user detail to a separate key value store and join in application layer. Deep Storage Availability Failure: Your S3 bucket becomes temporarily unavailable due to AWS region issues. What's the blast radius? Query serving continues normally for segments already cached on historical nodes. But new segment loads fail. If a historical node crashes during the outage, it can't reload its segments from S3, reducing cluster capacity. Coordinators can't assign new segments or rebalance existing ones. Your cluster is essentially frozen in its current state. Ingestion is affected differently. Real time tasks can still ingest and build segments in memory, but they can't persist to S3. This limits how long they can run before running out of memory. If tasks crash before S3 recovers, you lose unbounded data. Query Pathologies: Certain query patterns cause predictable failures. An unbounded time range query ("show me all clicks ever") forces scanning every segment in deep storage: hundreds of terabytes, potentially taking minutes or timing out. A high cardinality GROUP BY on a poorly indexed dimension (GROUP BY session_id with 1 billion distinct sessions) causes memory exhaustion at the broker during merge. Protection mechanisms: set query timeout limits (30 seconds default), enforce time range maximums (no queries over 90 days), use query result size limits, and monitor broker heap usage. For multi tenant environments, use query priority queues to isolate expensive queries from critical dashboards.