High Cardinality Aggregations and Memory Safety
High Cardinality Memory Risk
Aggregations (operations that compute summaries across documents, like counts, averages, or distinct values) execute in two phases: each shard computes local results, then a coordinator node (the node that received the query and orchestrates shard communication) merges these partial results into the final response. The critical risk is high cardinality fields (fields with millions of unique values). A terms aggregation counting unique values for a user ID field with 10 million distinct values must track counts for each value in heap memory (the working memory allocated to the search process).
Without safeguards, a single malicious or poorly designed query can allocate gigabytes of heap and crash nodes. Consider a dashboard query computing distinct values across all fields in a log index: if timestamp precision is milliseconds and logs span months, cardinality explodes into billions. Production systems must enforce cardinality limits through query validation, rejecting aggregations that would enumerate more than a configurable maximum (typically 10,000 to 100,000 unique terms).
Probabilistic Approximations
Probabilistic data structures solve cardinality problems by trading perfect accuracy for bounded memory. HyperLogLog++ (HLL++) computes distinct counts using only 16 KB of memory regardless of cardinality, with approximately 1 to 2% error. Counting 1 million unique user IDs uses the same 16 KB as counting 1 billion. The algorithm works by hashing values and tracking the longest run of leading zeros seen, which statistically correlates with cardinality.
T-digest and Q-digest compute percentiles over streaming data with 1 to 10 KB memory instead of storing all values. Computing p99 latency traditionally requires sorting all values and finding the 99th percentile, but T-digest maintains a compressed representation that approximates percentiles within configurable error bounds. These structures enable analytics over high-cardinality fields that would otherwise exhaust memory.
Doc Values and Field Optimization
Doc values store field values in columnar format on disk rather than requiring entire documents to load into heap for aggregations. When computing average price across a million documents, doc values read only the price column from disk. Numeric, date, and keyword fields automatically use doc values. However, analyzed text fields (full-text fields processed through tokenization) cannot use doc values because the original text is discarded during analysis.
The dangerous pattern is fielddata: loading analyzed text field values into heap for aggregations or sorting. If a user tries to aggregate on a text field without doc values, the system loads all unique terms into memory. A text field with 50 million unique terms across shards can consume 10 GB plus heap per shard. Prevention requires using keyword sub-fields for aggregatable text and disabling fielddata on analyzed text fields.
Circuit Breakers and Protection
Circuit breakers provide last-resort protection by estimating query memory requirements and rejecting queries that would exceed thresholds. A request circuit breaker rejects individual queries exceeding heap limits (typically 60% of heap). A fielddata circuit breaker caps total fielddata cache. An in-flight requests breaker limits concurrent query memory. When heap usage crosses 70 to 85%, breakers trip and reject new queries with explicit errors rather than allowing out-of-memory crashes.
However, circuit breakers are reactive protection, not preventive design. Relying on breakers means queries are already problematic when rejected. Better approaches include enforced cardinality limits in query templates, approximate aggregations for high-cardinality fields by default, query timeouts that terminate runaway aggregations, and monitoring dashboards that alert on memory pressure before breakers trip.