Deduplication at Scale: Memory and Performance Trade offs

The Memory Problem: Storing every event ID you have ever seen is impossible. At 500,000 events per second, you generate 43 billion events per day. If each event ID is 36 bytes (UUID) plus 64 bytes of overhead, that is 100 bytes per key. Keeping one day of state requires 4.3 terabytes of memory per partition. This does not fit in RAM. Window Based State: The most common solution is bounded windows. Keep only the last N hours of event IDs in memory. Events older than the window are assumed unique, even if they are duplicates. This trades completeness for bounded memory. For example, a 24 hour window at 100,000 events per second stores 8.6 billion keys. At 100 bytes each, that is 860 gigabytes. Distributed across 10 partitions, each holds 86 gigabytes, which fits in memory with RocksDB backing. Late arrivals beyond 24 hours bypass dedup but represent less than 1 percent of traffic in most systems. Approximate Structures: Bloom filters reduce memory dramatically. A Bloom filter for 1 billion keys with 1 percent false positive rate requires only 1.2 gigabytes instead of 100 gigabytes. However, false positives mean some unique events are incorrectly flagged as duplicates. False negatives are impossible, so you never accept a duplicate, but you may reject valid events. In practice, many systems use Bloom filters for hot path dedup in streaming, then rely on batch jobs to detect and correct any false positives. The batch layer has full context and can reprocess with exact logic. Batch Dedup at Scale: Batch dedup scans billions of rows and groups by dedup keys. The grouping operation is a distributed shuffle, which can bottleneck on network and memory. A single heavy hitter key (like a bot user_id appearing millions of times) can overload one reducer. The solution is two stage aggregation or sampling. First, group by a coarse key like date and region to spread load. Then within each group, apply secondary partitioning on user_id hash. This prevents any single reducer from handling more than a few million records.