Holdout Assignment: Deterministic Hashing and Cohort Management

Core Concept
Holdout assignment uses deterministic hashing with a fixed salt to permanently assign users to holdout or production. The assignment must be stable across months/years.
Deterministic Assignment
Use hash(user_id + "holdout_v1") mod 100 < 5 for 5% holdout. The salt "holdout_v1" stays constant indefinitely. Unlike experiments, holdout salt never changes - the same users stay in holdout forever.
This differs from experiment assignment where salt changes per experiment. Holdout salt is permanent infrastructure, not per-test configuration.
Cohort Management
New users joining after holdout creation are assigned to holdout or production based on the same hash. This maintains 5% holdout ratio over time. User churn affects both groups equally, maintaining balance.
💡 Key Insight: For time-limited holdouts (6-12 month refresh), change the salt ("holdout_v2") to reshuffle. This creates a fresh random assignment, but loses continuity of long-term measurement.
Independence from Experiments
Holdout assignment must be independent from experiment assignment. Use different salt families. A user in holdout should still be randomly assigned to experiment buckets (but never see the treatment since features are gated on non-holdout status).

💡 Key Takeaways

✓Use deterministic hash with fixed salt: hash(user_id + "holdout_v1") mod 100 < 5 for 5%

✓Holdout salt is permanent infrastructure, unlike experiment salts that change per test

✓New users assigned by same hash, maintaining ratio; user churn affects both groups equally

✓Holdout assignment must be independent from experiment assignment (different salt families)

📌 Interview Tips

1When explaining assignment: describe permanent salt (holdout_v1) that never changes unlike experiment salts

2For refresh: change salt to v2 for new random assignment, but loses measurement continuity

← Back to Holdout Groups & Long-term Impact Overview