What Are Holdout Groups and Why Do They Matter?

Definition
Holdout groups are a permanent subset of users (typically 1-10%) who never see new features, providing a baseline to measure cumulative long-term impact of all shipped changes.
Why Holdouts Matter
Individual A/B tests measure short-term effects. But many small changes compound over months. A feature that lifts engagement 1% might reduce retention 0.5% - invisible in a 2-week test, devastating over a year. Holdouts reveal this cumulative impact.
Without holdouts, you cannot measure total improvement from all experiments. Each experiment compares against the current state, but the current state keeps changing. Holdouts freeze a baseline for long-term comparison.
Holdout Types
Universal holdout: excluded from ALL new features. Measures total experimentation value. Feature holdout: excluded from specific feature area (e.g., all recommendation changes). Measures area-specific value. Time-limited holdout: held for specific period (6-12 months), then refreshed.
⚠️ Key Trade-off: Larger holdouts give more statistical power but sacrifice revenue/engagement from holding back improvements. 5% holdout is common balance.
Long-Term Measurement
Compare holdout to production on metrics like 90-day retention, lifetime value, annual revenue. These long-latency metrics are impossible to measure in standard 2-4 week experiments.

💡 Key Takeaways

✓Holdouts are permanent subset (1-10%) who never see new features, providing baseline for cumulative impact

✓Individual tests miss compounding effects: 1% engagement lift with 0.5% retention drop is invisible short-term

✓Universal holdout measures total experimentation value; feature holdout measures specific area value

✓5% holdout balances statistical power against lost revenue from withholding improvements

📌 Interview Tips

1When explaining holdouts: describe measuring 90-day retention and LTV impossible in 2-week experiments

2For holdout types: universal (all features), feature-specific (one area), time-limited (6-12 months then refresh)

← Back to Holdout Groups & Long-term Impact Overview