Learn→A/B Testing & Experimentation→Experiment Design (Randomization, Stratification, Power Analysis)→2 of 6

A/B Testing & Experimentation • Experiment Design (Randomization, Stratification, Power Analysis)Medium⏱️ ~3 min

How Does Stratification Reduce Variance in Experiments?

Definition
Stratification partitions users into homogeneous groups before randomization. Within each stratum, users are randomly assigned to control or treatment, ensuring balanced representation across important segments.
Why Stratify
Simple randomization can accidentally create imbalanced groups. In a 1000-user experiment, random chance might put 60% of iOS users in treatment. If iOS converts 2x higher than Android, this inflates your treatment effect estimate. Stratification guarantees equal iOS/Android splits in both arms.
Variance reduction is proportional to how predictive the stratification variable is. If platform explains 20% of outcome variance, stratifying reduces experiment variance by ~20%, meaning fewer users needed for same statistical power.
Choosing Variables
Good stratification variables are: known before randomization, predictive of outcome, and available for all users. Common choices: platform, country, user tenure, baseline engagement. Limit to 3-5 dimensions with 2-4 levels each - over-stratification fragments your sample.
💡 Key Insight: Post-stratification (analyzing by strata after experiment) gives similar variance reduction without complicating assignment. Use pre-stratification when strata are clearly defined and critical.
Implementation
Include stratum_id in hash: hash(user_id + experiment_id + stratum_id) mod 100. This ensures 50/50 splits within each stratum while maintaining sticky bucketing. Analyze by computing within-stratum effects, then combining with weighted averages.

💡 Key Takeaways

✓Stratification guarantees balanced representation across important segments, removing accidental imbalance variance

✓Variance reduction is proportional to predictive power of stratification variable (20% explained = 20% reduction)

✓Limit to 3-5 dimensions with 2-4 levels each to avoid fragmenting sample into too many small strata

✓Post-stratification gives similar variance reduction without complicating assignment logic

📌 Interview Tips

1When asked about variance reduction: explain stratification by platform/country with proportional variance reduction

2For implementation: describe including stratum_id in hash for within-stratum randomization

← Back to Experiment Design (Randomization, Stratification, Power Analysis) Overview