Learn→A/B Testing & Experimentation→Experiment Design (Randomization, Stratification, Power Analysis)→4 of 6

A/B Testing & Experimentation • Experiment Design (Randomization, Stratification, Power Analysis)Medium⏱️ ~3 min

How Does Event Triggered Assignment Reduce Noise?

Definition
Event-triggered assignment restricts randomization to users who reach a qualifying event, excluding users who could never be affected by the treatment.
The Dilution Problem
Top-of-funnel assignment randomizes all users on first request. For a checkout experiment, this includes users who never browse products, never add to cart, never reach checkout. These users contribute zero signal but add variance, diluting your ability to detect effects.
If only 5% of users reach checkout, you are measuring the treatment effect on 100% of users but only 5% could possibly be affected. The 95% dilution multiplies your required sample size by roughly 20x.
Triggering at the Right Moment
For a checkout experiment, trigger assignment when user reaches checkout page. For a recommendation algorithm, trigger at first recommendation display. The trigger event should be the last moment before treatment exposure.
💡 Key Insight: The trigger event must happen BEFORE the user can see the treatment. If you assign after showing the treatment, you introduce selection bias - users who bounced immediately are never assigned.
Trade-offs
Event-triggered experiments have smaller sample (only triggered users) but cleaner signal. They cannot measure upstream effects (does treatment change funnel entry rate?). Use top-of-funnel assignment when measuring total business impact; use event-triggered when measuring feature-specific effects.

💡 Key Takeaways

✓Top-of-funnel assignment dilutes signal by including users who never encounter the treatment

✓If only 5% reach the feature, 95% dilution multiplies required sample by ~20x

✓Trigger at the last moment before treatment exposure to maximize signal

✓Event-triggered experiments cannot measure upstream funnel effects

📌 Interview Tips

1When asked about dilution: explain that 5% checkout rate means 95% dilution, 20x sample multiplier

2For trigger timing: emphasize assigning BEFORE treatment exposure to avoid selection bias

← Back to Experiment Design (Randomization, Stratification, Power Analysis) Overview