Learn→A/B Testing & Experimentation→Experiment Design (Randomization, Stratification, Power Analysis)→4 of 6
A/B Testing & Experimentation • Experiment Design (Randomization, Stratification, Power Analysis)Medium⏱️ ~3 min
How Does Event Triggered Assignment Reduce Noise?
Event triggered experiments restrict randomization to users who reach a qualifying pre-exposure event, excluding users who could never be affected by the treatment. Top of funnel assignment randomizes all users on first request or login, which is simple but includes many irrelevant users who never reach the changed surface. This inflates variance and dilutes the observed effect size, extending required duration by weeks or months.
Consider a checkout flow experiment that only impacts users who add items to cart. Randomizing all visitors dilutes the effect because most never reach checkout. In one production scenario, randomizing all users required 646,290 samples and 181 days to detect a 10 percent relative lift in conversion at 2 percent baseline. Restricting randomization to users who triggered an add to cart event reduced the requirement to 29,502 samples and 41 days. The 22x reduction in sample size comes from higher effective treatment rate and lower noise from unaffected users.
Implementation requires an eligibility service that evaluates trigger conditions on streaming events, such as viewing a product page, entering search results, or clicking a notification. On trigger, the service calls assignment and emits an exposure log exactly once, using an idempotence key to prevent duplicates. The exposure includes unit ID, variant, stratum, trigger event, and timestamp. Both exposures and outcomes are ingested into a metrics pipeline that deduplicates, windows late arrivals, and computes intent to treat estimates.
Trigger definition is critical. The trigger must be a pre-exposure event that is unaffected by treatment, otherwise selection bias creeps in. Triggering on add to cart while the treatment influences browsing to cart excludes users who were impacted during browsing. Use triggers like page view, search query, or session start that happen before the treatment is applied. At Netflix and Airbnb, mid funnel experiments for ranking or pricing routinely use event triggers to focus power on the affected segment, cutting time to significance from months to weeks.
💡 Key Takeaways
•Top of funnel assignment randomizes all users, which inflates variance and dilutes effects when most users never reach the changed surface
•Event triggered assignment on add to cart reduced required sample from 646,290 to 29,502 (22x smaller) and duration from 181 to 41 days for a checkout flow experiment
•Trigger must be a pre-exposure event unaffected by treatment to avoid selection bias; page view or session start are safe, post-treatment events like add to cart can create bias
•Eligibility service evaluates trigger conditions on streaming events, calls assignment once, and emits exposure log with idempotence key to prevent duplicates
•Netflix and Airbnb use event triggers for mid funnel ranking and pricing experiments, cutting time to significance from months to weeks
•Exposure and outcome ingestion must handle deduplication, late arrivals, and windowing with end to end lag under 5 minutes for monitoring, hourly to daily for final analysis
📌 Examples
Pinterest restricts homefeed ranking experiments to users who scroll past the first 3 pins, excluding inactive users and reducing duration by 40 percent
Uber triggers surge pricing experiments on ride request events rather than app opens, focusing power on users actively booking rides
LinkedIn triggers feed ranking experiments when users load the feed, excluding users who navigate directly to profiles or messages