Learn→A/B Testing & Experimentation→Experiment Design (Randomization, Stratification, Power Analysis)→1 of 6

A/B Testing & Experimentation • Experiment Design (Randomization, Stratification, Power Analysis)Easy⏱️ ~3 min

What is Randomization and Sticky Bucketing in Experiments?

Definition
Randomization assigns users to experiment variants using a deterministic hash. Sticky bucketing ensures the same user always sees the same variant throughout the experiment.
Deterministic Hashing
Instead of flipping a coin per request, compute hash(user_id + experiment_id) mod 100. Result 0-49 means control, 50-99 means treatment. Same inputs always produce same output, so User 12345 in Experiment ABC always lands in the same bucket.
Including experiment_id prevents cross-experiment correlation. Without it, User 12345 would always be in control (or treatment) across all experiments, creating systematic bias.
Stable Unit Identifiers
The unit ID must persist across sessions. Logged-in user IDs are most reliable. For logged-out users, use device fingerprint plus client ID stored in local storage. Accept that 5-15% of traffic will have inconsistent assignment due to storage clearing.
⚠️ Key Trade-off: Unstable identifiers (IP, session ID) cause users to flip between variants, corrupting your data. Stable identifiers work cross-session but may not work cross-device.
Assignment Timing
Assign at first exposure to the feature, not at page load. If your experiment affects checkout, assign at checkout page. This reduces dilution from users who never reach the affected feature.

💡 Key Takeaways

✓Deterministic hashing (user_id + experiment_id) ensures same user always sees same variant across sessions

✓Including experiment_id in hash prevents cross-experiment correlation that would bias simultaneous tests

✓Stable identifiers (user IDs > device IDs > cookies) are critical; unstable IDs corrupt results

✓Assign at first exposure to the feature, not at page load, to reduce dilution

📌 Interview Tips

1When explaining bucketing: describe hash(user_id + experiment_id) mod 100, emphasizing determinism for consistency

2For logged-out users: mention device fingerprint + client ID, accepting 5-15% inconsistent assignment

← Back to Experiment Design (Randomization, Stratification, Power Analysis) Overview