Failure Modes: Selection Bias, Contamination, and Reshuffling

Selection bias and contamination are the highest risk failure modes. If assignment is not truly random and deterministic, or if it drifts over time due to user ID changes, the comparison becomes biased. Cross device identity resolution issues can cause a user to be in holdout on one device and treatment on another. In marketing systems, held out users might still receive messages through another channel or be re included due to workflow misconfiguration. Bluecore designed for campaign independence with per campaign salts to avoid cross test correlation; without that, users could be systematically in or out across many tests, biasing results. Reshuffling is a subtle problem sometimes called the Interstellar Problem. After a 50/50 experiment, moving to a 5 to 95 holdout can split past control and treatment users into the new holdout. Some users will lose features they previously saw, which can depress engagement and create an unrepresentative baseline. Keeping original control users in the long holdout avoids that but starves them of months of improvements. When they finally receive the accumulated changes, there can be a shock effect that confounds measurement. Metrics can decay or drift over the holdout window. Early novelty effects can inflate treatment performance, then fade. Algorithms can overfit to short term behavior and degrade as the population adapts. Disney observed decreasing or inconsistent impact over time for some initially successful experiments. If you rely on slow moving metrics like retention, the holdout may need to run for quarters, increasing the chance that external events like seasonality, marketing campaigns, or content releases confound the comparison. Operational errors are common and catastrophic. A single team ignoring holdouts can invalidate months of data. Microservices must all consult a shared assignment function or membership table. Cache layers must not leak holdout decisions across users. Analytics must handle membership churn and late arriving events. If training pipelines for ML models do not isolate holdout exposures, the model can learn from treatment behavior that includes held out users indirectly, leaking signal and biasing the comparison.

Bluecore per campaign salt design prevents cross test correlation where users systematically in or out across many tests would bias aggregate results

Marketing system contamination: held out users receive messages through alternate channel (email versus push) or workflow misconfiguration re includes them, violating holdout integrity