Holdout Assignment: Deterministic Hashing and Cohort Management

Holdout assignment must be deterministic, fast, and consistent across the user journey. The standard pattern uses salted hashing of a stable user identifier to assign users without requiring hot path database lookups. You hash the user ID concatenated with a cohort specific salt, map the result to a numeric range (typically 0 to 99 for percentages), and assign to holdout or treatment based on your split. This gives consistent assignment within a cohort, independence across different experiments or campaigns, and sub millisecond latency.

Bluecore's marketing platform implemented per campaign salts so that assignment is independent across campaigns and consistent within each one. A typical ML recommendation service allocates only 10 to 50 milliseconds for all feature flag checks and routing, so the assignment function must be pure and avoid data store calls. The function depends only on the user ID and salt, making it cacheable and reproducible for debugging.

Cohort windows require explicit definition. Disney Streaming uses a 3 month enrollment period where new and existing qualified users are sampled, followed by a 1 month evaluation period where no new users are added. Maintain a membership table keyed by user ID and cohort ID with start and end timestamps. Tag all exposure and outcome events with the cohort ID to simplify analysis and provide an audit trail. This allows you to handle membership churn, late arriving events, and recompute metrics without losing historical context.

For universal holdouts, rotate the salt quarterly to avoid permanently depriving the same users of improvements. For local or feature holdouts, use a feature specific salt and coordinate across teams to prevent collision. Bluecore's pre filter and post filter table pattern shows the value of an intermediary state: store the original audience before holdout filtering and the final audience after, so downstream systems remain unaware of the holdout while analytics can recompute or debug memberships.

💡 Key Takeaways

•Use salted hashing (user ID plus cohort salt) to assign deterministically without database lookups, keeping latency under 10 to 50 milliseconds on the hot path

•Define explicit cohort windows with enrollment periods (3 months at Disney) and evaluation periods (1 month), maintaining membership tables with user ID, cohort ID, and timestamps

•Rotate universal holdout salts quarterly to avoid permanently depriving the same users, while using feature specific salts for local holdouts to enable independence

•Bluecore's pre filter and post filter table pattern stores original audience before holdout filtering and final audience after, enabling downstream system transparency and analytics debugging

•Tag all exposure and outcome events with cohort ID to handle late arriving events, membership churn, and provide an audit trail for metric recomputation

📌 Examples

Bluecore salted hash assignment: hash(campaign_id || user_id || campaign_salt) mod 100, assigns consistently within campaign and independently across campaigns, avoids additional storage

Disney Streaming cohort management: 3 month enrollment window followed by 1 month evaluation, membership table tracks (user_id, cohort_id, start_ts, end_ts), all events tagged with cohort_id

← Back to Holdout Groups & Long-term Impact Overview