Implementing the Canary Control Loop

The canary control loop is a closed feedback system that watches for new revisions, deploys canary instances, defines a traffic plan, evaluates metrics in rolling windows, and decides to promote or rollback automatically. A typical traffic plan starts at 5 to 10 percent canary weight, increases by 5 percent per step, caps at 50 percent, and runs health checks every 30 to 60 seconds. The controller selects a baseline for comparison, either the stable version receiving the remaining traffic or a dedicated baseline instance set with matched size and availability zone distribution to avoid cross zone latency bias. At each interval, the system queries metrics from both canary and baseline over a rolling window (last 3 to 5 intervals or last 2 to 5 minutes). It applies pass or fail logic to each guardrail: success rate must be at least 99 percent, P99 latency under 500 ms, error rate must not increase more than 50 percent versus baseline, CPU under 90 percent, memory under 95 percent, and business metrics like CTR must not drop more than 5 to 10 percent. If all checks pass for the majority of the window (for example, 3 out of 5 intervals), the controller increases canary weight by the step amount. If failed checks reach a threshold (5 to 10 consecutive failures or total failures), it immediately routes all traffic back to stable, scales down canary instances, and records the decision with telemetry and notifications. Promotion happens when canary reaches the cap (typically 50 percent) with all checks passing for a sustained period. The controller marks the canary as the new primary, routes 100 percent traffic to it, and scales down the old stable version. Rollback is idempotent: multiple rollback commands result in the same end state (0 percent canary, 100 percent stable). Tools like Flagger integrate with Kubernetes, Istio, and Prometheus to automate this loop declaratively. You define a Canary resource with thresholds, step size, interval, and metric queries, and Flagger manages the lifecycle. For ML serving, layer the rollout. Start with shadow mode to validate latency, resource usage, and prediction distributions with no user impact. Move to a small online canary at 5 percent, gate on fast guardrails (latency, error rate, CPU). Add slower moving model metrics (AUC drift, calibration error, CTR) in background analysis. Promote to 50 percent if all pass, then to 100 percent after final validation. Keep feature parity checks to ensure canary input schema matches stable, monitor for distribution shift in both feature inputs and prediction outputs, and maintain promotion and rollback actions as versioned, auditable policies in source control.

Flagger Canary resource defines step size 5 percent, interval 30 seconds, max weight 50 percent, thresholds for request success rate 99 percent and P99 latency 500 ms, integrates with Istio for traffic splitting and Prometheus for metric queries

Netflix Kayenta compares time series from canary and baseline, computes statistical scores for each metric, aggregates to overall pass or fail decision, triggers promotion or rollback via deployment API with full audit trail