Introduction
In March 2023, a fraud-scoring team at a mid-size payments processor pushed a retrained model straight to 100% of transaction traffic on a Friday afternoon. The new model had a better offline AUC — 0.94 versus 0.91. Nobody checked what that 3-point gain did to the false-positive rate on transactions over $2,000. By Monday, the support queue had 1,400 tickets from customers whose legitimate purchases got blocked, and the on-call engineer spent six hours rolling back a change that took four minutes to deploy. This is the exact failure canary releases and shadow mode exist to prevent.
Core Concept
Canary releases route a small, controlled slice of live traffic to a new model version while the rest keeps hitting the current one. You start at 1% or 5%, watch the metrics that matter — latency, error rate, business KPIs like approval rate or click-through — and ramp up in steps: 5% → 25% → 50% → 100%, usually over hours or days, not minutes. If anything degrades past a threshold, you roll back the percentage to zero instead of rolling back a deployment.
Shadow mode is a different animal. Every request still goes to the production model and that response is what the user gets. But the same request also gets duplicated — usually via an async fan-out — and sent to the candidate model in parallel. The candidate’s output never reaches the user. It gets logged, compared against the production model’s output, and scored on agreement rate, latency, and resource cost. Shadow mode carries zero user-facing risk, which is exactly why teams underuse canary and overuse shadow when they should combine both.
