Published
Canary Testing: What It Is, How It Works and When to Use It
Short answer
Canary testing releases a new version to a small slice of production traffic first, compares its metrics with the version already running, and only then widens the rollout. If error rate or latency gets worse, traffic goes back to the old version. It catches defects that pre-production tests miss, because it uses real users and real data, while limiting how many users a bad release can reach.
Last updated: 2026-11-14
What is canary testing?
Martin Fowler defines a canary release as a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure. The name comes from miners who carried a canary into a coal mine as an early warning of toxic gas.
"Canary testing" and "canary release" are used interchangeably in most teams. The release is the mechanism and the testing is the observation: you watch the canary and decide whether it is healthy. It is a form of testing in production, so it sits after your unit, integration and end-to-end tests, not in place of them.
Google's SRE workbook gives the reason it exists. By the time a release is ready to deploy, test environments are not 100% identical to production and tests do not cover 100% of scenarios, so some release-related defects only show up when real traffic hits the service.
How does a canary release work?
The steps are the same across tools:
- Deploy the new version next to the current one. The current version is the stable or control group.
- Route a small share of traffic to the new version, the canary. Google's SRE workbook describes sending a small portion of traffic to the canary segment and comparing it against the control.
- Compare metrics for the canary and the control over a fixed window.
- If the canary looks healthy, raise its share in steps. If not, send all traffic back to the stable version.
- At 100%, promote the canary to be the new stable version and retire the old one.
Fowler notes that the rollback strategy is simply to reroute users back to the old version until you have fixed the problem. Traffic can be split by weight at a load balancer or proxy, by region, by brand, or by user group. Google's workbook lists backend weights on a load balancer, proxy configurations and round-robin DNS as options.
What metrics should a canary watch?
The SRE workbook is direct about this: metrics should indicate problems, and each one needs a definition of acceptable behavior. Too strict and you get false positives. Too loose and a bad canary goes unnoticed. For a typical web service, it says the best signals are usually HTTP return codes and response latency, because their degradation most closely maps to a problem that users feel.
Three details from that guidance are easy to miss:
- Break metrics down by population. If a canary serving 5% of traffic fails 20% of requests, the whole service shows only about a 1% error rate. You need the canary and control error rates side by side.
- Match the metric window to the canary duration. A metric computed per hour will give a muddled signal on a 30-minute canary, because errors from before the canary leak into the comparison.
- Exclude noise you cannot attribute. A noisy neighbor on the same machine is a real problem, but not one caused by the change being canaried. The workbook suggests excluding 400-level codes and adding black-box checks for a particular URL, so odd user behavior does not decide the result.
Fowler adds that you can monitor business metrics too, and roll back automatically on a statistically significant regression, an approach he calls a cluster immune system.
What does automated canary analysis look like?
Progressive delivery controllers automate the loop. Two common Kubernetes examples:
Argo Rollouts lets you list steps. The basic ones are setWeight, which sets the percentage of traffic sent to the canary, and pause. If a pause has no duration, the rollout waits until someone removes the pause condition. Its docs say there is no agreed standard for a canary deployment, so the controller lets you define your own. An analysis block runs in the background, and if the analysis fails, the rollout is aborted. Without a traffic router, Argo Rollouts approximates the weight by scaling replicas: 10% of 10 replicas is 1 canary pod and 9 stable pods.
Flagger runs a control loop that shifts traffic gradually while measuring HTTP request success rate, request duration and pod health. Its documented example uses a 1 minute interval, a 2% step weight, a 50% max weight and a threshold of 10 failed checks. The docs give the time to promote as interval * (maxWeight / stepWeight), which is 25 minutes for that setup, and the time to roll back a failing canary as interval * threshold. Flagger can also call webhooks, such as conformance or load tests, and halt advancement if a webhook fails.
You can build the same loop without Kubernetes. A deploy script that shifts weight at a load balancer, reads error rate and p99 latency from your monitoring system after each step, and reverts on a threshold breach is a canary.
Canary vs blue-green vs A/B testing
| Canary | Blue-green | A/B test | |
|---|---|---|---|
| Purpose | Detect regressions before full rollout | Switch versions with fast rollback | Test a hypothesis between variants |
| Traffic | Gradual: 1%, 5%, 25%, 100% | All at once, after the new stack is ready | Split for the length of the experiment |
| Rollback | Shift traffic back | Flip the router back | Stop the experiment |
| Cost | Small extra capacity | Roughly double the resources, per the SRE workbook | Depends on the experiment |
| Duration | Minutes to hours | Minutes | Often days |
Fowler says canary releases can be used to implement A/B testing because the technical implementation is similar, but recommends not conflating the two. A canary looks for problems and should finish in minutes or hours, while an A/B test can take days to reach statistical significance, and mixing them can interfere with the results.
When is canary testing a poor fit?
Fowler lists the limits. You have to manage multiple versions of your software at once, so keep the number of concurrent versions low. It is hard when the software is installed on users' computers or phones, because you do not control when they upgrade. Database changes need care: his suggestion is parallel change, so the schema supports both versions during the rollout.
Low-traffic services have a statistical problem too. A canary at 5% of 40 requests an hour will not give you a trustworthy error rate in 30 minutes. In that case, use a longer window, a larger share, or synthetic checks that run against the canary directly.
What does a canary not catch?
A canary reports on metrics. It catches crashes, error spikes and latency regressions. It does not tell you that the discount calculation is wrong while returning HTTP 200, or that a flow nobody visited in the canary window is broken.
That is the job of tests before the release. Keep a set of scenarios for the flows that matter, run them against the canary pods or a staging environment, and make sure you can see which scenarios have no test. If a coding agent wrote the change, the agent may also have written the tests that check it, so a human-owned scenario list matters more. TestChimp keeps scenarios as Markdown in your repo, links Playwright tests to them and shows the gaps. See test coverage metrics and GitHub pull request checks for how to gate merges before a canary ever starts.
Frequently asked questions
What is canary testing?
Canary testing is releasing a new version to a small subset of production traffic or users first, comparing its metrics with the stable version, and widening the rollout only if it looks healthy. If it does not, traffic is routed back to the old version.
What is the difference between canary and blue-green deployment?
A canary shifts traffic gradually, so a defect reaches only a small share of users. Blue-green keeps two full environments and moves all traffic at once, which makes rollback a router flip but uses roughly twice the resources.
How long should a canary run?
Long enough to gather a meaningful signal at the traffic share you chose. Fowler says a canary rollout should complete in minutes or hours, not days. Google advises matching metric windows to the canary duration.
Which metrics should I watch in a canary?
Start with HTTP error rate and response latency, compared between the canary and the control group, since those most directly reflect user impact. Add business metrics if you can monitor them reliably.
Is canary testing the same as A/B testing?
No. The traffic splitting looks similar, but a canary checks for regressions and should finish quickly, while an A/B test tests a hypothesis and may take days to reach significance. Fowler recommends keeping them separate.
Which tools automate canary releases?
On Kubernetes, Argo Rollouts and Flagger shift traffic in steps and abort or roll back on failed metric checks. You can also script the same loop with a load balancer and your monitoring system.
Catch the bugs a canary cannot see
Keep scenarios in your repo, link tests to them and see which have none. 14-day free trial, no credit card.