A canary and shadow traffic both ask for something the chain has to be able to do - the second binary is the cheap half of either one.
👋 Hi, I am Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This block has been about the things a fleet has to do identically or every service reinvents them; this last part is about the moment after all of them: a change is built, and the only question left is what stands between it and everybody. Running notes live on my GitHub: github.com/brilliant-almazov.
This is how I do it right now, with the price attached - maybe you already do it better, maybe you have been through it, maybe you see it differently.
Where this sits
A short primer, so this reads on its own.
A PHP monolith is being taken apart into Go services. The Go API gateway is the only door into the new services from outside; behind it the extracted services talk to each other over gRPC, and every one of them is raised from the same shared platform library - which is where health endpoints, metrics, tracing and the messaging layer come from, rather than from each service writing its own. Some work is answered inside the call; the rest is published as a domain event and picked up later by a background handler in a separate process.
Two things follow from that shape and both matter below. The platform supplies the observation, so the question is never "do we have metrics" but "do the metrics separate the thing we are comparing". And the publish boundary is a real boundary: everything past it is a different process, a different tree in the trace, and - as it turns out - a different answer to whether a copy of a request is harmless.
What stands between a commit and everybody
Before any opinion about canaries, the inventory - counted rather than described.
-
Three pipelines. A build fires on a pushed tag
vX.Y.Zand stamps the version and the commit into the binaries. Tests run on every pull request, on every push to the main branch, and nightly on a schedule. Deployment is a manual pipeline run with two parameters: which version, and which services to move -server,worker,all. -
67 records in the machine-checked metrics snapshot: 59 of them come from the platform library, 6 from the service, plus one
<dynamic>entry. 13 domain metrics sit on top of that. Drift between the snapshot and reality fails CI. - Nine alert rules, described in text and created on the fleet monitoring side. The dashboard is assembled by a generator next to the rest of the fleet dashboards; neither dashboards nor alerts live as files in the service repository.
-
86.7 % line coverage, checked on every run against a ratchet whose threshold currently sits at
0. - Zero requests routed into a canary.
That last line is not an omission in the list. There is no routing of a marked request into a new version in production on my side, and the previous part is the reason: the mark has to reach the last hop, and mine would not have.
What "checking a release before everyone" actually means
Two techniques, and it is worth writing them down in one line each, because they are routinely discussed as if they were the same thing.
A canary. Part of the live traffic is sent to the new version while everybody else stays on the old one, and the two are compared on the metrics that matter.
Shadow traffic. A copy of the live traffic is sent to the new version, and the answer the user receives still comes from the old one. Nothing the new version produces is shown to anybody.
Both answer exactly one question: does the new version behave the same way on real input. Neither of them answers it by being deployed. They answer it by being observable, which is a different requirement and a more expensive one.
The tempting reading is that the hard part is running two versions at once. It is not - on a deploy that already takes a version as a parameter, standing the second one up costs one more pipeline run. The hard part is everything the two techniques assume about the chain around them, and that is what the rest of this is about.
How this is usually done
Four shapes come up in public write-ups and standards, and all four are reasonable: a gradually increasing share of traffic with the failure rate watched as the dial moves (CanaryRelease); mirroring requests into a second, non-answering path; two colours side by side, where a whole second environment is raised and traffic is switched to it at once (BlueGreenDeployment); and a sandbox on a copy of the environment, exercised with no live traffic at all.
The previous part compared those four by blast radius - how much of the world meets one mistake. This part compares them along a different axis, and it is the axis that decides whether you can have them at all: what each one demands from the chain and from the data. I am naming no gateways, no service meshes and no tracing products; the choice of product is not what makes any of this work or fail.
What each way needs
Three rows. The last one is not a technique, it is what is actually there.
| Way | What has to work | What it costs | Have it |
|---|---|---|---|
| canary | mark reaches the last hop; two generations, one set of rows | a share of traffic; a version cut in metrics | no |
| shadow traffic | a copy with no side effects; writes split from reads | reads served twice; a separate write path | no |
| what I have | executable checks before, metrics and alerts after | a defect found after everyone met it | yes |
Read the first row from the right: the two things a canary needs before it is worth anything are a marked request that survives every hop and metrics that can tell the generations apart. Neither of them is the second binary.
There are no numbers in that table on purpose. The share of canary traffic, the observation window, the rollback threshold, how long a rollout takes - none of those has been measured on my side, and a plausible figure would be worse than a visible hole.
The canary runs into the wall from the previous part
Short, because that part already did the work. A canary needs the mark on the request to survive every hop; if it stops after the first one, the first service runs the new code and everything it calls afterwards runs the old one - one user action executed by two generations, with each half looking correct from where it stands. It needs the contract to be compatible in both directions, and data written by the new version to be readable by the old one. It needs whoever is testing to know which branch they are in, or the debugging is done blind.
None of that is technical difficulty. It is two generations of a system being true at the same time, and people holding both in their heads while they work.
There is a second wall, and it is the quieter one: there is nothing to compare the generations by. The tenant identifier is deliberately kept out of metric labels for cardinality reasons, and there is no version dimension in the snapshot today either. A canary without metrics that separate the two versions is not a controlled experiment - it is a rollout on live people with someone watching a graph that cannot tell them apart.
Where a shadow contour stops being free
Shadow traffic looks like the safe one of the two, because nothing it produces reaches a user. That holds exactly as long as the copied request does not write anything.
The facts on my side that decide it:
- delivery is at-least-once, and the broker does not deduplicate;
- a redelivery carrying the same
message_idis normal behaviour, not an error - the incident is a spike in redeliveries, not their existence; - the normal state of the outgoing queue is empty, which is what makes a spike legible at all.
So a mirrored request that gets as far as publishing does not produce a shadow message. It produces a real message, on the real stream, for the real consumers, indistinguishable from one a user asked for.
Which gives the rule in one line: a shadow contour is meaningful exactly up to the point where writing starts. Past that point it needs a separate path for every write, and building that path costs about what building the new version cost. That is the whole reason shadow traffic is rare in systems that publish events, and common in systems that mostly read.
What actually checks a release here
With both techniques unavailable, the honest question is what is left - and the answer is a list of things that fail the build rather than a list of intentions.
Before the deploy, the test pipeline runs, in this order:
build
linter
bring up the shared test containers one script, one container per type, per run
run the tests with coverage
check the time budget a separate step, not a note in the log
tear the containers down if: always()
coverage report
threshold check
ratchet raising it is allowed, lowering fails the PR
There is one container of each infrastructure type for the whole run - isolation between tests is done with schemas and data, never by raising another container - and the teardown is explicit rather than left to a reaper. The time budget is a step of its own for the same reason everything else here is: a run that quietly got three times slower is a defect, and a defect that fails nothing is a defect nobody will look at.
Nightly, a separate job runs the race detector and the volume tests with -p 1, so that a comparison of ns/op between runs means something.
Two catalogues fail the build on drift. The environment variable catalogue - 59 variables, 45 from the platform and 14 from the service - and the metrics snapshot above. Both are machine-checked artefacts, not documents somebody is supposed to update.
One check exists purely because of a silent failure mode. The version is stamped into the binary with a linker flag, and the linker silently ignores a flag aimed at a symbol that does not exist: the build stays green and the version stays dev - in the audit trail, in the trace attribute, everywhere. So the build target compiles both binaries with a test tag and greps the binary for it. Empty output means the symbol is wrong.
After the deploy, /health and /ready are served by the platform, and there is no "wait for health" step in the pipeline - that check belongs to the environment. Then the metrics and the nine alert rules.
The case: the ratchet is wired, the bar is not set
This is the one I would fix first, and it is small enough to state in three lines.
coverage 0 % 86.7 % 100 %
|------------------------------------|-----------------|
threshold ^ 0 ^ actual
the ratchet can only move right - and it starts here
What is there. A coverage ratchet: the threshold can be raised, never lowered, and a pull request that tries to lower it fails.
What it costs today. The threshold sits at 0 while actual coverage is 86.7 %. So the mechanism is wired and works in one direction only - it cannot be walked back - but the bar it is holding is on the floor. Lowering coverage is impossible; shipping a release with coverage far below 86.7 % is not, and nothing in the pipeline would notice.
What is holding form next to it. The two drift checks and the container discipline above are the same idea done properly: an artefact that fails the build, not a rule somebody is meant to remember.
The reading that generalises: a release here is checked by executable checks before the deploy and metrics after it. A canary is not in that row, and neither is anything else that depends on a marked request travelling further than the first hop.
Why it is arranged this way
Because a check you can miss is not a check. The things listed above all share one property: they fail something. The linter fails, the threshold fails, the snapshot drift fails, the ratchet fails a pull request. That is worth more per hour spent than any technique whose failure mode is "nobody looked at the dashboard". So the order is: ratchet and prohibiting tests first, canary second.
Because the observation is not there yet. With no way to separate the two generations in the metrics, a canary would produce a feeling rather than a comparison. Building the version cut is the prerequisite, not the follow-up.
What it costs
Stated plainly, because the arrangement above is a trade, not a victory.
- A defect that tests did not catch is found after everyone has met it. The blast radius of a mistake is the entire traffic. That is the price of the whole section above, and it is not a small one.
-
The ratchet holds nothing at the moment.
0against 86.7 % - wired, one-directional, and set on the floor. - The nightly signal arrives a day late. Races and volume are checked once a night, not per pull request, so a race introduced in the morning is a tomorrow problem.
- One check exists only because a tool fails silently, and silent failures of that shape are almost certainly not down to one.
- A human is part of the deploy procedure. A manual run with two parameters means somebody picks the version and the service set, and that choice is checked by nothing.
When not to do this
- When traffic is small, a canary gives you no statistics. A share of a small number is a smaller number, and a comparison on it is noise wearing a lab coat.
- When writes are not separated from reads, a shadow contour is dangerous, not cautious. The copy stops being a copy at the first publish.
- When the mark does not reach the last hop, a canary checks the first step and lies about the rest - and a half-migrated request is worse than no rollout mechanism at all.
And the boundary of this write-up, said outright: there is no canary and no shadow contour here, there is no routing of a marked request in production, there are no declared SLAs and no formal SLO, and the coverage bar is not set. Everything above is a breakdown of what these techniques require and what they cost, plus an inventory of what is holding the line instead - not a report on a working scheme.
The multiplier line
An automated executor makes the mechanical half of this cheap: wiring the threshold check, writing the teardown step, generating the snapshot, adding the version cut to a metric. What it does not do is decide that a check which cannot fail the build is not a check, or notice that a mirrored request stops being a copy at the first publish. Those are judgements, and they are the whole content of this article. Speed amplifies whoever set the definitions; it does not supply them.
Access, tracing, rollout - Part 6. That closes this block: who arrived, what they may do, where their request has been, what decides a branch, how one request reaches a new version, and - here - what checks that version when nothing routes traffic to it yet.
If you do this better, tell me what your release has to pass before it reaches everybody, and which of those checks can actually fail a build. If you have been through this, what did your shadow copy turn out to write that nobody expected? If you see it differently, say where shipping to everyone at once and watching the graphs is the honest answer. How is it solved on your side, and what broke there?
Top comments (0)