Week 3. "The artifact is sitting in the registry. Somebody still has to actually run it. Who, and how?"
Previously
Developer
↓
Runner
↓
Cache
↓
Artifact
Today
↓
Deployment
Junior Engineer: So the image is built, tagged, sitting in the registry. What's the simplest possible way to actually get it running in production?
Senior Engineer: The simplest way is also the most dangerous way. Want to guess it first?
Junior Engineer: Stop the old container. Start the new one.
Senior Engineer: That's it. That's genuinely how a lot of side projects deploy. Now — what's wrong with it?
Junior Engineer: There's a gap, right? Between "stop old" and "start new," nothing's running.
Senior Engineer: Exactly. That gap has a name — downtime. Might be half a second. Might be twenty, if the new container is slow to boot, run migrations, or warm up a cache. Either way, every request that lands in that gap fails. For a side project, nobody notices. For a payment API, that gap is real money and real angry users.
📝 Production Note
The naive "stop old, start new" approach is technically a valid deployment strategy — it's just usually the wrong one. It has a name too: recreate. Some internal tools genuinely use it on purpose, because a few seconds of downtime for an admin dashboard nobody's actively using at 3 AM is a fair trade for simplicity. The mistake isn't using it — it's using it without deciding to.
The Real Question Underneath
Senior Engineer: So the actual engineering question isn't "how do I run the new version." It's: can old and new both be running at the same time, safely, long enough for the switch to happen with zero gap?
Junior Engineer: Both running at once sounds messy. Wouldn't they conflict?
Senior Engineer: Only if you let them share something they shouldn't — like both writing to the same in-memory cache in incompatible ways. Mostly, though, this is exactly what modern deployment strategies are built around: overlap on purpose, then cut over cleanly.
Blue-Green Deployment
Senior Engineer: First strategy: blue-green. You keep two full production environments — call them Blue and Green. Right now, Blue is live, serving all real traffic. Green sits idle.
To deploy, you start the new version on Green. Nobody's routing traffic there yet — it's just running, warming up, ready to be checked. You hit its health endpoint, run a smoke test, maybe watch its logs for a minute.
Junior Engineer: And once you're confident it's healthy?
Senior Engineer: You flip the router. All traffic that used to go to Blue now goes to Green. Instantly — not gradually. One moment Blue is live, the next moment Green is.
Junior Engineer: And Blue?
Senior Engineer: Stays up, idle, doing nothing. On purpose. If something's wrong with Green that only shows up under real traffic, you flip the router back to Blue. Instantly. No rebuild, no redeploy — Blue was never touched.
👦 Junior Framing / 👨🦳 Senior Framing
Junior framing: "Rollback means redeploying the old version."
Senior framing: Rollback means flipping a router back to something that was already running the whole time. The fastest rollback is one where you never had to deploy anything to perform it.
Rolling Deployment
Junior Engineer: Blue-green needs double the infrastructure, though — two full environments, most of it sitting idle. That seems expensive for a small team.
Senior Engineer: It is, and that's exactly why most teams don't run pure blue-green. More common: rolling deployment.
Say you have ten instances of your app running behind a load balancer. Rolling deployment replaces them a few at a time — stop one old instance, start one new instance, wait until it's healthy, move to the next. Two or three at a time, not all at once.
Junior Engineer: So at any given moment during the rollout, some requests hit the old version and some hit the new one?
Senior Engineer: Exactly. For a minute or two, both versions are genuinely serving real production traffic simultaneously. That's usually fine — but it's also exactly why "both versions must be able to coexist safely" isn't a throwaway line. If the new version writes data in a shape the old version can't read, rolling deployment will surface that mid-rollout, in production, with real users.
Junior Engineer: That sounds like a real risk, not a theoretical one.
Senior Engineer: It's one of the most common causes of a "weird, intermittent bug for about ninety seconds" report — someone hit an old instance, someone else hit a new one, and the two disagreed about something.
📒 Senior Engineer's Notebook
A rolling deployment isn't "old version, then new version." For a window in the middle, it's both — genuinely, simultaneously, in production. Any deployment strategy that overlaps old and new is really asking one question: are these two versions allowed to be strangers to each other for a few minutes?
Canary Deployment
Junior Engineer: Is there something more cautious than rolling? What if I don't trust the new version at all yet?
Senior Engineer: That's canary deployment. Instead of replacing instances in bulk, you send the new version a small slice of real traffic first — maybe 5%. Everyone else stays on the old version.
Junior Engineer: Why 5% specifically?
Senior Engineer: No fixed number — depends on your traffic volume and risk tolerance. The idea is: expose the new version to real conditions, real users, real edge cases your tests never thought to write — but limit the blast radius if something's wrong. If 5% of users hit a bug, that's bad. It's a much smaller bad than 100% of users hitting it.
You watch error rates, latency, whatever your metrics track, on that 5% specifically. Looks healthy — increase the slice. 5%, then 25%, then 100%. Looks unhealthy at any point — send that slice back to the old version and stop the rollout.
Junior Engineer: So canary is basically a controlled experiment running in production.
Senior Engineer: That's precisely what it is. The name comes from coal mining — a canary in the mine would show signs of trouble from bad air before a miner did. Same idea: let a small, contained group hit trouble first, before it reaches everyone.
🎯 Interview Perspective
Interviewer: When would you choose canary over blue-green?
Weak answer: Canary is safer, so always use canary.
Strong answer: Blue-green gives you instant rollback but costs double the infrastructure and validates the new version with synthetic checks, not real traffic diversity. Canary costs less infrastructure and exposes the new version to genuinely real, varied traffic — but a rollback means shifting traffic back gradually, and any bug still affects some real users before it's caught. Blue-green suits changes you're fairly confident in but want an instant undo for. Canary suits changes you're genuinely unsure about and want real signal on before committing.
Health Checks — The Thing That Makes All of This Safe
Junior Engineer: Every strategy you've described leans on "check it's healthy" before shifting more traffic. How does that check actually work?
Senior Engineer: A health check — usually a specific endpoint, something like /healthz, that the new instance exposes. It doesn't just return "yes I'm running." A container can be running and still be broken — maybe it started but couldn't connect to the database.
A real health check verifies the things that would make the instance actually useless if they failed: can it reach the database, can it reach dependent services, has startup fully finished. Only once that endpoint returns healthy does the deployment system consider the instance eligible to receive real traffic.
Junior Engineer: So without a health check, all of this — blue-green, rolling, canary — is just... hoping?
Senior Engineer: Precisely. The strategy decides how much traffic shifts and when. The health check decides whether it's actually safe to shift any at all. Remove the health check and you've built an elaborate system for exposing users to a broken deploy slightly more gradually — which isn't actually much of a win.
🚨 Beginner Mistakes
"I'll skip the health check, the container starting is proof enough." A container can start successfully and still be completely unable to serve a real request. "Started" and "ready" are different claims.
"I'll shift 100% of traffic to canary if the first ten requests look fine." Ten requests aren't a sample size. Real problems — memory leaks, rare edge cases, load-dependent bugs — often only appear after sustained, varied traffic.
"I'll test the new version by deploying straight to production and watching what happens." That's canary deployment without the safety net — no controlled slice, no automatic rollback path, no separation between "found a problem" and "everyone already hit the problem."
Who Gets to Flip the Switch
Junior Engineer: All of this is mechanics. But somebody has to actually decide "yes, ship it." Who?
Senior Engineer: Depends entirely on the company — same spectrum we talked about back in Episode 1.
At a small startup — usually, whoever pushed the code. Pipeline goes green, they deploy, no one else involved.
At a growing company — a required approval step. Someone else has to click "approve" in the pipeline before it proceeds to production, even if the pipeline itself passed everything.
At a bank — multiple named approvers, a change ticket, a scheduled deployment window, and a full audit log of exactly who approved what and when. Not because engineers there are less trusted — because the cost of an unapproved, unreviewed production change is categorically higher when real money moves through the system.
Junior Engineer: So "who can deploy" isn't a technical question at all.
Senior Engineer: It's a risk question wearing technical clothes. The pipeline enforces whatever answer the company has already decided on.
🏢 Office Reality
- "Blue is live." In a blue-green setup, this tells you which environment is currently receiving real traffic — useful shorthand nobody has to explain twice.
- "The canary is unhealthy." The small traffic slice on the new version is showing errors or bad metrics. Rollout should stop, not continue "to see if it improves."
- "We're doing a cutover at 2 AM." A full traffic switch, scheduled for low-traffic hours specifically because it's still considered risky enough to want minimal witnesses.
- "Freeze the deploys." No new deployments allowed for a period — common right before a major event (a sale, a product launch) when the cost of something breaking is unusually high.
A Canary That Did Its Job
Junior Engineer: Has canary actually caught something real, in your experience — or is it mostly theoretical safety?
Senior Engineer: Very real, once. A new version of a checkout service went out to 5% of traffic. Nothing dramatic — error rate ticked up, barely. Would've been easy to dismiss as noise.
Someone didn't dismiss it. Turned out the new version had a bug that only triggered for a specific payment method, used by a small fraction of users. If that had gone to 100% immediately, it would've meant every user with that payment method, all day, failing at checkout — probably discovered from a spike in support tickets, an hour later, the hard way.
Junior Engineer: Instead it was five percent of five percent of traffic, for a few minutes.
Senior Engineer: And a rollback that took seconds, because the other 95% was never touched in the first place. That's the entire value of canary in one sentence — not that it prevents bugs, but that it shrinks the blast radius of the ones you didn't catch.
🪞 If I asked you this in an interview
"Walk me through what happens, end to end, when a canary deployment detects a problem."
A small percentage of traffic is routed to the new version while the rest stays on the old one. Metrics — error rate, latency, whatever the team monitors — are compared between the two. If the new version's metrics degrade beyond a threshold, the deployment system automatically routes that traffic slice back to the old version and halts the rollout, rather than increasing the percentage further. Because only a small slice was ever exposed, the blast radius of the bug is limited to that slice, for that window of time.
🎤 Explain It In One Minute
Imagine explaining this to a teammate — without using the words blue-green, canary, or rolling.
Why would a company deliberately send its own new code to only a small fraction of real users, instead of just launching it for everyone at once?
If you can answer that without the vocabulary, you understand the reasoning — not just the pattern names.
Whiteboard Moment
Artifact in registry (tagged, immutable)
↓
Deployment approved (who's allowed depends on the company)
↓
New version starts on target infrastructure
↓
Health check confirms it's actually ready — not just running
↓
Traffic shifts according to strategy:
Recreate → all at once, brief gap
Blue-Green → instant full switch, old stays warm as fallback
Rolling → a few instances at a time, both versions coexist briefly
Canary → small % first, expand gradually if healthy
↓
Metrics watched throughout
↓
Healthy → rollout continues to 100%
Unhealthy → traffic reverts, rollout halts
Junior Engineer: So the pattern across every strategy is the same three questions, just answered differently: how much traffic moves, how fast, and what happens if it goes wrong.
Senior Engineer: That's the whole subject, compressed into one sentence. Everything else is implementation detail.
What You Should Be Able to Explain Now
(Without looking at Google)
Can you explain:
- Why "stop old, start new" causes downtime, and when that tradeoff is actually acceptable?
- What blue-green buys you, and what it costs?
- Why rolling deployments mean two versions are briefly live together — and why that matters?
- Why canary deployments limit blast radius instead of preventing bugs?
- Why a health check is what makes every one of these strategies actually safe?
- Why "who's allowed to deploy" is a risk decision, not a technical one?
If yes — you understand deployment the way it's actually practiced, not just the vocabulary used to describe it in a job posting.
Junior Engineer: Okay. So say the rollout finishes. 100% of traffic, new version, all healthy. Are we done?
Senior Engineer: For about as long as it takes someone to stop watching the dashboard.
Junior Engineer: Meaning?
Senior Engineer: Meaning "the deploy succeeded" and "the deploy is fine" are, once again, two different claims — same as "pipeline is green" back in Episode 1. A deploy can look perfect for twenty minutes and then start leaking memory. Or degrading under real end-of-day traffic that your canary window never saw.
Junior Engineer: So something has to keep watching, after the deploy is technically over.
Senior Engineer: Now you're standing right at the edge of the whiteboard we drew all the way back in Episode 1. Deployment. Then — Monitoring.
(End of Episode 5)
Top comments (0)