DEV Community

Mikhail Dorokhovich
Mikhail Dorokhovich

Posted on

Blue-Green vs Canary Deployment: The Principle Is to Stop Choosing and Start Combining

The problem in context

The releases that hurt most are the ones where a team has to decide, live, whether things are bad enough to roll back. Error rate looks a little high — is that the new version or normal noise? Nobody agreed on a threshold in advance, so the argument happens in the incident channel while users suffer. That single failure mode — rollback as a live debate — is what a deployment strategy exists to prevent, and it is why the perennial "blue-green vs canary deployment" argument is usually the wrong frame.

Treated as a ranking problem, it produces a stalemate: blue-green people cite instant rollback, canary people cite limited blast radius, and the team picks one and inherits the other's weaknesses. The reframe is that these are not rivals to rank. They are tools that answer three different questions — how do we roll out, how do we roll back, how do we limit blast radius — and a real delivery system needs a different answer at different blast radii.

The principle

The principle here is that deployment strategy is blast-radius management, and different releases have different blast radii. Once you accept that, the design stops being "pick the one right strategy" and becomes "assemble a layered system where each tool covers the radius it is best at." A combination almost always wins: trunk-based development plus feature flags plus canary for small daily releases, and blue-green for the big drops where instant, explainable rollback is the whole point. A detailed treatment of choosing and combining strategies walks the full model; the compressed version is four tools with four jobs.

Blue-green keeps two identical production environments — the pattern Martin Fowler documented in 2010. Blue serves users; green is prepared calmly for the next release; when you are confident you flip traffic in one router change, and if something is wrong you flip back just as fast. Its virtue is that rollback is instant and trivial to explain to a business stakeholder — a fintech spotting a EUR-payments bug and flipping back in about two minutes, with a tiny fraction of operations affected, is the kind of story that sells it to leadership.

Canary rolls a change out to a small subset first — what Danilo Sato describes on Fowler's site — starting at 1–5% of traffic, watching errors, latency, resource use, and one business metric, then ramping 5 → 10 → 25 → 50 → 100% and comparing at each step. Its virtue is a naturally small blast radius: a bad version is seen by few before the system reacts.

Rolling updates are the quiet workhorse for stateless services — the default strategy in the Kubernetes docs, where maxSurge and maxUnavailable govern how aggressively pods are replaced while the service stays available throughout.

Feature flags are the one that changes cadence most, because they separate two actions people wrongly treat as one: deploying code and releasing a feature. Ship dark, enable for staff, then a percentage, then everyone, and kill the switch without redeploying — the technique Pete Hodgson documents as feature toggles, which pairs naturally with trunk-based development.

The unifying move that ends the 2 AM arguments is writing stop-criteria before the release, as executable rules rather than live judgement:

IF error_rate_canary > error_rate_baseline * 1.5 THEN rollback
IF latency_p99_canary > latency_p99_baseline * 1.3 THEN rollback
IF conversion_rate_canary < baseline * 0.95 THEN rollback
Enter fullscreen mode Exit fullscreen mode

With those in place, rollback stops being a debate and becomes a reflex — the system pulls the new version before a human opens the dashboard. Netflix's canonical case is exactly this: a gradual ramp where SmartTV performance degraded at one step and triggered an automatic rollback in roughly two minutes.

Trade-offs

No strategy is free, and the honest way to reason is per blast radius:

Strategy Best for Rollback Real cost
Blue-green Big, risky drops (payments rewrite, framework upgrade) Instant single flip ~2× resources during release; DB and session state are hard; watch DNS TTL
Canary Everyday releases Automatic on stop-criteria Needs percentage routing; slow to reach confidence on low traffic; pin users by ID
Rolling update Stateless services on Kubernetes Gradual, batch by batch Old and new pods coexist — API contracts must stay compatible across versions
Feature flags Decoupling deploy from release Kill switch, no redeploy Forgotten flags rot; every flag needs an owner and an expiry

Two constraints deserve emphasis because they are where teams get burned. First, the rolling-update coexistence problem: because old and new pods serve traffic simultaneously, forward-compatible database and API changes are not optional — deployment strategy and schema evolution are two halves of one discipline. Second, a flag with no owner is technical debt with a fuse on it; the discipline that makes flags safe is an owner and an expiry date on every one, with dead code removed once the feature is universal.

And one distinction worth nailing down, because conflating them wastes time: canary is not A/B testing. Canary asks "is the new version not worse?" and optimizes for safety. A/B testing asks "which version is better?" and optimizes for a product decision — split traffic, pin each user to a variant, and wait a week or two to smooth out weekday and seasonality effects before running the stats. Use canary to protect releases and A/B tests to choose product directions; using one for the other's job produces noisy, untrustworthy conclusions.

How to adopt

The mistake is adopting all four at once. Sequence by where the pain is loudest.

  1. Write the stop-criteria before the next release and make them executable. This is the single highest-leverage move; it converts rollback from a live argument into an automatic reaction and costs almost nothing.
  2. Add feature flags to decouple deploy from release. Ship dark, enable gradually, keep a kill switch. Put an owner and expiry on every flag from day one so the debt never accumulates.
  3. Make rolling updates safe on Kubernetes with small batches, health probes, and a pause between batches: 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 2
  template:
    spec:
      containers:
      - name: my-app
        image: my-app:v2
        readinessProbe:
          httpGet: { path: /health/ready, port: 8080 }
        livenessProbe:
          httpGet: { path: /health, port: 8080 }
Enter fullscreen mode Exit fullscreen mode

Batches too large increase risk and make rollback painful; skipping the pause means you notice problems too late.

  1. Reserve blue-green for the big drops where instant, explainable rollback justifies double the resources. Solve the session problem with a shared session store and graceful connection draining, and watch DNS TTL, which can delay a switch you thought was instant.
  2. Automate the canary gates rather than hand-rolling them. Argo Rollouts encodes exactly these gates — setWeight and pause steps plus automated analysis that advances the traffic weight only while metrics stay healthy.

Put together, a normal day looks like this: a change merges to trunk, deploys dark behind a flag, and enables as a canary to 5% while the automated gates watch; if they stay green it ramps to 100% over an hour, and if not the flag flips off while the code stays deployed but inert. A large, risky drop reaches for blue-green instead. Each tool covers a different blast radius, and together they mean no release requires heroics.

Where this goes next

The direction of travel is toward progressive delivery as a fully automated control loop — the stop-criteria you write by hand today become the analysis templates your rollout controller evaluates on its own, promoting or reverting without a human in the path. Tools like Argo Rollouts and Flagger are early forms of this; the interesting frontier is richer signals feeding the gate, including model-based anomaly detection that can catch a regression no static threshold would.

The deeper point is that all of this rests on the same foundation: a legible definition of "healthy" and forward-compatible changes underneath the traffic shifting. Teams that have captured what "not worse" means as executable criteria are the ones who will safely hand more of the promote/rollback decision to automation — and eventually to AI-assisted release agents that reason over the same signals. The layered strategy is not just what ends the 2 AM rollbacks today; it is the substrate the automated release systems of the next few years will need in order to be trusted at all.

Sources & further reading

Top comments (0)