octave Nkurunziza

Posted on Oct 18 Originally published at blog.hubfly.space

Canary releases with a weighted load balancer: 10% to v2, proven with 200 requests

#devops #docker #loadbalancing #tutorial

TL;DR

What you get: a canary release that sends exactly 1 in 10 requests to a new version, a 7-line script that proves the split, and a rollback that takes effect in under a second.

Who it's for: anyone running plain containers (no Kubernetes, no service mesh) who wants to ship v2 to a slice of users first.

Time: about 15 minutes.

You've got v2 ready. The tests pass. Your stomach still says "not for everyone at once."

A canary release fixes that: v2 gets a small slice of real traffic, you watch it, then you turn the dial up or pull it out. Setting a 90/10 split takes a minute. Checking that you actually got one is the part people skip, and a 90/10 setting can quietly give you something else.

How a weighted canary works

A load balancer sits in front of two backends. Each backend gets a weight. With weights 9 and 1, a weighted round robin sends 9 requests to v1, then 1 to v2, and repeats.

           ┌──────────────────────┐
Internet → │  load balancer (WRR) │ ──9──→ app-v1 (stable)
           │  canary-router       │ ──1──→ app-v2 (canary)
           └──────────────────────┘

Three things decide whether this works in practice:

The strategy must actually read the weights. Some strategies ignore them.
Sticky strategies change who sees v2. IP hashing pins each client to one backend, so a 10% canary means "10% of clients", and you may never be one of them.
Rollback must be one action. If pulling v2 out takes a redeploy, it isn't a canary, it's a hope.

Build it on Hubfly, then check all three.

Step 1: deploy v1 and v2 side by side

Here, both versions are hashicorp/http-echo, a tiny server that replies with a fixed string. That makes counting trivial: every response says v1 or v2. In real life, these are your two image tags.

services:
  app-v1:
    image: hashicorp/http-echo:1.0
    command: ["-listen=:5678", "-text=v1"]
  app-v2:
    image: hashicorp/http-echo:1.0
    command: ["-listen=:5678", "-text=v2"]

In the Hubfly dashboard: open your project, click Create → Compose Stack, open the YAML tab, paste the file, click Import YAML, then Deploy Stack. Neither service publishes a port. The load balancer will be the only way in.

The stack is up in about 33 seconds.

Step 2: create the load balancer

Click Create → Create Load Balancer. The editor has three columns: ingress domains on the left, the balancer in the middle, backend targets on the right.

Balancer name: canary-router.
Backend targets → Add Target Container: pick app-v1, set Port to 5678, Weight to 9, click Add Target. Repeat for app-v2 with weight 1.
Ingress domains → Add Domain → Hubfly Free → Attach Free Domain. You get an HTTPS subdomain like canary-router-xx-xxx.eu1.hubfly.app.
Strategy: choose Weighted Round Robin. Not "Round Robin". This is the step the rest of this post is about.
Click Deploy Load Balancer.

Watch the port: the target form defaults to 3000. This app listens on 5678, so leaving the default points the balancer at a closed port.

Hubfly's load balancer docs describe three strategies (round_robin, least_conn, ip_hash) and a canary recipe with weight: 9 and weight: 1. The dashboard offers eight. Only the weighted strategies apply your weights.

Step 3: prove the split (don't trust the percentages)

Open your load balancer's domain in a browser tab. It shows v1 or v2. Then open the browser console (F12 → Console) on that tab and paste:

const counts = {};
for (let i = 0; i < 200; i++) {
  const r = await fetch('/?n=' + i + '&t=' + Date.now(), { cache: 'no-store' });
  const body = (await r.text()).trim();
  counts[r.status + ' ' + body] = (counts[r.status + ' ' + body] || 0) + 1;
}
console.table(counts);

It sends 200 sequential requests from the same origin, skips the cache, and counts each status-plus-body pair. A run takes about 80 seconds (median 402 ms per request).

Here is what you get with the same two containers and the same 9/1 weights, changing only the strategy:

Strategy	Targets / weights	Requests	v1	v2	v2 share
Round Robin	app-v1 w9, app-v2 w1 (dashboard showed 90% / 10%)	200	78	122	61%
Round Robin, 2nd run	same	200	100	100	50%
Round Robin	3 × v1 + 1 × v2, all weight 1	200	150	50	25%
Weighted Round Robin	app-v1 w9, app-v2 w1	200	180	20	10%
Weighted IP Hash	app-v1 w9, app-v2 w1, one client	40	40	0	0%

Every response was a clean v1 or v2 reply, with no errors.

What the table says:

Round Robin ignores your weights. The target cards still show "90%" and "10%", which is what makes it dangerous. A repeat run settles into strict alternation: v1, v2, v1, v2. Your "10% canary" is a coin flip. (A first run can start with 15 v1 responses, then a long v2 streak, then the same alternation; that run gave v2 61%.)
Weighted Round Robin is exact. v2 answered request 1, 11, 21 and so on. Precisely 20 of 200. No randomness, so you don't need thousands of requests to trust it.
Target count also works. Three v1 containers and one v2 under plain Round Robin gave exactly 25%. That's the fallback if you're on a balancer without weights, but it costs you extra containers.
IP hash pins you. With Weighted IP Hash, all 40 requests from a single browser go to v1. That's what you want for sessions. It's terrible for checking your canary from your own laptop.

Step 4: watch v2, then promote or roll back

While v2 takes its 10%, watch two numbers per version: error rate and latency. On Hubfly, each container has its own Logs and Metrics tabs, so you can check app-v1 and app-v2 separately. One Traefik canary guide rolls back when the canary's 5xx rate passes 5%, and alerts when p95 latency stays above 2 seconds for 5 minutes. Pick numbers that match your app.

Promote: raise app-v2's weight in steps (for example 1 → 3 → 9 against v1's 9), checking each step with the script. When v2 carries everything, remove app-v1 from the targets. The balancer details panel edits weights live, and every step uses the same control. The balancer also health-checks targets with lightweight requests to paths like /health and drops a failing one from rotation, as described in the docs. Give v2 a real health endpoint so that check means something.

Roll back: open the balancer, click the trash icon on the app-v2 target. With a request loop running, the last v2 response comes back about 0.66 s after the click, and none of the 108 requests in that loop fail.

What can go wrong

1. "I set 90/10 and v2 gets half the traffic."
Your strategy is plain Round Robin. It ignores weights, even though the UI shows percentages. Pick the Weighted version of whichever strategy you use, then rerun the script.

2. "I can't reproduce the bug users report on v2."
You're on an IP-hash strategy, and your IP hashes to v1. Weights then decide which clients land on v2, not which requests. Test v2 directly before adding it to the balancer, or temporarily use Weighted Round Robin while you test.

3. "v2 never answers after I add it."
Check the target port. The form defaults to 3000. If your container listens elsewhere (here: 5678), the balancer is knocking on a closed door. Also check that v2 is running before you add it.

Two smaller gotchas from the dashboard: the + button next to a weight steps by 5 (1 → 6 → 11), and scrolling the mouse wheel over a weight slider changes it. Re-read the weights before you close the panel.

The cost

Four shared-tier containers plus the balancer project to $0.00/month in the dashboard. The shared tier counts against the free 360 compute hours a month.

For a production-sized canary, say both versions as dedicated containers with 1 vCPU, 0.5 GB RAM and 5 GB disk, each one costs about $3.71/month always on:

CPU: 1 × $0.004 × 730 h = $2.92
RAM: 0.5 × $0.002 × 730 h = $0.73
Disk: 5 × $0.00001536 × 730 h ≈ $0.06

The canary only exists during the rollout. Running v2 next to v1 for a 24-hour canary adds about $0.12.

Takeaway

Setting a weight is not the same as getting a split. Pick a weighted strategy, then count. Two hundred requests and one table tell you more than the dashboard's percentages do.

Deploy this

Paste the Compose file above into a new Hubfly project, add a load balancer with Weighted Round Robin and weights 9 and 1, and run the script. The free tier needs no card, and this setup projects to $0.00/month. Start on Hubfly.

DEV Community