I shipped a broken release on purpose: canary deploys and auto rollback on my own server
At 16:40 on a Friday I had a one line change. I stared at the deploy button for a full minute. Nothing in the diff was scary. What scared me was that if I was wrong, every visitor would see it at once, and I would find out from an angry message instead of from a graph.
That fear has a simple cause. A normal deploy is all or nothing: the new version either takes 100% of traffic or none. I tested the alternatives on purpose. I built a small app, made version 2 answer every page with a 500, and shipped it two ways: as a staged canary and as a plain deploy. I used Levelrail, the open source self-hosted platform I work on, so weigh the tool choice accordingly. The behavior is the part worth stealing.
TL;DR
| Scenario | What I did | Result |
|---|---|---|
| Canary of a broken image |
apps canary start with stages 10, 50, 100 |
rolled back on its own, 60 of 1000 requests failed |
| Canary of a good image | same command | passed both stages, promoted itself |
| Plain deploy of the same broken image |
apps deploy, auto rollback left on default |
rolled back in about 37 seconds, 490 of 1200 requests failed under heavy load |
| Deploy policy | registry allow list set to enforce | deploy refused with a 422 and a fix hint |
Where the box ran
Everything here ran on my laptop against a throwaway local Levelrail instance, not a rented VPS. The control plane and CLI were a local build from 10 November 2026, and I read the docs from the v1.0.1 release branch. Containers ran in OrbStack, and images came from a local registry container. Load was a shell loop of curl calls, about 13 requests a second. Treat the timings as a shape, not a benchmark.
The test app
A Node server of 10 lines. Every version answers /healthz with 200. Version 1 and 3 answer / with their name. Version 2 answers / with a 500.
const version = process.env.VERSION || "1";
const broken = process.env.BROKEN === "1";
http.createServer((req, res) => {
if (req.url === "/healthz") { res.end("ok"); return; }
if (broken) { res.statusCode = 500; res.end("v" + version + " exploded"); return; }
res.end("shop v" + version);
}).listen(3000);
This matters because the broken version is healthy by every start-up check. The process runs, the port accepts connections, the health route says ok. That is the kind of bug that slips through: it only fails for real requests.
One gotcha cost me time. A canary needs an image that comes from a registry. My first attempt used an image that only existed in the local Docker daemon, and Levelrail refused with built from source, canary supports image apps. Pushing the three tags to a local registry container fixed it. A normal server pulling from a real registry never hits this.
Canary deploys: let 10% find the bug
A canary runs the new image next to the current one and sends it a share of requests. Levelrail splits per request, not per user, using a repeating cycle of 100 requests, so a weight of 10 sends 10 of every 100 to the canary. A canary that is not ready gets no traffic.
The staged form does the babysitting for you:
levelrail-cli apps canary start w2c-shop \
--image localhost:42301/w2c-web:2 --stages 10,50,100 --interval 20s
I used a short interval so the test would finish. The docs default is 5 minutes per stage. Each stage compares the canary's 5xx rate and p95 latency against the stable release over the same window. It needs at least 30 canary requests before it judges anything, and it never promotes a canary it has not observed.
Here is canary status while my loop ran against the broken image:
stage: 1 of 3 ([10 50 100])
canary: 50 requests, 100.0% errors, p95 47 ms
stable: 460 requests, 0.0% errors, p95 79 ms
next: error rate 100.0% vs 0.0% on the stable release (limit +2.0 points)
The next status call said no canary is running, and last canary: rolled_back with the same reason. Of the 1000 requests in that loop, 60 returned a 500, every one from the canary, before it left. The rest came from the stable release, and the page kept saying shop v1.
That is the whole pitch of a canary in one number: 6% of requests saw the bug, instead of 100%.
Then I ran the same command with a good version 3. Stage 1 passed once it had enough traffic, stage 2 passed, and the status ended with last canary: promoted ... canary auto promote. Promotion goes through the normal deploy path, so the readiness gate and rollback pinning still apply. When I stopped sending traffic mid-test, the canary just waited at waiting for traffic (0 of 30 requests). It does not promote on silence.
The limits worth knowing
The docs list them and I hit one myself. One canary per app. Image apps only, with a single replica and no pinned host port. The canary has its own container but shares your database, so if the bug writes bad rows, a canary writes them too. A canary protects you from crashes and error spikes. It does not protect you from a migration.
The deploy nobody canaried
You will still push a plain deploy sometimes, at 17:00, from your phone. So I deployed the broken image straight onto the stable app, with no canary.
The readiness gate passed it. My app had no health check declared, so Levelrail fell back to a TCP connect on the port, and the broken server accepts connections. This is where the post-deploy watch comes in. For the first 5 minutes after cutover (default) it checks every 10 seconds for OOM kills, restart loops and a 5xx share above 20% once 20 requests have arrived. The default mode is auto, which means it rolls back by itself.
levelrail-cli apps watch get w2c-shop
levelrail-cli apps deploy w2c-shop --image localhost:42301/w2c-web:2
levelrail-cli deploy status w2c-shop
About 37 seconds after I started the deploy, a new deploy row appeared with reason: AutoRolledBack, the app was serving version 3 again, and the failed one carried the reason PostDeployErrorRate. The audit log and the deploy's gate timeline record it too, according to the docs.
During those 37 seconds my loop sent hundreds of requests, and 490 of 1200 failed. A canary would have kept that number near 60. Auto rollback caps how long the bug lives, and it needs nobody awake. It does not cap how many people hit it.
Two details I read but did not test: the watch also has a soft latency signal that only alerts unless you opt in to rolling back on it, and the repo's own status page lists that latency signal as unit tested only. The hard signals (5xx spike, crash loop) have live tests on a real Docker engine.
Deploy policies: refuse the bad deploy first
Canaries and rollbacks help after the fact. A policy refuses a deploy that breaks your own rules. Rules apply at organization, project or environment scope, each one is enforce, warn or off, and the most specific scope wins. The rules include an allowed registry list, a required readiness probe, memory and CPU ceilings, a replica cap and required labels.
levelrail-cli policy set org \
--rule allowed_registries=enforce:ghcr.io \
--rule require_readiness_probe=enforce
levelrail-cli policy check --fail
levelrail-cli apps deploy w2c-shop --image localhost:42301/w2c-web:1
policy check is a dry run. The dry run said the app would fail and exited with code 3, so it can sit in a CI step. The deploy itself came back as a 422:
image registry "localhost:42301" is not on the allowed list. Push the image to one of: ghcr.io, or add the registry to the policy.
The docs say rollbacks skip policy, so restoring a known state is never blocked, and a root caller can override with a recorded reason. I did not test the override. I cleared the policy afterwards.
Comparing the three safety nets
| Net | What it catches | User impact in my run | Use it when |
|---|---|---|---|
| Readiness gate (default) | app that never starts or never listens | none, old version keeps serving | every deploy |
| Staged canary | errors and slow responses on real traffic | 60 failed requests of 1000 | risky changes with steady traffic |
| Post-deploy watch | crash loops, OOM kills, 5xx spikes after cutover | 490 failed of 1200 under heavy load, about 37 seconds | everything else, as the backstop |
A canary needs traffic to judge anything, so on a site with two visits an hour, a 5 minute stage will sit and wait. The watch has the same need for its 20 request minimum. For tiny sites, a good health check and a hand check beat both.
FAQ
Do canary deploys work on a single server?
Yes. The canary is a second container of the same app on the same machine, with traffic split by the ingress. In my run that was one laptop.
Does auto rollback undo a database migration?
No. It redeploys the last good image and its config. Data stays as it is. Plan schema changes so the old version can still read the new schema.
Can I set canary defaults per app?
Yes. Defaults come from APP_CANARY_* environment variables on the control plane, and an app can override them in its spec. This block is from the docs and I did not run it:
services:
web:
image: myorg/web:1
strategy: canary
canary:
stages: [10, 25, 50, 100]
interval: 10m
max_error_rate: 0.01
max_p95_ratio: 1.3
That block only saves the policy. The stable app still deploys blue-green, and a canary starts when you start one.
What happens if the control plane restarts mid-canary?
The docs say the state lives in the database and a reconciler picks it up again, so a restart reopens the current stage's observation window. I did not restart it during a canary.
Where I landed
The fear was never about the diff. It was about not knowing how many people would see a mistake before I did. A canary turns that into a number you pick, and a watch puts a ceiling on the time. What percentage of traffic would you let a first look at your next release?
glincker
/
levelrail
Self-hosted alternative to Vercel and Coolify. Git push, HTTPS, logs, rollback. No Kubernetes.
Levelrail
Your own Vercel, on your own servers.
Push to deploy with HTTPS, logs, metrics and one-step rollback. One Go binary, no Kubernetes. Apache 2.0.
Website · Docs · Install · Compare · FAQ · Roadmap · Discussions
English · 简体中文 · Español · Português (BR) · Deutsch · Français · 日本語
Turn any Linux server into a private cloud: Git deploys, HTTPS, databases with verified backups, metrics, logs and alerts, without Kubernetes and without a monitoring stack to bolt on. Pre-1.0: single-server is production ready, multi-node is beta (Feature status).
idle RAM, 1.5% of one CPU Go control plane, Docker runtime one-click service templates MCP tools for coding agents
200 MB
measured, method included
1 binary
no Kubernetes
360
smoke-tested
240+
scoped, read-only by default
Install
curl -fsSL https://levelrail.com/install.sh | sudo sh
Needs Linux (amd64 or arm64) with systemd and root; the installer adds Docker. Open ports…
If this saved you time, a star on the repo helps others find it.
Sources checked
- Levelrail docs on the v1.0.1 release branch: docs/canary-deploys.md (staged canaries, limits, defaults, per-app spec), docs/deploy-safety.md (readiness gate, post-deploy watch defaults, auto rollback), docs/deploy-policies.md (rules, 422, override, rollbacks skip policy), docs/feature-status.md (post-deploy watch stable for hard signals, latency signal unit tested only)
- Local runs on 2026-11-10 and 2026-11-11 described above, on a laptop with OrbStack


Top comments (0)