DEV Community: Mikhail Dorokhovich

The #1 beginner Kubernetes bug: Service with no endpoints (and why labels are the glue)

Mikhail Dorokhovich — Sat, 15 Aug 2026 19:15:32 +0000

Chapter 7 of a local-Kubernetes series: the three manifests you actually need — Namespace, Deployment, Service — and the "aha" that they're glued together by labels and selectors, not hard references.

Key takeaways

Namespace = a folder for your resources. Don't dump apps in default; a dedicated namespace makes cleanup, limits, and isolation easy. Same-namespace access uses the short name (myapp); cross-namespace needs the FQDN (myapp.myapp.svc.cluster.local).
The object hierarchy: Deployment → ReplicaSet → Pods. You describe only the Deployment; Kubernetes creates the rest. Pods are ephemeral (new IP on every recreate), so you never address them directly.
The rule everyone trips on: spec.selector.matchLabels must match spec.template.metadata.labels. Diverge and Kubernetes rejects the manifest at apply time.
Service (ClusterIP) gives a group of Pods one stable virtual IP + DNS name, reachable only inside the cluster; it tracks which Pods match its selector via EndpointSlices.
port vs targetPort: port is what the Service listens on (we use 80); targetPort is the Pod's port (8080). Defaults to port if omitted.
The single most common bug: the Service's selector doesn't match the Pods' labels → kubectl describe svc shows Endpoints: <none> → the service exists but traffic goes nowhere. One command to diagnose.
apply (declarative, idempotent) vs create (imperative, fails if it exists). Always apply for Git-tracked manifests. kubectl diff -f before applying.
Keep manifests in Git next to the code (a flat k8s/ folder); reach for Kustomize base+overlays when environments diverge.

Full article: https://dorokhovich.com/blog/local-k8s-kubernetes-manifests?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-kubernetes-manifests

Local Kubernetes Dev — Part 7: Kubernetes manifests for your service

Mikhail Dorokhovich — Thu, 13 Aug 2026 09:49:08 +0000

"The Service exists, but traffic never reaches the pods, and Endpoints is empty" — the most common beginner bug in Kubernetes. And the cause is almost always the same one.

Part seven of the series is the chapter where, instead of copy-pasting YAML, you finally understand HOW Kubernetes objects connect to each other. We write three minimal manifests for myapp — Namespace, Deployment, Service — and unpack what you really need to internalize: labels and selectors are the glue that binds Deployment and Service together. No hard references like "Service, here are the IDs of these pods" — it's all held together by matching labels.

The key rule everyone trips on: a Deployment's spec.selector.matchLabels MUST match its spec.template.metadata.labels. And the big trap — the Service selector doesn't match the pod labels (a typo, or you forgot to update it): the Service just hangs, and kubectl describe svc shows Endpoints: <none>. The service seems to exist, but traffic goes nowhere. You can verify it instantly.

Plus: how ClusterIP differs from a pod's direct address, the difference between port and targetPort (80 vs 8080), declarative kubectl apply vs imperative create, and where to keep your manifests (spoiler: in Git next to the code, in a k8s/ folder). https://dorokhovich.com/blog/local-k8s-kubernetes-manifests?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-kubernetes-manifests

Storytelling in Software Design: The Principle That Makes Architecture Docs Actually Stick

Mikhail Dorokhovich — Wed, 12 Aug 2026 12:32:07 +0000

The problem in context

Every engineering org I have worked with has the same artifact: a wiki page that opens with something like 'Three-tier architecture: presentation, business logic, data access.' It is accurate. It is correct. And it is completely inert. New hires read it, nod, and then spend their first month asking questions the page technically already answered.

The instinct is to blame the writer or the tooling — not enough diagrams, wrong wiki, stale content. That instinct is wrong. The information was fine. What failed was the form. We had hidden the meaning behind jargon and boxes-and-arrows and stripped out the one thing human cognition is actually built to retain: a throughline. A component inventory is a list of nouns. A system is a sequence of things happening to actors over time — and our brains file the second away far more reliably than the first.

That gap between what we produce (inventories) and what people retain (narratives) is the whole problem. It shows up as slow onboarding, as design reviews that miss the point, and as decisions that evaporate the moment the person who made them leaves.

The principle

The principle here is that story is a design technique, not decoration. A narrative — actors, an event, a resolution — is a way of structuring concepts so that intent is visible, and intent is the part that has to survive new people and the passage of time.

Here is the same system, documented two ways:

Inventory form: 'Three-tier architecture: presentation, business logic, data access.'

Narrative form: 'Clients (presentation) consult experts (domain) who consult the archive (data) and return a decision. Each request is a plotline that crosses actors and comes back with a resolution.'

The narrative version is not fluffier — it is more precise, because it exposes intent rather than just structure. A reader who absorbs the second version can reason about where a new feature belongs. A reader who memorizes the first can only recite it. That is the entire thesis in miniature: narrative scales because intent scales, and implementation details do not.

The leverage point is that you do not need a new methodology to apply this. The mature design practices are already narrative — most teams just run them mechanically and miss it. There is a detailed treatment of this reframing in a longer essay on software design as storytelling if you want the full argument; the short version is that five techniques you likely already own are stories wearing engineering clothes.

Domain-Driven Design (DDD). Bounded contexts are chapters; the ubiquitous language is the cast's shared dialogue. Ask 'who are the characters here and what words do they use?' and boundaries that looked arbitrary suddenly have reasons.
Behavior-Driven Development (BDD). Given / When / Then, the grammar Dan North coined for BDD, is a plot: a setup, an event, a resolution — readable by every stakeholder and executable as a spec.
Event Storming. Alberto Brandolini's workshop format models events, commands, and policies on a wall until the arc of the system emerges in sticky notes. You watch the plot appear.
Domain Storytelling. The most literal technique, documented by Stefan Hofer and Henning Schwentner and featured on the Thoughtworks Technology Radar: domain experts draw their workflow as numbered steps — actors, work objects, activities, in order. 'The dispatcher assigns the order to a driver' hands you the language and the boundaries for free.
Architecture Decision Records (ADRs). Michael Nygard's format is the memoir: context, decision, consequences — the why, preserved for whoever inherits it. Pair it with Simon Brown's C4 model for the map, but keep the rule that every diagram earns a sentence of narrative or it is just decoration.

Trade-offs

Narrative design is not free, and treating it as an unqualified good is how it degrades into whimsy. The honest comparison:

Dimension	Inventory docs	Narrative design
Onboarding retention	Read once, forgotten	Retained and reasoned about
Authoring cost	Low up front	Higher up front, lower over time
Precision	Exact but intent-free	Exact and intent-bearing when done well
Failure mode	Inert, ignored	Vague metaphor if intent is hidden
Cross-team alignment	Repeated meetings	Shared story spine
Decision durability	Lost when people leave	Preserved in ADRs
Business 'why'	In one senior head	Embedded in the artifact

The two costs that matter: narrative takes more effort to write well, and it fails badly when done poorly. A story that hides detail instead of exposing intent is worse than a dry doc — you have added metaphor without adding clarity. So the discipline is that narrative must be more precise, never less. If your reframing makes the system harder to reason about, revert it.

How to adopt

The adoption failure I see most often is boiling the ocean — a team decides to rewrite every document as a story and stalls under the weight. Stage it instead.

Start with the single most-ignored document, usually the onboarding page, and rewrite it as a story with actors, a journey, and a resolution. Prove the principle on the artifact that hurts most before you touch anything else.
Write your next architecture decision as an ADR the day you make it. An ADR written at decision time preserves intent; one reconstructed six months later preserves a guess. This is the highest-leverage habit on the list.
Run one flow through Event Storming as a group. Put a single user-facing flow on a wall — events, commands, actors — and narrate it end to end out loud: 'the customer does this, which triggers that, which billing decides on, and the resolution is this.' Within an hour the arc emerges, and so do the boundary disagreements the team has been silently carrying. Saying the story together is what makes the ubiquitous language stick — it arrives by conversation, not decree.
Enforce the vocabulary. If the story's words and the code's names drift apart, the throughline is gone. Keep them in sync or the whole effort decays.

Notice what this is not: no new framework, no tool purchase, no reorg. You are changing the form of what you already produce so the meaning survives contact with new people and with time.

Where this goes next

The direction of travel is that narrative stops being how you document a system and becomes how you design it. Once a team pitches features as plotlines — 'here is the actor, here is what changes, here is the resolution' — reviews sharpen automatically, because a reviewer can ask 'whose story is this and where does it resolve?' and expose a design gap no checklist would catch.

There is a second horizon worth watching: as more of the design surface gets handed to AI assistants, intent-bearing artifacts become the differentiator. A model can generate a component inventory trivially; what it cannot invent is why your bounded contexts sit where they do. Teams that have captured that why as narrative and ADRs will hand their tools far better context than teams sitting on boxes-and-arrows. The organizations that treat storytelling software design as a first-class engineering practice — rather than a nicety for the onboarding wiki — are the ones whose intent will outlast both their people and their tooling.

Sources & further reading

Stefan Hofer & Henning Schwentner — Domain Storytelling, and its Thoughtworks Technology Radar entry
Alberto Brandolini — EventStorming
Martin Fowler — Bounded Context (Domain-Driven Design)
Dan North — Introducing BDD (the origin of Given/When/Then)
Michael Nygard — Documenting Architecture Decisions (the original ADR format)
Simon Brown — The C4 model for visualising software architecture
A longer reference treatment of reframing software design as storytelling

A production Dockerfile for FastAPI: the two mistakes beginners always make

Mikhail Dorokhovich — Tue, 11 Aug 2026 19:31:20 +0000

Chapter 6 of a local-Kubernetes series. Not another "your first Dockerfile" — it leads with the two things beginners get wrong that have real consequences: running as root (a security liability) and cache-busting layer order (a productivity tax).

Key takeaways

Layer order is caching. Put what rarely changes first. COPY requirements.txt + pip install before COPY ./app, so a one-line code edit doesn't reinstall every dependency. The reverse (COPY . . before install) is the most common anti-pattern.
Run as a non-root user. adduser --disabled-password --uid 10001 appuser then USER appuser; reinforce with securityContext.runAsNonRoot: true in the Pod. Least privilege, smaller attack surface.
Multi-stage build: install deps into a venv in a build stage, COPY --from=build /opt/venv /opt/venv into a clean runtime image — no compilers or pip caches in the final layer.
Base image trade-offs: slim (glibc, best wheel compatibility — the sane default), alpine (musl → pip often compiles from source, slow/fragile, DNS quirks), distroless (no shell/package manager, great secure runtime, exec-form only, pair with multi-stage).
exec form of CMD is not optional: ["fastapi","run",...] makes the app PID 1 and receives SIGTERM directly. Shell form wraps it in /bin/sh, SIGTERM never reaches the app, and graceful shutdown breaks.
FastAPI specifics: fastapi run (prod-tuned uvicorn) not bare uvicorn; never --reload in the cluster image; PYTHONUNBUFFERED=1 or logs never reach kubectl logs; add --proxy-headers behind Ingress.
HEALTHCHECK without curl: slim/distroless have no curl, so probe with Python's urllib.
The finale (and the trap): docker build does NOT make the image visible to k3d — its nodes run isolated containerd. Deliver it via k3d image import myapp:dev -c dev (+ imagePullPolicy: IfNotPresent) or push to the built-in registry.

Full article: https://dorokhovich.com/blog/local-k8s-containerizing-your-service?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-containerizing-your-service

A Raspberry Pi VNC Remote Desktop From Anywhere: The Principle of Multiplexing a Tunnel You Already Own

Mikhail Dorokhovich — Tue, 11 Aug 2026 19:26:01 +0000

The problem in context

SSH into a home Raspberry Pi gives you a black text console — perfect for editing configs, useless for anything with a mouse. Most of the time that is fine, but there is always a stubborn 10%: a graphical config tool with no CLI equivalent, or a camera app whose live preview you genuinely need to see. For those, you need VNC — screen-sharing that streams the Pi's desktop to your laptop and sends clicks and keystrokes back.

The naive path is to install a VNC server, forward port 5900 on the router, and connect. Two things kill that plan. First, a Pi behind carrier-grade NAT has no router port to forward — the same constraint that forces an outbound tunnel for SSH in the first place. Second, and more important, exposing raw VNC to the internet is a genuinely bad idea. The protocol's own spec, RFC 6143, concedes its password check is "cryptographically weak and is not intended for use on untrusted networks," and recommends tunneling it inside IPsec or SSH. So the real problem is not "how do I reach VNC" but "how do I get a graphical desktop remotely without adding any new attack surface."

The principle

The principle here is that an SSH tunnel is not limited to one port — the same encrypted connection that already carries SSH can carry a desktop too, so you multiplex a pipe you already own rather than opening a new one. If a reverse SSH tunnel already forwards the Pi's port 22 out to a cloud bridge, VNC is just a second reverse forward on that same connection. Nothing new is exposed; the Pi still only ever dials outward; the traffic is wrapped in SSH the entire way.

The mental model that makes it click is three ports and one path:

Port 5900 on the Pi is where the VNC server already listens — per the official Raspberry Pi remote-access docs, a VNC server (wayvnc on current Raspberry Pi OS) is built in, so nothing gets installed.
Port 5901 on the bridge becomes the public front door that forwards into the tunnel.
Everything hitting EC2:5901 gets redirected, encrypted, down to RPi:5900.

VNC Viewer  -->  EC2:5901  -->  encrypted SSH tunnel  -->  RaspberryPi:5900  -->  desktop image back the same way

There is a detailed treatment of the full VNC-over-tunnel build with the port map and security-group walkthrough; the mechanical change is a single added flag. An existing systemd tunnel unit carrying only SSH looks like:

ExecStart=/usr/bin/ssh -i /home/pi/.ssh/id_tunnel ... -R 2222:localhost:22 tunnel@YOUR-EC2-IP

Adding a second -R — the same remote-forwarding mechanism from the ssh(1) man page, just another instance of it — carries the desktop:

ExecStart=/usr/bin/ssh -i /home/pi/.ssh/id_tunnel ... -R 2222:localhost:22 -R 5901:localhost:5900 tunnel@YOUR-EC2-IP

After systemctl daemon-reload and a restart, ps aux | grep ssh | grep tunnel showing -R 5901:localhost:5900 in the running process is the checkpoint that the tunnel picked up the new forward.

Trade-offs

Multiplexing an owned tunnel beats every VNC-in-the-cloud SaaS on the axes that matter for a homelab, but the choice carries real security decisions you have to make deliberately:

Decision	Convenient / default	Safer choice	Why it matters
Bridge port binding	Loopback (SSH default)	`GatewayPorts yes`	A loopback-bound forward is reachable from the bridge but never from your laptop
Security-group source	`0.0.0.0/0` on 5901	Scope to your own IP	Open means the whole internet can reach your VNC login prompt
VNC transport	Raw VNC over the internet	Wrapped in the SSH tunnel	The protocol's own auth is weak by its spec's admission
VNC server	RealVNC free tier	TigerVNC / wayvnc	RealVNC increasingly pushes cloud-brokered connections, the opposite of a self-owned tunnel

The GatewayPorts line is the one that eats an hour. Check the bridge with sudo ss -tulpn | grep 5901 and you will likely see it bound to 127.0.0.1 — localhost only, reachable from the bridge but not from across the internet. This is SSH's safe default; the sshd_config(5) man page notes sshd binds remote forwardings to the loopback address unless told otherwise. One line, GatewayPorts yes, plus systemctl restart ssh and a tunnel restart, rebinds it to 0.0.0.0:5901. The rule worth internalizing: a 127.0.0.1-bound forward is not broken, it is just missing GatewayPorts.

The deeper trade-off is ownership versus convenience. Routing desktop pixels through a third-party VNC broker is easier, but the whole point of the owned tunnel is that no pixels ever leave infrastructure you control — the bridge is your box, the transport is plain SSH, and the only thing installed is a viewer on your laptop. That is worth defending, which is why the RealVNC-pushing-toward-cloud-brokering drift matters: if a direct address:port connection gets gated behind a subscription, TigerVNC or wayvnc speaks the same protocol over the same -R 5901:localhost:5900 forward, and the tunnel does not care which server sits on port 5900.

How to adopt

This assumes the reverse SSH tunnel already exists — VNC rides on top of it, so build that foundation first if you are starting from scratch.

Add the second -R forward to the existing systemd unit and confirm it in the running process.
Flip GatewayPorts yes on the bridge, restart sshd, then restart the tunnel on the Pi so the forward re-establishes against the updated server. Order matters — re-check ports on the bridge after each restart. A local nc -zv localhost 5901 on the bridge confirms the forward actually reaches the Pi before you touch the firewall.
Open exactly one firewall port, scoped tight: a security-group inbound rule for TCP 5901 sourced to your own IP, not 0.0.0.0/0. If you need roaming access, keep the VNC password strong and layer SSH key auth in front.
Point a viewer at EC2-IP:5901, enter the VNC password, and the desktop fills the window — mouse and keyboard, live.
Make the session pleasant, not just working. Drop the viewer's color depth for a big responsiveness win over a home uplink; set a sane fixed resolution on the headless Pi so the remote window fits your laptop; and prefer wired Ethernet on the Pi, which removes a whole class of "why is this laggy" questions before you ask them.

Two operational choices keep it maintainable. Keep the VNC forward in the same systemd unit as SSH — one process, one thing to monitor, one place to look when something is wrong, and removing VNC is deleting one flag and reloading. And understand the reconnection behavior: because VNC rides the same managed tunnel, a network drop restarts both, and the viewer simply reconnects once the forward is back, usually within the ten-second restart window. A frozen screen after a Wi-Fi blip is not a failure; it comes back on its own.

Where this goes next

The immediate generalization is that any service on the Pi can ride this same connection — a metrics endpoint, a database, an internal dashboard — each one a -R forward on the tunnel you already trust, none of them exposed at home. Once you have internalized "one outbound connection, many services multiplexed back," the pattern scales from a single Pi to a small fleet without ever revisiting the router.

The larger direction of travel is that this hand-built setup is a concrete instance of where secure remote access is heading industry-wide: identity- and key-gated access to individual services over an outbound-initiated, encrypted transport, rather than exposed inbound ports on a trusted network. That is the same premise underneath zero-trust access and the managed desktop-brokering services — the difference is only who owns the hops. Building it yourself, one -R flag at a time, is the best way to understand what those products are actually doing, and to keep the option of owning every hop when the traffic is your own desktop. The tooling will keep getting easier; the principle of multiplexing an owned, outbound tunnel is the part worth keeping.

Sources & further reading

RFC 6143 — The Remote Framebuffer Protocol — VNC's own spec, on why its auth is weak and should be tunneled.
Raspberry Pi — Remote access documentation — enabling the built-in VNC server.
OpenSSH ssh(1) man page and sshd_config(5) man page — the -R forward and GatewayPorts.
TigerVNC — an open-source VNC server/viewer that speaks the same protocol without a subscription.
A longer reference treatment of VNC over a reverse SSH tunnel — the port map and AWS security-group walkthrough behind this framing.

A Reverse SSH Tunnel for a Raspberry Pi: The Principle of Dialing Out Instead of Being Reached

Mikhail Dorokhovich — Mon, 10 Aug 2026 08:12:29 +0000

The problem in context

Consider a common homelab bind: a Raspberry Pi at home records a camera feed, runs a couple of cron jobs, and hosts a small dashboard. On the LAN it is fine. Then you travel, open a laptop, type the familiar ssh pi@home, and get a timeout. The instinct is to log into the router and forward port 22 — except there is no public IP to forward to. The ISP has quietly moved the connection behind CGNAT (carrier-grade NAT), the shared address space reserved in RFC 6598, which means the "public" IP is shared with hundreds of subscribers and you control none of it.

That is the real problem, and it is worth naming precisely because it explains why the usual fixes all fail: port forwarding is meaningless when the port lives on someone else's equipment. DDNS points at an address you cannot open, UPnP does nothing, and VPN appliances want a static IP you do not have. Every one of those approaches assumes you can accept an inbound connection at home. Behind CGNAT, you cannot — and no amount of router configuration changes that.

The principle

The principle here is that NAT blocks inbound connections but happily allows outbound ones, so you stop waiting for the world to knock and have the Pi knock on a server you own — then keep that door propped open. This is a reverse SSH tunnel, and it is the cleanest escape hatch behind CGNAT, a locked-down office network, or any router you would rather not touch.

Concretely: rent a tiny always-on box in the cloud with a real public IP, have the Pi open an SSH connection out to it, and use SSH remote port forwarding (-R) so a port on the cloud box tunnels straight back through that connection to port 22 on the Pi. You connect to the cloud box, hop through the tunnel, and you are on the Pi — no router config, no exposed ports at home. The architecture is three parts:

Your laptop (anywhere)  -->  EC2 bridge (public IP, :2222)  -->  reverse tunnel  -->  Raspberry Pi (:22, behind CGNAT)

EC2 bridge: Ubuntu on a t2.micro (free-tier eligible); its only job is to be reachable and forward port 2222 back down the tunnel.
Raspberry Pi: initiates and holds the tunnel open, restarting automatically if the network hiccups.
The tunnel: an SSH remote port forward (-R 2222:localhost:22) wrapped in a systemd service so it survives reboots and drops.

The -R flag is the whole trick — the ssh(1) man page defines it as forwarding connections to a given TCP port on the remote host back to the local side. There is a detailed treatment of the full reverse-SSH build with every security-group rule and connection method; the mechanics compress to a handful of config lines below.

Trade-offs

A reverse SSH tunnel is not the only way to reach a machine behind NAT, and it is worth being honest about where it wins and loses against the managed options:

Approach	What it costs	What it buys	Best when
Reverse SSH tunnel	You run and own a cloud bridge	Depends on nothing but SSH + systemd; every hop is yours; no third-party control plane	You want to own every hop and understand exactly what carries your packets
Cloudflare Tunnel	Traffic rides a third-party control plane	Outbound-only `cloudflared`; firewall can block all inbound; least setup	You want the shortest path and don't want to run infrastructure
Tailscale	A coordination service brokers peers	Clever STUN/ICE NAT traversal builds direct peer-to-peer links	You want a mesh across many devices with minimal fuss

All three are good; the choice is about what you are optimizing for. Cloudflare Tunnel is the easier button. The reverse SSH tunnel wins on ownership: it depends on nothing but tools already on the Pi, it does not route traffic through anyone else's control plane, and the cloud bridge is a box you can rebuild in minutes. The cost you accept in exchange is running that bridge and securing an internet-facing port yourself — which is a real cost, and the section below is mostly about paying it correctly.

How to adopt

Build it in layers, proving each one before wrapping it in automation.

Prepare the bridge. Launch a t2.micro, open port 22 to your own IP and 2222 to 0.0.0.0/0, and set three lines in /etc/ssh/sshd_config:

GatewayPorts yes
ClientAliveInterval 60
ClientAliveCountMax 3

GatewayPorts yes is the line most people miss. As the sshd_config(5) man page notes, sshd binds remote forwardings to loopback by default, so without it port 2222 stays invisible from the internet — the tunnel looks healthy but is unreachable from your laptop. The keepalives let the server reap dead tunnels instead of leaving zombie forwards holding the port.

Create a dedicated, unprivileged tunnel user on the bridge whose only reason to exist is holding this forward. Least privilege: if it is ever compromised, the blast radius is one useless shell.
Prove a manual tunnel from the Pi with a fresh ED25519 key whose public half is in the tunnel user's authorized_keys:

ssh -i ~/.ssh/id_tunnel -fN -R 2222:localhost:22 tunnel@YOUR-EC2-IP

On the bridge, sudo ss -tulpn | grep 2222 should show it LISTENing.

Make it permanent with systemd, because a manual tunnel dies the moment the Pi reboots or Wi-Fi blinks:

[Unit]
Description=Reverse SSH Tunnel to EC2
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=pi
ExecStart=/usr/bin/ssh -i /home/pi/.ssh/id_tunnel -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=no -N -R 2222:localhost:22 tunnel@YOUR-EC2-IP
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Each option earns its place: ExitOnForwardFailure=yes kills the process if the forward cannot bind, so systemd retries a clean tunnel; Restart=always — which the systemd.service(5) man page defines as restarting regardless of exit status — with RestartSec=10 recovers within ten seconds; and ServerAliveInterval=60 stops idle NAT timeouts from silently severing the link. StrictHostKeyChecking=no is required because a headless unit has no human to accept a fingerprint prompt.

Make it pleasant with a two-hop ProxyCommand in your laptop's ~/.ssh/config so ssh rpi transparently connects through the bridge and onto the Pi:

Host rpi
  HostName localhost
  Port 2222
  User pi
  IdentityFile ~/.ssh/id_rpi
  ProxyCommand ssh -i ~/.ssh/rpi-tunnel-key.pem -W localhost:2222 ubuntu@YOUR-EC2-IP

The non-negotiable part is the security posture. Opening 2222 to the world is reasonable only because it is key-gated: disable password auth on the Pi, add fail2ban on the bridge, and treat that as the same-day work, not a follow-up. The gotchas that cost real time all trace back to the config above — a forgotten GatewayPorts yes (tunnel connects but stays localhost-bound), a missing StrictHostKeyChecking=no (headless service hangs on the fingerprint prompt), and zombie forwards after a hard drop (cleared by the server keepalives plus ExitOnForwardFailure).

Two operating habits keep this honest. Verify monthly — systemctl is-active on the Pi, ss -tulpn | grep 2222 on the bridge, and ssh rpi 'uptime' end to end — because a tunnel you never check fails silently the day you need it, and journalctl -u shows the reason when it does. And treat the bridge as cattle, not a pet: its entire config is a few sshd_config lines and one user, so rebuilding takes minutes, and nothing irreplaceable lives on it, which means nothing irreplaceable is exposed on it.

Where this goes next

The immediate extension is that the same tunnel carries more than SSH: add a second -R 5901:localhost:5900 forward and the headless Pi becomes a full remote desktop over VNC, reusing everything above. Once you have internalized "the device dials out and multiplexes services back," the pattern generalizes to any port you want to reach — a dashboard, a database, a metrics endpoint — without ever exposing them at home.

The larger direction of travel is that this is a hand-rolled instance of a principle the whole industry has converged on: outbound-only connectivity as the default security posture. Cloudflare Tunnel, Tailscale, and the broader zero-trust and SASE movement all rest on the same idea that a device should reach out to a control plane rather than expose inbound ports, and that identity and keys, not network location, gate access. Building the reverse tunnel by hand is the best way to understand what those managed services are actually doing for you — and to decide, per deployment, whether owning every hop is worth running the bridge yourself. The mental model is the durable part; the tooling that implements it will keep getting easier.

Sources & further reading

OpenSSH ssh(1) man page — the -R remote port forwarding flag.
OpenSSH sshd_config(5) man page — the GatewayPorts option.
systemd.service(5) man page — Restart= and RestartSec= semantics.
RFC 6598 — IANA-Reserved IPv4 Prefix for Shared Address Space — the standard behind CGNAT.
Cloudflare Tunnel docs and Tailscale's "How NAT traversal works" — the managed alternatives.
A longer reference treatment of this reverse SSH tunnel build — the security-group rules, port reference, and troubleshooting matrix behind this framing.

Release Orchestration for Microservices: The Principle of the Control Tower, Not the Light Switch

Mikhail Dorokhovich — Sun, 09 Aug 2026 08:50:42 +0000

The problem in context

Picture the release a lot of microservices shops actually run. An engineer merges to main, CI builds the images, someone clicks deploy, and thirty seconds later two dozen services are rolling simultaneously with no coordination between them. Most nights it works. The night it doesn't, a service ships a change its neighbor isn't ready for, the neighbor starts throwing 500s, the service behind it times out waiting, and within four minutes the checkout path is down. Users no longer forgive minute-long outages, and this kind lasts a lot longer than a minute.

The post-mortem is always the same shape: no staged rollout, no automated health gate, no clean way to undo, and zero visibility for anyone outside the on-call channel. That is the core problem release orchestration addresses: a fleet of 60 interdependent services being shipped with the coordination model of a single monolith. The reflex is to blame the change, or the engineer, or CI. But CI did its job — it built and delivered the code. What was missing was the layer that decides whether the whole system is ready for that code to go live, and in what order.

The principle

The principle here is that a release is air-traffic control, not a light switch. Planes can technically take off on their own; without a control tower you get chaos in the sky. Each microservice can update itself — but without a control layer deciding order, timing, and approval, one backward-incompatible change becomes a company-wide outage.

That analogy is also the cleanest way to see the difference between CI/CD and orchestration. CI/CD is the assembly line that builds and delivers code. Orchestration is the control tower deciding which system updates first, how changes synchronize, and who signs off. Most teams that suffer domino outages have a working assembly line and no tower. There is a detailed treatment of release orchestration for microservices that walks the full model; the compressed principle is that orchestration is a control layer over the pipeline, not a fancier pipeline.

The control tower is built from three techniques, best adopted in order of pain relieved:

Feature flags decouple deploy from release. Pete Hodgson's feature toggles guide names this a release toggle: ship the code, keep the behavior dark. A risky change sits dormant in production, gets flipped on for internal users then everyone, and flips off in seconds if something smells wrong — no redeploy.
Canary releases cap blast radius. Instead of 100% at once, a new version goes to a slice of traffic — what Danilo Sato's canary release definition frames as rolling out to a small subset before the whole fleet. On Kubernetes with Argo Rollouts, the config is almost embarrassingly small, mirroring the setWeight/pause model in the Argo Rollouts canary reference:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: example-app
spec:
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 10m}
      - setWeight: 50
      - pause: {duration: 10m}
      - setWeight: 100

Ten percent, pause and watch, fifty, pause and watch, then full. The pause windows are where automated health checks live — if error rate or latency crosses a threshold during a pause, the rollout aborts and holds at the last safe weight instead of marching to 100%.

Blue-green handles the changes you can't canary — schema-coupled services, mostly — by running two identical environments, the pattern Martin Fowler described in 2010. Users stay on blue while green bakes; when green is verified, traffic flips instantly and blue stays warm as a fallback.

Trade-offs

None of the three is universal; each buys a different guarantee at a different cost. The honest comparison:

Technique	Best for	Rollback	The cost to respect
Feature flags	Decoupling deploy from release	Flip off in seconds, no redeploy	Stale flags become hidden branches — debt with a fuse
Canary	Everyday releases where partial traffic makes sense	Auto-abort at last safe weight	Only as smart as the metrics behind the pause steps
Blue-green	Schema-coupled changes you can't slice	Instant flip back to blue	Roughly double resources; DB/session state is hard

Two constraints do the most damage when ignored. A canary is only as smart as its metrics: if the pause steps aren't backed by real health signals — error rate, latency, saturation — you have added slow-motion to a bad deploy, so wire the checks before you trust the automation. And config is production code: the worst incidents tend to come from YAML and flag flips, not application code, so deployment config needs the same review gate. Tooling choice is a smaller trade-off than teams expect — Argo Rollouts if you are already all-in on Kubernetes, Flagger for a lighter automatic metric-driven loop, Spinnaker for multi-cloud sprawl. The real cost is not tools; it is the team time to design the process.

How to adopt

Do not boil the ocean. The staged path that works is three steps.

Assess. Answer honestly: how long does a release take, how many manual steps, how often do you roll back, does the business have any visibility? Writing those answers down is uncomfortable and clarifying, and it gives you the baseline to prove improvement.
Start small — process before tech. Introduce release checklists, code review on config changes, and a written rollback plan for every release; then flags for new functionality, monitoring on key metrics, and automated health checks.
Scale up. Only then template the canary config across services.

Budget for the culture lag: the tools install in a day, but getting everyone to write a rollback plan per release takes a quarter. It is worth it. And when you need to justify the work, frame it in leadership's units, not yours — nobody buys "canary rollouts," they buy fewer outages and faster shipping. Calculate the cost of one hour of downtime, multiply by historical incident frequency, and set that against the near-zero cost of free tooling; the rollback plan alone tends to pay for the initiative on paper. The counterintuitive headline is that decoupling deploy from release and capping blast radius does not slow teams down — it lets them ship more, because each ship is cheap to undo. Competitors shipping 3–5x more often aren't smarter; they have just made each release cheap to reverse.

Where this goes next

The direction of travel is toward the control tower making more of its own decisions. AIOps — machine learning that predicts release risk and flags suspicious metrics automatically — is the natural next layer, feeding richer signals into the same pause-and-abort gates that a human tunes today. But it is an accelerant on top of good orchestration, not a substitute for it: you cannot ML your way out of not having a control tower in the first place.

The deeper forward-looking point is that every one of these techniques produces the structured signal that smarter automation will need — labeled rollouts, health-gated pauses, explicit rollback plans, config under review. Teams that build the tower now are not just avoiding tonight's domino outage; they are assembling the legible, well-instrumented substrate that AI-assisted release agents will reason over next. The control layer is the thing that stays valuable as the intelligence sitting on top of it improves — which is exactly why it is worth building before the automation arrives, not after.

Sources & further reading

Danilo Sato, Canary Release — martinfowler.com.
Martin Fowler, Blue Green Deployment — martinfowler.com.
Pete Hodgson, Feature Toggles (aka Feature Flags) — martinfowler.com.
Argo Rollouts — Canary Deployment Strategy — official docs.
Flagger — progressive delivery for Kubernetes — official docs.
A longer reference treatment of release orchestration for microservices — the tool comparison and the assess/start-small/scale roadmap behind this framing.

Local Kubernetes Dev — Part 6: Containerizing your service — writing a Dockerfile

Mikhail Dorokhovich — Sun, 09 Aug 2026 08:50:40 +0000

Two mistakes beginners make in Dockerfiles constantly — and both have real consequences: running as root (a security hole) and the wrong layer order (a tax on build speed).

Part six of the series is a production-ready Dockerfile for the FastAPI service myapp. Not "your first Dockerfile," but a breakdown of what separates a solid image from a bloated, insecure one. Inside: why COPY requirements.txt BEFORE COPY ./app — and pip install gets cached instead of reinstalling everything on every code change; a multi-stage build that carries only the finished venv into the final image, without compilers and pip caches; base image choice (slim vs alpine with its musl and compile-from-source vs distroless with no shell); and the security backbone of the chapter — an unprivileged user (adduser --uid 10001 + USER appuser) plus runAsNonRoot in the manifest.

Plus the important details that break prod silently: the exec form of CMD (otherwise SIGTERM never arrives and graceful shutdown breaks), fastapi run instead of bare uvicorn, PYTHONUNBUFFERED=1 (otherwise you see no logs in kubectl logs), HEALTHCHECK via Python (there's no curl in slim).

And the finale — the main trap: docker build does NOT make the image visible to k3d. Two ways to deliver it: k3d image import and the built-in registry. https://dorokhovich.com/blog/local-k8s-containerizing-your-service?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-containerizing-your-service

A GitOps Adoption Roadmap: The Principle of Walking the Arc Instead of Leaping It

Mikhail Dorokhovich — Sat, 08 Aug 2026 17:16:57 +0000

The problem in context

Anyone in the industry long enough remembers the sleepless release night: the coffee, the collective prayer that a deploy would go smoothly, the one guru who alone understood the process, no documentation, and rollbacks so frequent they stopped feeling like exceptions. The canonical horror story — a telecom billing release that ran 14 hours, involved twelve people, crashed three times, and needed another six hours to recover — is only an exaggeration by degree. Plenty of teams still gather on Friday evenings for six-to-twelve-hour manual releases.

The reflex, once the pain is acute enough, is to buy the shiniest platform and leap straight to it. That reflex is exactly what makes transformations stall. The problem is not that a team lacks GitOps; it is that manual-deploy shops try to adopt GitOps as a single jump, skipping the rungs — script automation, CI/CD, declarative delivery — that GitOps quietly assumes already exist. Framed that way, escaping 3 AM deploys stops being a unique curse and becomes a solved problem with a well-worn path.

The principle

The principle here is that a GitOps adoption roadmap is a layered arc, not a purchase: each layer assumes the one below it, so you walk scripts → CI/CD → GitOps → progressive delivery rather than leaping. The industry traveled this arc for a reason, and repeating its order is a good sign you are not skipping steps.

The mental model is four phases, each earning the next:

Assess. Measure before you automate. A blunt maturity checklist, answered honestly, tells you where the pain is and gives you the baseline you will later use to prove progress:

# Maturity assessment checklist
[ ] Deployment time (target: < 30 minutes)
[ ] Release frequency (target: weekly or more often)
[ ] Manual steps count (target: 0)
[ ] Mean time to restore (target: < 1 hour)
[ ] Change success rate (target: > 95%)
[ ] Automated test coverage (target: > 80%)
[ ] Monitoring and alerting in place
[ ] Rollback and disaster recovery procedures

Those targets line up with the throughput and stability signals DORA formalizes as its core delivery metrics. Assess organizational readiness too — executive sponsorship, willingness to change, DevOps expertise, budget — because skipping any of those is how transformations stall.

Quick wins. Chase visible pain reduction: standardize release checklists and runbooks, containerize apps, put infrastructure in code, stand up basic CI, add health checks.
Scale. Make Git the single source of truth with GitOps and a reconciling controller like Argo CD.
Optimize. Layer on progressive delivery — feature flags, canary, observability-driven automatic rollback.

There is a detailed treatment of the full manual-to-GitOps learning path with the tools comparison and timeline behind each phase; the compressed principle is that quick wins are how you earn permission for the disruptive changes. Shave one service's deploy from hours to under an hour with a basic pipeline and health checks, and the skeptics stop arguing whether automation is worth it — the proof is in front of them. Momentum is a currency, and quick wins are how you mint it.

Trade-offs

The tooling arc recapitulates the industry's evolution, and each rung trades setup effort for tighter integration and stronger guarantees. The honest comparison:

Rung	What it buys	The cost / when it fits
Script automation (Ansible)	Turns tribal knowledge into documentation-as-code; agentless, readable playbooks	Still push-based and imperative; fine as the first rung, not the destination
CI/CD (GitHub Actions)	Build/test/deploy folded into version control; low barrier	Some setup effort; pipeline logic can sprawl
GitOps (Argo CD)	Git as source of truth; versioned, reviewable, self-healing, no drift	Requires declarative discipline and Kubernetes maturity underneath
Progressive delivery (Argo Rollouts)	Analysis-gated canaries; automatic rollback on metric breach	Needs trustworthy metrics and the three rungs below it already solid

The trade-off that matters most is ordering discipline: each rung assumes the one below. GitOps on top of a cluster nobody can describe declaratively just relocates the chaos into YAML; progressive delivery without trustworthy metrics automates a decision you cannot yet make. A representative Ansible playbook and an Argo CD Application make the two ends concrete:

- name: Deploy web application
  hosts: webservers
  become: yes
  tasks:
    - name: Update application files
      copy: { src: /builds/myapp-v2.0/, dest: /opt/myapp/ }
    - name: Check application health
      uri: { url: http://localhost:8080/health, status_code: 200 }

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  source:
    repoURL: https://github.com/company/myapp-config
    path: k8s
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The deeper trade-off is where you spend effort. The tempting failure modes are all one-dimensional: changing everything at once (chaos and resistance), fixing only tooling while ignoring process and culture (minimal impact), and treating security or metrics as an afterthought. The costliest of these is cultural — the technology is the easy part; getting people to trust automation over heroics, and to run blameless post-mortems instead of assigning blame, is the real work.

How to adopt

Start with the audit, not the tooling, and walk the arc one measured phase at a time.

Run the maturity audit and baseline the metrics. You cannot prove an improvement you never measured.
Find the single biggest pain point and fix it with a quick win. Prove it with the metric you baselined. Bank the visible, uncontroversial success before touching anything invasive.
Introduce GitOps once CI and containers are stable. With Argo CD, every change becomes a reviewable pull request, every state is versioned and instantly revertible, and the cluster stops drifting from what Git says — the four OpenGitOps properties (declarative, versioned, pulled, continuously reconciled) doing the work.
Add progressive delivery last. Encode a canary that pauses, runs an automated analysis against an error-rate query, and advances only if the metric stays healthy. That is the moment releases become genuinely low-drama: the system, not a stressed human at 3 AM, decides whether to proceed.
Invest in the team as much as the tools. Training reliably returns more than buying yet another platform, and it is what converts a blame-and-burnout culture into a blameless, continuous-learning one.

Perfect is the enemy of progress — a small improvement today beats the perfect plan still being refined next quarter.

Where this goes next

The direction of travel is toward fully closed-loop delivery: Git as the declared intent, controllers reconciling reality to it, and progressive-delivery analysis promoting or reverting releases without a human in the path. Once the arc is walked and the metrics are trustworthy, the natural next step is handing more of the promote/rollback judgement to the system — and, increasingly, to AI-assisted operations that reason over the same signals to forecast capacity, flag anomalies on release, and draft the changes that reconcile drift.

None of that can be bolted onto Friday-night deploys later; it compounds on the layers beneath it. The teams walking the roadmap now are not just escaping the sleepless nights — they are building the legible, declarative, well-instrumented substrate that the next generation of autonomous delivery tooling will need in order to be trusted at all. That is the real reason to walk the arc rather than leap it: each rung you lay down is what makes the next one, human or machine, safe to stand on.

Sources & further reading

Ansible playbooks and GitHub Actions — the script-automation and CI/CD rungs of the arc.
Argo CD docs and OpenGitOps principles — declarative, Git-as-source-of-truth delivery.
Argo Rollouts — Canary strategy — automated, analysis-gated progressive delivery.
DORA, DORA metrics — the baseline metrics that prove each phase actually helped.
A longer reference treatment of the full manual-to-GitOps learning path — the four-phase roadmap, tools comparison, and timeline/budget detail.

Trunk-Based Development vs GitFlow: The Principle Is to Match Your Branches to Your Cadence

Mikhail Dorokhovich — Fri, 07 Aug 2026 08:53:57 +0000

The problem in context

A lot of repositories are museums of good intentions. Feature branches live for weeks, sometimes months, drifting further from the mainline every day until merging one means a full day of conflict archaeology. main has no protection, so a direct push can — and eventually does — destabilize a release. Hotfixes go out and never get back-merged, so the same bug reappears two releases later. And because unrelated features get mixed into a single release branch, no release is ever truly reproducible.

The reflex is to blame Git, or to reach for whatever branching model a louder team swears by. But the underlying issue in the perennial trunk-based development vs GitFlow debate is subtler: a branch in Git is just a lightweight pointer to a commit, so creating, merging, and deleting branches is cheap — which means long-lived branches are a choice, not a necessity. The teams drowning in conflicts are almost always shipping often while branching as if they shipped quarterly. That mismatch, not the tool and not the model, is the problem.

The principle

The principle here is that branch lifetime should track shipping frequency: the more often you ship, the shorter your branches must live and the stricter your CI gates into the mainline must be. Pick the model to match the cadence, rather than inheriting one by accident.

That reframe dissolves the "which model is correct" argument, because branching models have a clear arc and each fits a different cadence. From 2008–2012 it was GitFlow with many long-lived branches; 2013–2018 simplified to GitHub Flow; from 2018 onward the industry moved to trunk-based development with short branches and feature flags. GitFlow is not wrong — its own author now notes it suits versioned software with multiple supported releases rather than continuously deployed web apps. If releases are infrequent and you run many parallel supported versions, its explicit release and hotfix branches are genuinely clearer. If you ship frequently, the answer is trunk-based plus feature flags plus strict CI. There is a detailed treatment of choosing and running a branching model that walks the full decision; the compressed rule is: cadence dictates model.

The second, deeper principle is to know what branches are actually for, because every merge-hell symptom is one of their benefits inverted. Branches buy exactly four things:

Risk isolation — unfinished work does not break the stable line. (Long-lived branches invert this by drifting.)
Parallelism — features, releases, and hotfixes proceed at once.
Quality control — PRs, mandatory reviews, green-CI gates. (An unprotected main inverts this.)
Traceability — commits, tags, and release notes document exactly what shipped. (Mixed release branches invert this.)

Reframing the goal as "preserve those four properties" is what makes the specific rules feel principled rather than arbitrary. It also fixes the vocabulary: main is the always-deployable, protected production source of truth; develop is an optional integration buffer for sprint cadences; feature/* is short-lived and one-PR-per-goal; release/* is stabilization only (fixes and version/CHANGELOG bumps, never new features); hotfix/* branches off main and integrates back into both main and develop.

Trade-offs

The choice is not binary good-vs-bad; it is a fit question. The honest comparison:

Dimension	GitFlow (long-lived branches)	Trunk-based (short branches + flags)
Best-fit cadence	Infrequent, versioned releases; multiple supported versions	Frequent / continuous deployment
Merge cost	Grows with branch age — conflict archaeology	Small and frequent
Release reproducibility	Clear via explicit release branches	Clear via trunk + tags + flags
Incomplete work	Hidden in a long branch	Merged behind a feature flag
CI strictness required	Moderate	High — the mainline gate is load-bearing
Main failure mode	Parallel branches drift out of sync	Undisciplined flags accumulate as debt

The trade-off worth stating plainly: trunk-based buys small merges and fast flow, but only if you pay for it with strict CI and feature-flag discipline — flags are what make merging incomplete work safe, and a flag with no owner or expiry is future debt. GitFlow buys clean parallel-version management, but the bill arrives as merge cost that scales with branch age. Choosing trunk-based while keeping weeks-long branches gives you the costs of both and the benefits of neither.

How to adopt

Do not adopt a whole new model overnight — that is the failure mode, and it invites the resistance you fear. Sequence by leverage.

Protect main first. Require reviews and green CI; forbid direct pushes. This is the single highest-payoff change: the moment direct pushes are impossible, "unstable release" incidents essentially stop.
Shorten feature branches. Aim for hours to a couple of days. Add a lightweight branch-age report to CI that flags any feature branch older than a few days, so drift becomes visible before it becomes painful. Making the invisible cost visible changes behaviour more than any policy memo.
Adopt one release lifecycle and walk every release through it. Preparation branches off a clean mainline:

git switch develop && git pull --ff-only
git switch -c release/1.8.0
git commit -am "chore(release): bump to 1.8.0 & update CHANGELOG"

The release merges into main with a no-fast-forward merge and an annotated tag, then — the step teams skip — back-integrates into develop so nothing is lost:

git switch main && git pull --ff-only
git merge --no-ff release/1.8.0 -m "release: 1.8.0"
git tag -a v1.8.0 -m "Release 1.8.0"
git push origin main --tags

Hotfixes follow the same discipline in miniature: branch off main, fix, merge back into both main and develop. Skipping that final back-merge is the exact bug that resurrects old defects. This release/hotfix structure is the part worth keeping from Vincent Driessen's original model even as you shorten everything else toward trunk.

Make the good path the quick path. Standardize a handful of commands and aliases so the right thing is also the easy thing:

git switch -c feature/login
git fetch --prune
git pull --ff-only
git switch -c release/2.0.0 develop
git switch main && git merge --no-ff release/2.0.0 && git tag -a v2.0.0 -m "Release 2.0.0"

Drill one rule hardest: never rebase a published branch — rewriting shared history breaks teammates; for public branches prefer merge --no-ff. Pair that with Conventional Commits and auto-generated changelogs and semantic releases come almost for free.

The cultural surprise is that resistance rarely materializes once aliases and protected main are in place. Nobody enjoys conflict archaeology, so a workflow that quietly removes it sells itself after the first clean release. You do not mandate discipline so much as remove the friction that was rewarding the bad habits.

Where this goes next

The direction of travel is trunk-based plus feature flags, branch protection expressed as policy-as-code, and automated releases that compress the cycle to hours. Flags are the piece that makes short branches safe — you merge incomplete work behind a flag instead of hiding it in a long-lived branch — and combined with canary or blue-green delivery, branching stops being a source of risk and becomes bookkeeping.

The further horizon is that as more of the commit-to-release path gets handled by automation and AI-assisted tooling — bots that open, review, and land small changes; agents that assemble release notes from conventional commits; policy engines that gate merges on live quality signals — the value of a short, linear, well-tagged history compounds. Automated systems reason far more reliably over a clean trunk than over a thicket of drifting branches. The teams that match their branching to their cadence today are the ones whose history will be legible enough for the next generation of tooling to safely act on.

Sources & further reading

Vincent Driessen, A successful Git branching model — the original GitFlow, plus the author's later note on when it no longer fits.
Trunk-Based Development — the short-lived-branch model and why it scales with shipping frequency.
Conventional Commits — the commit convention that powers auto-generated changelogs and semantic releases.
Pete Hodgson, Feature Toggles — how flags make short branches and trunk-based work safe.
A longer reference treatment of choosing and running a Git branching model — every command and the GitFlow-vs-trunk decision for a given cadence.

The Release Automation Business Case: The Principle for Getting Delivery Work Funded

Mikhail Dorokhovich — Thu, 06 Aug 2026 09:29:18 +0000

The problem in context

Engineering asks for investment in delivery automation and gets a polite no — repeatedly, and usually for the same reason. The pitch walks into the room talking about GitOps, pipelines, and Kubernetes, and watches the executives' eyes glaze over. It is answering a question nobody in that room asked. To leadership, release management reads as an IT cost center, a necessary evil, not a lever on the business. So the failed release automation business case is almost never a failure of the underlying work; it is a failure of translation.

This matters because the gap is asymmetric. The engineers know the automation is valuable and cannot understand the no; the executives are not being obtuse — they simply have no line of sight from "pipeline" to anything on their own scorecard. Until someone builds that line of sight, the money stays where the budget-holder can already see the return.

The principle

The principle here is that a delivery investment has to be argued in the units the audience already uses to rank the world, and for a budget-holder those units are revenue, risk, and operating cost — in that order. Modern release orchestration is not a technical nicety; the speed and reliability of shipping software directly determine a company's ability to capture markets and retain customers. That is not a slogan — it is the central finding of DORA's multi-year research program, which has repeatedly shown that software delivery performance predicts organizational performance. That sentence is something a CFO can act on. "We need GitOps" is not.

The mental model that makes the translation mechanical is that every delivery metric is a business outcome wearing an engineering costume. The signals DORA tracks as the Four Keys, which Google Cloud packaged into the open-source Four Keys project, restate cleanly into money and risk:

Lead time → revenue. Getting products in front of customers sooner; the headline framing is up to a +23% revenue effect from faster delivery.
Change failure rate → risk. Dramatically fewer operational incidents — on the order of an 85% reduction — which is avoided downtime cost.
Time to restore → risk and cost. Recovery in minutes not days; and automation-driven utilization cuts infrastructure OpEx by roughly 40%.

Lead the conversation with their ranking — revenue first, risk second, cost third — because that is the order the person holding the budget already thinks in. There is a detailed treatment of the evolution and business case for release automation with the full executive summary and metrics table; the compressed principle is: translate before you present, and order by their priorities, not yours.

The single most effective artifact is a plain table mapping each metric from current state to target to business effect — abstractions do not get funded, concrete deltas do:

Metric	Before	After	Business effect
Release frequency	Quarterly	Daily	Ship value continuously
Change lead time	2–6 months	Under 1 day	~98% faster
Time to restore	1–7 days	Under 1 hour	~95% faster recovery
Failed releases	15–30%	0–5%	~83% fewer bad releases

Each row is a business outcome in disguise. "Lead time from months to a day" is really "we can respond to a competitor or a regulation in a day." Pair it with one real company's shape — a fintech with $50B in assets that cut time-to-market from 8 months to 6 weeks and booked +$15M in first-year revenue — so the percentages have a concrete silhouette behind them.

Trade-offs

The business case is itself a design with trade-offs, and getting them wrong is how a strong underlying investment still gets a no:

Choice in the pitch	Weaker option	Stronger option	Why
Framing	Technology capabilities	Business outcomes	The budget-holder scores outcomes, not capabilities
Evidence	Architecture diagrams	Before/after deltas + one real case	Concrete deltas get funded; abstractions do not
"Is this a fad?" objection	Left unanswered	A 25-year industry arc	Makes the direction feel inevitable, not speculative
Payback	Unmentioned	12–24 month window, quantified	Reframes "can we afford it?" as "can we afford not to?"
Rollout risk	Big-bang transformation	Instrumented pilot on one service	A measurable small win is an easy yes

The honest cost of doing this well is that translation takes real work and some numbers are necessarily estimates — overclaim and you lose credibility on the first missed target. So the discipline is to bring ranges, name the assumptions, and let one real case carry the specificity the estimates cannot.

The "is this a fad?" objection deserves its own move, because it sits under every budget no. Defuse it with the arc the industry has already traveled: the heroic-scripts era of 2000–2005 (6–12 hour releases, 30–50% rollbacks, downtime at $100K–$1M+ per hour), the first automation wave of 2005–2012, the CI/CD revolution of 2012–2020 — the inflection Martin Fowler frames as Continuous Delivery, keeping software deployable at any time — and the 2020s move to GitOps, IaC, and business-metric-driven rollout. Presented this way, the ask stops sounding like a gamble on new technology and starts sounding like catching up to where elite teams already operate. No leadership team wants to be running 2005-style releases in 2026.

How to adopt

If you are stuck getting funding for delivery automation, stop refining the architecture slide and build the case in this order.

Reorder by their priorities. Open with revenue and risk, close with cost. The same content in the wrong order still reads as an IT expense.
Build the before/after table with your real current-state numbers — you need the baseline anyway to prove the improvement later.
Bring one real company's numbers so the abstractions have a shape, and present the 25-year arc so the direction feels inevitable.
Anchor to a payback window. Be honest that this is an ongoing capability, not a one-time purchase, and set the expectation at 12–24 months depending on scale.
Propose an instrumented pilot on a single non-critical service, measured with the same three metrics, before the full rollout. A funded pilot with a measurable result is a far easier decision than a big-bang transformation — and the delta it produces makes the second, larger ask almost automatic.

Notice the throughline: the technology was never the hard part. The hard part is translating lead time, change failure rate, and MTTR into revenue, risk, and OpEx, and ordering them the way the budget-holder ranks the world.

Where this goes next

The most durable version of this argument positions the automation as a foundation rather than a feature. The same pipeline discipline that speeds releases today is the substrate that AI-assisted operations, predictive capacity forecasting, and zero-touch pipelines will run on tomorrow — none of which can be bolted onto a manual release process later. That reframes the question one final time, from "can we afford this project?" to "can we afford to still be building on 2015 foundations when our competitors are running on 2025 ones?"

The forward-looking case is that delivery capability is compounding, not linear: each layer of automation lowers the cost of the next, and the teams funding it now are buying the option to adopt whatever the next wave of AI-driven operations turns out to require. The business case you make for release automation today is, increasingly, the business case for being able to adopt anything at all at the speed the market will demand.

Sources & further reading

DORA research program — the longest-running study linking software delivery performance to organizational and business outcomes.
DORA, DORA metrics (the Four Keys) and Google Cloud, Using the Four Keys to measure your DevOps performance — the throughput and stability signals behind the ROI story.
Martin Fowler, Continuous Delivery — the capability the investment actually buys.
A longer reference treatment of the evolution and business case for release automation — the executive summary, metrics table, and ROI framing behind this argument.

Blue-Green vs Canary Deployment: The Principle Is to Stop Choosing and Start Combining

Mikhail Dorokhovich — Wed, 05 Aug 2026 10:05:14 +0000

The problem in context

The releases that hurt most are the ones where a team has to decide, live, whether things are bad enough to roll back. Error rate looks a little high — is that the new version or normal noise? Nobody agreed on a threshold in advance, so the argument happens in the incident channel while users suffer. That single failure mode — rollback as a live debate — is what a deployment strategy exists to prevent, and it is why the perennial "blue-green vs canary deployment" argument is usually the wrong frame.

Treated as a ranking problem, it produces a stalemate: blue-green people cite instant rollback, canary people cite limited blast radius, and the team picks one and inherits the other's weaknesses. The reframe is that these are not rivals to rank. They are tools that answer three different questions — how do we roll out, how do we roll back, how do we limit blast radius — and a real delivery system needs a different answer at different blast radii.

The principle

The principle here is that deployment strategy is blast-radius management, and different releases have different blast radii. Once you accept that, the design stops being "pick the one right strategy" and becomes "assemble a layered system where each tool covers the radius it is best at." A combination almost always wins: trunk-based development plus feature flags plus canary for small daily releases, and blue-green for the big drops where instant, explainable rollback is the whole point. A detailed treatment of choosing and combining strategies walks the full model; the compressed version is four tools with four jobs.

Blue-green keeps two identical production environments — the pattern Martin Fowler documented in 2010. Blue serves users; green is prepared calmly for the next release; when you are confident you flip traffic in one router change, and if something is wrong you flip back just as fast. Its virtue is that rollback is instant and trivial to explain to a business stakeholder — a fintech spotting a EUR-payments bug and flipping back in about two minutes, with a tiny fraction of operations affected, is the kind of story that sells it to leadership.

Canary rolls a change out to a small subset first — what Danilo Sato describes on Fowler's site — starting at 1–5% of traffic, watching errors, latency, resource use, and one business metric, then ramping 5 → 10 → 25 → 50 → 100% and comparing at each step. Its virtue is a naturally small blast radius: a bad version is seen by few before the system reacts.

Rolling updates are the quiet workhorse for stateless services — the default strategy in the Kubernetes docs, where maxSurge and maxUnavailable govern how aggressively pods are replaced while the service stays available throughout.

Feature flags are the one that changes cadence most, because they separate two actions people wrongly treat as one: deploying code and releasing a feature. Ship dark, enable for staff, then a percentage, then everyone, and kill the switch without redeploying — the technique Pete Hodgson documents as feature toggles, which pairs naturally with trunk-based development.

The unifying move that ends the 2 AM arguments is writing stop-criteria before the release, as executable rules rather than live judgement:

IF error_rate_canary > error_rate_baseline * 1.5 THEN rollback
IF latency_p99_canary > latency_p99_baseline * 1.3 THEN rollback
IF conversion_rate_canary < baseline * 0.95 THEN rollback

With those in place, rollback stops being a debate and becomes a reflex — the system pulls the new version before a human opens the dashboard. Netflix's canonical case is exactly this: a gradual ramp where SmartTV performance degraded at one step and triggered an automatic rollback in roughly two minutes.

Trade-offs

No strategy is free, and the honest way to reason is per blast radius:

Strategy	Best for	Rollback	Real cost
Blue-green	Big, risky drops (payments rewrite, framework upgrade)	Instant single flip	~2× resources during release; DB and session state are hard; watch DNS TTL
Canary	Everyday releases	Automatic on stop-criteria	Needs percentage routing; slow to reach confidence on low traffic; pin users by ID
Rolling update	Stateless services on Kubernetes	Gradual, batch by batch	Old and new pods coexist — API contracts must stay compatible across versions
Feature flags	Decoupling deploy from release	Kill switch, no redeploy	Forgotten flags rot; every flag needs an owner and an expiry

Two constraints deserve emphasis because they are where teams get burned. First, the rolling-update coexistence problem: because old and new pods serve traffic simultaneously, forward-compatible database and API changes are not optional — deployment strategy and schema evolution are two halves of one discipline. Second, a flag with no owner is technical debt with a fuse on it; the discipline that makes flags safe is an owner and an expiry date on every one, with dead code removed once the feature is universal.

And one distinction worth nailing down, because conflating them wastes time: canary is not A/B testing. Canary asks "is the new version not worse?" and optimizes for safety. A/B testing asks "which version is better?" and optimizes for a product decision — split traffic, pin each user to a variant, and wait a week or two to smooth out weekday and seasonality effects before running the stats. Use canary to protect releases and A/B tests to choose product directions; using one for the other's job produces noisy, untrustworthy conclusions.

How to adopt

The mistake is adopting all four at once. Sequence by where the pain is loudest.

Write the stop-criteria before the next release and make them executable. This is the single highest-leverage move; it converts rollback from a live argument into an automatic reaction and costs almost nothing.
Add feature flags to decouple deploy from release. Ship dark, enable gradually, keep a kill switch. Put an owner and expiry on every flag from day one so the debt never accumulates.
Make rolling updates safe on Kubernetes with small batches, health probes, and a pause between batches:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 2
  template:
    spec:
      containers:
      - name: my-app
        image: my-app:v2
        readinessProbe:
          httpGet: { path: /health/ready, port: 8080 }
        livenessProbe:
          httpGet: { path: /health, port: 8080 }

Batches too large increase risk and make rollback painful; skipping the pause means you notice problems too late.

Reserve blue-green for the big drops where instant, explainable rollback justifies double the resources. Solve the session problem with a shared session store and graceful connection draining, and watch DNS TTL, which can delay a switch you thought was instant.
Automate the canary gates rather than hand-rolling them. Argo Rollouts encodes exactly these gates — setWeight and pause steps plus automated analysis that advances the traffic weight only while metrics stay healthy.

Put together, a normal day looks like this: a change merges to trunk, deploys dark behind a flag, and enables as a canary to 5% while the automated gates watch; if they stay green it ramps to 100% over an hour, and if not the flag flips off while the code stays deployed but inert. A large, risky drop reaches for blue-green instead. Each tool covers a different blast radius, and together they mean no release requires heroics.

Where this goes next

The direction of travel is toward progressive delivery as a fully automated control loop — the stop-criteria you write by hand today become the analysis templates your rollout controller evaluates on its own, promoting or reverting without a human in the path. Tools like Argo Rollouts and Flagger are early forms of this; the interesting frontier is richer signals feeding the gate, including model-based anomaly detection that can catch a regression no static threshold would.

The deeper point is that all of this rests on the same foundation: a legible definition of "healthy" and forward-compatible changes underneath the traffic shifting. Teams that have captured what "not worse" means as executable criteria are the ones who will safely hand more of the promote/rollback decision to automation — and eventually to AI-assisted release agents that reason over the same signals. The layered strategy is not just what ends the 2 AM rollbacks today; it is the substrate the automated release systems of the next few years will need in order to be trusted at all.

Sources & further reading

Martin Fowler, BlueGreenDeployment and Danilo Sato, CanaryRelease — the canonical write-ups of both patterns.
Kubernetes docs, Deployments — the RollingUpdate strategy with maxSurge/maxUnavailable.
Argo Rollouts — Canary strategy — automated canary analysis and traffic-weight gates.
Pete Hodgson, Feature Toggles and Trunk-Based Development — decoupling deploy from release.
A longer reference treatment of choosing and combining deployment strategies — the sequences and stop-criteria behind this framing.