For the first year of Edilec, shipping a release meant blocking out an afternoon and hoping nothing else came up. A deploy was: SSH into the box, git pull, stop the service, run migrations by hand, restart, tail the logs for ten minutes, and cross your fingers. When it worked, it took maybe forty minutes of actual effort. When it didn't, it ate the rest of the day and usually part of the next one too.
We didn't set out to fix this. We got forced into it by a release that went from "should be quick" to a three-day incident.
The release that broke the calendar
We were shipping a schema change alongside an API update — nothing exotic, just a new column and a service that needed to read from it. The migration ran fine in staging. In production, it ran fine too, technically — it just ran for eleven minutes on a table nobody had checked the size of in months, holding a lock the whole time. Requests backed up, timeouts cascaded into our queue workers, and by the time someone noticed, we had a partial deploy: half the fleet on the new code, half on the old, both talking to a database mid-migration.
Rolling back wasn't clean, because "rollback" meant someone manually reversing steps they'd typed by hand twenty minutes earlier under pressure, from memory, while alerts were firing. We got the system stable within a few hours. We spent the next two and a half days on cleanup: reconciling data written by mixed-version code, re-running migrations that had partially applied, and writing an incident doc nobody wanted to write.
The postmortem's real finding wasn't about the migration. It was that our entire release process had no repeatable shape. Every deploy was a slightly different set of manual steps performed by whoever was on point that day, which meant every deploy carried the risk of a step skipped, reordered, or misremembered. Speed wasn't really the problem. Reproducibility was.
What we built instead
We rebuilt the pipeline around one rule: nothing in a release should depend on a person remembering to do it correctly under time pressure.
One artifact, tested once. CI builds a single versioned, immutable artifact per commit. That artifact — not source, not a branch — is what moves through every environment. If it passed staging, it's bit-for-bit what reaches production. We'd previously had environment-specific build steps that meant "tested in staging" and "running in production" weren't always the same code.
Migrations run and gate separately from deploy. This was the direct fix for the incident. Migrations now run as their own pipeline stage, against a replica first to estimate lock time and row count, and anything projected to hold a lock past a threshold fails the pipeline instead of running blind in production. Schema changes and code changes are sequenced, not bundled — expand, deploy, contract, each as its own step with its own gate.
Progressive rollout with automatic health checks. New versions go to a small slice of traffic first, watched against error rate, latency, and a couple of business-specific metrics for a fixed window. If those stay within bounds, rollout proceeds automatically in increments. If they don't, the pipeline halts and rolls back the slice on its own — no one has to notice and decide, the decision is already encoded.
Rollback is a pipeline stage, not an improvisation. Every release keeps the prior artifact and its migration state addressable, so reverting is running the same pipeline backward against a known-good version, not someone reconstructing steps from memory. We tested this by rehearsing rollbacks on purpose, on a schedule, so the first real rollback wasn't also the first time anyone had run one.
Humans approve intent, not mechanics. A person still decides whether to ship and when. Nobody decides how — how is the same every time, which is the entire point.
What it bought us
Release time went from an afternoon, best case, to about forty minutes end to end, most of which is intentional soak time during progressive rollout rather than anyone doing manual work. That's the headline number, but it undersells the real change: releases stopped being events. We ship several times a day now instead of batching changes into a dreaded weekly window, which paradoxically made each individual release lower-risk, because smaller diffs are easier to reason about and easier to roll back cleanly.
The three-day incident cost us trust in our own process. The fix wasn't more caution — more caution just makes deploys slower and rarer, which concentrates risk instead of removing it. The fix was making the safe way the only way, automatically, every time.
We build this kind of infrastructure discipline into every system we ship at Edilec — if a release process depends on someone remembering the right order of steps, it's not a process yet, it's a hope.
Top comments (0)