Count the items on your deployment checklist. If there are more than a handful, the list is not telling you that your team is careful. It is telling you how much infrastructure your product team still owns.
That is the argument of this article. A deployment checklist earns its length from application risk: the ways your own code, data, and rollout can hurt users. Every other item — certificates, load balancers, image provenance, autoscaling, connection draining — is a piece of plumbing you are verifying by hand because nothing else verifies it for you. Those items do not need better wording. They need an owner, and the question worth asking is why that owner is still your product team.
This assumes you are already running applications in production, with a pipeline, a rollback story of some kind, and an incident history worth reading. The question is not how to start deploying safely. It is which of the risks you currently carry are actually yours.
Two Kinds Of Risk On One List
Open your current checklist and sort every item into two columns.
The first column is application risk. Can this schema change be undone? Will the old consumer understand the new job payload? What happens to sessions already in the cache? These come out of decisions your team made in a pull request. Nobody outside the team can answer them, because nobody else knows what your code means.
The second column is infrastructure risk. Is the certificate valid? Did the image come from the right commit? Are autoscaling rules correct? Are logs shipping? Is the previous version still around? These are recurring platform responsibilities, not application-specific judgment calls. Every production team faces them, which is why they should be enforced consistently instead of rechecked by hand.
The first column is a checklist. The second is a list of things that should be answering themselves. Most teams write both into the same document, then wonder why it is too long to use.
Column One: Build It From Your Own Incidents
Do not copy the application half of the list from a blog post, including this one. Build it from the specific ways your system has already broken.
Pull your last ten incidents. For each one, ask what single check, run before the deploy, would have caught it. A pattern appears fast. Most teams find three or four causes explain the bulk of their outages: bad config, an unsafe schema change, a dependency that was not ready, a rollback that did not work.
Four items you use beat forty you ignore. Google's SRE team makes a related point in its chapter on release engineering, where the goal is releases that are boring and repeatable rather than careful and heroic. A working postmortem practice keeps the list current, since every incident should add an item or delete one.
One structural warning. Any item that depends on a human remembering will eventually fail, because people get tired and pipelines do not. Each item should be automated, or explain why it cannot be. The ones that cannot be automated are usually judgement calls about your data model, which is exactly what belongs in column one.
The Application Questions Worth Asking
Add these to whatever your incidents produced. They cover the failure modes that hit nearly every team, and all of them are about your code rather than your plumbing.
Can I undo this in under two minutes? Not whether a rollback plan exists on paper. Whether it has been tested this week, on this service. An untested rollback is a hope, not a control.
Is the config this code needs already in place? A new variable must be set before the code that reads it arrives, not after. The twelve-factor approach keeps config in the environment rather than the code, which makes the ordering explicit.
Is the database change safe on its own? That one gets its own section below.
Does the app report its own health honestly? A running process is not a working app, so the health endpoint has to check what the app actually depends on. Kubernetes formalises the distinction with readiness and liveness probes: readiness decides whether traffic reaches the instance, liveness decides whether it gets restarted. Confusing the two turns a slow start into an endless restart loop. Writing an honest health check is application work. Checking by hand that the rollout waited for it is not.
How many users see it first? Shipping to everyone at once turns every mistake into a full outage. Shipping to five per cent turns most mistakes into a blip.
None of these ask whether the code is correct. That is what review and tests are for. A deployment checklist asks a narrower question: can delivering correct code still hurt you?
Three Application Risks That Are Rarely On The List
These catch experienced teams, because nothing in the pull request hints at them.
Background workers are the first. Web processes and queue consumers ship from the same codebase but not always at the same moment. For a few minutes, a new producer may write payloads an old consumer cannot read. Those jobs fail quietly into a dead letter queue and get found hours later. Payloads need the same backward compatibility rule as database columns.
Serialised data is the second. Anything written into a cache, a session store, or a message body is effectively a schema. Change the shape of a cached object and old readers break, even though no database was touched. Use a new cache key instead.
Cold-start capacity is the third, and it sits on the boundary. A rolling deploy briefly serves normal traffic with fewer warm instances. If your app needs forty seconds to warm its connection pool and the grace period is thirty, you find out at the worst time. Knowing the warm-up time is application work. Enforcing it on every deploy is a configuration you should set once, not a line you reread each time.
Database Migrations Are Where Rollbacks Go To Die
You can roll back code in seconds. You cannot roll back a dropped column. No tooling anywhere will make this someone else's problem, which is why it is the strictest item on the list.
Never ship a schema change and the code that depends on it in the same deploy. Split it. Add the new column and leave the old one alone, then deploy. Ship code that writes to both. Ship code that reads the new one. Only much later, once nothing references it, drop the old column.
The pattern is called expand and contract, and it is more work. It also means that at every step, old code and new code can both still run, and that property is what makes fast rollback possible at all. Blue-green deployment keeps two full environments and switches traffic between them, which only works if your data layer serves both versions. Migration discipline is the foundation; the deploy strategy sits on top.
One more rule. Never run a long migration inside the deploy. A change that locks a large table takes your app down while the pipeline reports success. Run it ahead of time and let the deploy be a fast swap of code.
Separate Deploying From Releasing
Deploying means the code is on the server. Releasing means users can reach the behaviour. Most teams treat these as one event, which is why every deploy feels risky.
Put new behaviour behind a flag and the two come apart. Deploy with the flag off, confirm the app is healthy, then enable it for internal users, then one per cent of traffic, then everyone. If something looks wrong you flip the flag instead of redeploying, and recovery takes seconds.
This also shortens the list. A change behind a default-off flag really is low risk, so it can skip most items, and the checks move to the flag flip where the risk lives. Flags cost something too, since old ones become dead code, so set a removal date when you add one.
Decide Your Rollback Trigger Before You Ship
Rollbacks happen late mostly because nobody agreed in advance on what "bad" looks like. Under pressure, people negotiate and wait one more minute to see if the errors settle.
Write the trigger down before the deploy. Pick two or three numbers, each with a threshold and a window. Error rate above one per cent for two minutes. Latency at the ninety-fifth percentile above double its normal value. Queue depth climbing without recovery. Roll back when any one is crossed, with no discussion.
Watch percentiles, not averages. If one request in twenty now takes eight seconds, the average barely moves while a real share of users suffer.
Then be honest about detection. How long would it take you to notice each signal today? If that is longer than your watch window, the problem is not your checklist. It is your monitoring.
Column Two: Take It Off The List
Now go back to the infrastructure column and ask a different question of each item. Not "is this worth checking" — it is — but "why is a person checking it?"
The obvious answer is to automate. Certificate renewal becomes automatic. Image provenance becomes a pipeline control that refuses to promote an unverified build. Config drift becomes a deployment-time check. Each line you convert genuinely leaves the list, and a team that does this for a quarter will have a visibly shorter document.
Then look at what is left after the easy passes, because the remainder is where the argument actually is. Health-gated rollout, where traffic shifts only after new instances pass their checks. Zero-downtime replacement: draining connections, waiting for in-flight requests, sequencing instance turnover, honouring readiness before moving traffic. Keeping the previous release built and warm enough to return to in seconds. A preview environment per pull request, built the way production is built, so that "it worked in staging" stops being a category of incident.
These do not respond to a scripting afternoon. Each is a distributed systems problem with edge cases you discover in production, and each stays on the list as a human verification step until it is fully solved. Teams that commit to solving them properly are not writing scripts anymore. They are building and staffing an internal delivery platform, which is a real product with a roadmap, an on-call rotation, and a maintenance cost that does not end. The work is legitimate. It is also the same work, done once per company, that a platform as a service has already amortised across everyone using it, with the edge cases found by someone else's outages.
Rollback shows the gap most clearly. Owned in-house, it is a procedure a person performs under pressure, and the only thing that makes it trustworthy is rehearsing it often enough that the rehearsal itself becomes a standing cost. Owned by the platform, the checklist item collapses to confirming that the previous release exists — which the platform also guarantees. The reliability difference is not a matter of how well the runbook is written.
There is a real limit to this. Opinionated platforms constrain unusual networking, specialised hardware, and fine-grained tuning of the layers they abstract, and teams with those requirements will hit the edges and should stay where they are. It is worth checking honestly whether that describes you, because most teams shipping web applications and APIs assume it does long before it is true.
Keep It Short, Keep It In The Repo
What survives is a list about your application, and it fits in a pull request template. Six tick boxes in front of the reviewer beat a wiki page every time.
A workable version reads like this. Confirm the change is backwards compatible with the running version, including job payloads and cached data. Confirm any schema change ships in an earlier deploy. Confirm new config already exists in the target environment. Name the rollback trigger. Say how the change reaches users. Name who watches for the first fifteen minutes.
That last item is underrated. A deploy nobody is watching is a bet, which is also why shipping at the end of the day hurts so often. The deploy is fine. The empty room is the problem.
Then measure whether the list works. The four DORA metrics give you the read you need. Change failure rate says whether your checks catch real problems, recovery time says whether your rollback story is honest, and deployment frequency says whether the process has grown too heavy to use. The teams that deploy most often break production least often, because speed and safety both come from a delivery path so automated that little is left for a human to get wrong.
So treat the checklist as a running record of risks a human is still absorbing on the system's behalf. Automate what automation can reach, and for the rest, be clear-eyed that keeping the item means keeping the infrastructure that produced it. A shrinking checklist is a maturing system. A list that stays long is a decision, not an accident.
Top comments (0)