Nobody breaks prod on purpose. It happens on a Tuesday when someone "just fixes" the nginx timeout on the live box, plans to update the repo "later," and later becomes never. Six weeks and four similar fixes later, the server that gets rebuilt from the repo is a different animal than the one taking traffic — and the first person to find out is whoever runs the restore or the failover at 2am.
That's config drift: the gap between what your repository says your infrastructure is and what it actually is. It's invisible right up until the moment it's the only thing that matters — a rebuild, a failover, a restore, a new hire following the docs.
Why drift happens (it's not carelessness)
- Pressure. The incident is now; the PR is later. Every engineer alive has hotfixed prod with good intentions.
- No single source of truth. Config scattered across a repo, a wiki, a Confluence page from 2021, and three people's memories.
- Manual steps that nobody scripted. "Edit the config file on the box" is a step that exists outside git, which means outside history.
The fix isn't a policy memo. It's a habit with a short loop.
The 5-minute Friday drift check
Run this every Friday, or automate it and never think about it again:
1. Pick one critical service. Not all of them — one. The one that bills customers or serves users.
2. Diff live config vs. repo config. Literally: copy the live config file, copy the repo version, run diff. Any line that differs is either (a) an undocumented fix or (b) a bug waiting for a rebuild. Both need an answer.
3. Ask for each diff: should this be in the repo? If yes — commit it this week. If no — fix the live side to match the repo. Either way, the gap closes. Drift that's discovered is cheap; drift that's discovered during a rebuild is a 2am incident.
4. Write the one-line answer down. "2026-XX: bumped timeout to 90s in prod only — committed to repo 2026-XX." Six words now saves an archaeology project later.
The escalation path: from habit to automation
Once the weekly diff is a habit, automate it:
- Config-as-code minimum: every config file for your critical service lives in the repo, and the repo version is treated as the truth. Live changes require a PR — even under pressure, especially under pressure.
- Scheduled drift detection: a weekly job that diffs live vs. repo and posts the result to a channel humans actually read. A drift alert nobody sees is just a log line.
- Immutable deploys: the strongest version — never edit config on a box at all; rebuild and redeploy. Then drift becomes impossible instead of merely detectable.
You don't need the full platform on day one. You need the diff.
The 2am test
Here's how you know your drift status is acceptable: imagine the box dies right now and a competent teammate rebuilds it from the repo at 2am. Would it come back up? Would it come back up with the same behavior as the box that just died?
If any answer is "no," that gap is a date with destiny. Friday is cheaper.
The Automation Starter Pack includes ready-made scheduled-check scripts (drift detection, disk, service monitors) for small teams that can't staff a platform team. Pair it with the Ops Starter Kit for the runbook and audit side, and the Agent Ops 24/7 playbook for running 24/7 monitoring without a night shift. Everything lives at Hive80 Lab.
What's the most infamous config drift story at your shop? Wrong answers welcome in the comments.
Top comments (0)