DEV Community

Hive80-lab
Hive80-lab

Posted on

We had no change management process. Then a Friday config tweak cost us the weekend.

We had no change management process. Then a Friday config tweak cost us the weekend.

Nobody on a small team wants a change advisory board. I get it — the phrase alone sounds like a committee meeting with worse coffee. But the alternative we ran for years was the honor system: anyone could ship anything, as long as they felt confident about it.

The honor system failed us the same way it fails everyone. One Friday afternoon, someone tweaked a session timeout "real quick" to help a customer demo. The change worked. The change also silently logged out every user in the billing app — including the two people running that demo — and the rollback instructions turned out to be aspirational fiction. The weekend that followed was longer than the change was quick.

The postmortem asked the classic small-team question: why wasn't this reviewed? And the honest answer was: because review meant a meeting, and meetings are where momentum goes to die. So we didn't build a committee. We built a ten-minute record and four questions.

The 4 questions that replace a CAB

Before any "normal" change (not a routine deploy, not an emergency fix):

  1. What breaks if this goes wrong? Blast radius in one sentence. If you can't name it, you haven't scoped it.
  2. How do we know within 15 minutes that it went wrong? A named alarm, a log query, a manual test. "We'd notice" is not detection.
  3. What's the rollback — and has anyone actually run it? This is the killer question. Rollbacks that were never executed are where bad Fridays become bad weekends. If the answer is "undo the change," you don't have a rollback, you have a hope.
  4. Who's watching it for the first hour? Unwatched changes are discovered by customers, which is the most expensive detection channel there is.

A change with no answer to #3 doesn't ship. That single rule has caught more bad changes for us than any process doc ever did.

The 10-minute change record

One row in a shared doc, closed within 24 hours of shipping:

  • Change: "Billing app: session timeout 30d → 8h"
  • Class / risk: Normal — auth flows touched, ~400 users affected
  • Rollback: SESSION_TTL=30d in config, restart app pool (tested Aug 30)
  • Detection: error-rate alarm >2% for 5 min; manual login test at +10 min
  • Window / watcher: Wed 14:00–16:00, on-call engineer watching
  • Outcome: shipped 14:22, error rate flat, closed same day

That's the whole artifact. Twelve dated records a quarter beats a process nobody maintains — and when an auditor, customer, or insurer asks "do you do change management?", the records are the answer.

The rules that make it stick

  • The requester is never the approver for normal changes. One other pair of eyes, five minutes, async Slack is fine. This rule alone catches half of all bad changes.
  • Emergency changes get a retro-record, not a pass. The rule isn't "no changes without approval" — it's "no changes without a record." Fix forward, write it up within 24h, and the process survives contact with a real incident instead of being abandoned the first time it's inconvenient.
  • Freeze windows by default: Friday after 2pm, the last two days of a quarter, the day before a demo, and any window overlapping an on-call handover. Changes can still ship in a freeze — they just need an approver who isn't the person doing them.
  • Failed rollbacks are finding #1. When a change fails and the rollback also fails, that's two defects, and the rollback one is the expensive lesson. Test rollbacks on boring weeks.
  • Standardize until only normal changes remain. Every change that happens twice becomes a script or a runbook, reclassifies as "standard," and stops costing review time. That's the automation path: fewer human decisions, same safety.

The free version of this — the full checklist, the change record table, and the freeze-window rules — lives on our ops notes site, next to the postmortem and incident-plan templates.

If you'd rather not assemble this yourself: the Ops Starter Kit Vol. 2 ($27) includes the full change-control pack — change records, freeze calendar, rollback log — plus the advanced incident response and comms templates. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.

And for the part where changes stop needing a human at 2am: the Automation Starter Pack ($19) covers the pick-first workflows — the routine, repeatable changes that should never have been manual in the first place.

What's your team's equivalent of the Friday config tweak? The one that shipped on vibes and cost a weekend — I'm collecting stories for the postmortem hall of fame.

Top comments (0)