The classic change advisory board — a weekly meeting where changes are read aloud to people who don't operate them — deserves its reputation. In small teams it decays into one of two failures: forty changes rubber-stamped in forty minutes (a queue wearing a meeting's clothes), or no review at all until the Friday migration melts the billing database.
Both miss the point. A change review exists to answer, before merge, five questions that predict most change-induced incidents. Answer them in writing, in the pull request, and the board dissolves back into the work.
You can read the full note on the HIVE80lab ops-notes site — here is the working version.
The five-question change card
One line each, in writing. The writing is where the thinking happens.
- What breaks if this fails? — the blast radius in customer terms: orders cannot be placed, not the orders service errors.
- How do we undo it? — named and real: revert PR #482 and run down-migration-07.sql. "Re-deploy the previous version" is an answer only if you've done it recently. An untested rollback is a hope.
- Who is on-call when it ships? — the change ships inside someone's shift, and that person should know it's coming.
- What does it touch that customers feel? — this selects the risk class, and with it the scrutiny.
- When does it ship, and why then? — timing is a decision. A schema change at 4:55pm Friday is also a decision.
Risk classes: short, or pointless
- Class 1 — data (migrations, backfills, deletes): irreversible, so the undo is tested on a copy of production data before merge.
- Class 2 — auth and permissions: fails silently. A broken role produces no error, only a customer who cannot work.
- Class 3 — money paths (billing, payments): doubled and missed charges both count as failures, and both surface in support first.
- Class 4 — everything else: ship it, mention it in the deploy log.
Keep the list short by force. When most changes are "high risk," the label stopped measuring risk and started measuring politics — and reviewers stop reading, which is how the one genuinely dangerous change slips through wearing the same label as the other thirty-nine.
The meeting, replaced
Under ~20 people, the review is async: the five answers are required PR fields, and Class 1–3 changes need one named second approver (rotate it — a single permanent approver is a single point of failure with a calendar invite).
What survives as a meeting is a 10-minute weekly slot with one agenda: the changes that failed or nearly failed this week. Not re-approving the week's changes — reviewing the process that let the near-misses through. Someone says "the backfill's undo answer was 'we hope it's fine' and we merged anyway," and the template gets fixed instead of the person blamed.
For 3am decisions, pre-write the authority: a standing-orders ladder answers who decides when the decider is asleep.
The five traps
Approval theater. Forty changes approved in forty minutes is a signature factory. If reviewers can't remember what they approved yesterday, the board reviews nothing.
The emergency loophole. If urgent changes bypass review, "urgent" becomes a deadline costume and the board only ever sees the safe changes. Repair: emergency changes get reviewed retrospectively, within 48 hours, in the weekly slot. Not as good as before; infinitely better than never.
Review by people who can't operate the thing. Include the on-call who will catch the pager. They've usually seen this system fail before; the board usually hasn't.
The freeze as the only tool. A change freeze is for exceptional windows. As a substitute for review it backfires: changes queue behind it and ship in one avalanche when it lifts.
No record. Verbal approvals leave nothing. One line in the deploy log — what shipped, who approved, what class — is what a post-mortem starts from on Thursday.
Worked example
A nine-person logistics SaaS runs the classic CAB: Thursdays, 45 minutes, 23 changes, all approved. The approvals mean so little that a Thursday-approved database migration ships Friday with a rollback script that has never been run — which references a column renamed two migrations ago. Eleven hours of recovery.
The redo takes one afternoon: five required fields in the PR template, four classes defined, Thursday shrinks to a 10-minute failures-only review. A month in, the card catches two backfills whose honest undo answer was "we can't" — they die in a comment thread instead of production. Quarter over quarter: change failure rate 31% → 11%, meeting time 45 → 10 minutes.
The counter-example is the sister team that answered the same incident class with a six-week freeze: changes piled up behind it and shipped in one three-week burst of big-bang releases that out-produced the incidents the freeze prevented.
From the HIVE80lab kit
- Score your incident response free — 12 questions, tiered fix-first plan
- The First 30 Minutes — free incident quick-start
- Ops Starter Kit — incident response for small teams — $14
- Ops Starter Kit Vol. 2 — advanced incident response & communications — $27
- Ops Mega Bundle — all 5 kits in one download — $49
Related: the deployment rollback checklist is what question two commits you to, the change-freeze window policy covers the exceptional windows, and the incident post-mortem template is where a failed change gets its receipt.
Top comments (0)