DEV Community

Hive80-lab
Hive80-lab

Posted on Originally published at hive80-lab.github.io

The power failure IT checklist: the UPS, the shutdown, and the cold-start order

#sysadmin #security #devops #smallbusiness

published: true
title: "The power failure IT checklist: the UPS, the shutdown, and the cold-start order"
tags: devops, security, smallbusiness, sysadmin

canonical_url: https://hive80-lab.github.io/ops-notes/power-failure-it-checklist.html

Power blips are the incident category every small team assumes is someone else's problem — until the Tuesday the lights flicker twice, the NAS dies mid-write, and the file that "saved" last night turns out to be the backup that never finished. A ten-person office does not need a generator. It needs a UPS on every critical device, an auto-shutdown script that has actually fired once, and a laminated cold-start card that says which box powers on first.

1. The UPS audit: every critical device, a number, a date

  • Assign every device a tier. Tier 1 is "dirty shutdown costs us data or hours": NAS, file server, router and modem, the desktop running payroll or point-of-sale. Tier 2 is "annoying but recoverable": workstations, printer. Every Tier 1 device gets a UPS sized to run it long enough to shut down cleanly — ten to twenty minutes for most small-office NAS boxes, not two hours.
  • Write the runtime number down, don't guess it. The printed "30 minutes at half load" is a sales number. Measure your load once and write it on the unit in marker: "runtime 14 min at actual load, tested 2026-03." An untested runtime estimate is a rumor with a battery attached.
  • Batteries expire on a clock that isn't yours. Small-office UPS batteries die at three to four years regardless of use — and they die silently, showing a green light while holding ten minutes of charge. Install date on the unit, battery-replacement reminder in the calendar, quarterly self-test, log the result.
  • The router and modem get their own small UPS. The cheapest high-leverage move on this page: when everything else is dark, office internet staying up for an hour means you can take the incident call and check status pages from a laptop on battery. One $40 unit under the desk.

2. Auto-shutdown: configured, cabled, and fired once

  • A UPS without a USB cable is a battery, not a system. The grace period only converts to safety if the NAS knows the power died. Vendor agent or nut/apcupsd, policy: "on battery 60 seconds or below 40% charge → graceful shutdown." Sixty seconds filters out the two-second blips.
  • Decide what shuts down and what stays up — in writing, in the config. Router and modem on their little UPS stay up so you can see what's happening; NAS and file server shut down gracefully; workstations are human decisions.
  • Test the script like you test backups: by firing it. During a scheduled window, unplug the UPS input cord for thirty seconds. Verify the agent saw the event, the shutdown began, the NAS came back clean. A shutdown script that has never run is a hypothesis.
  • Wire the on-battery alert to your normal alert channel. A power blip is informational; a power blip plus "shutdown initiated" is actionable.

3. The cold-start order: one page, taped to the wall

  • Power-on order is a dependency graph, not a scrum. Modem and router first (wait for sync lights), switch second, NAS and servers third (let boot checks finish before the layer that mounts shares), workstations last. Wrong order doesn't usually break things permanently — it breaks them confusingly, at 8:05 a.m. with four people standing behind you.
  • The card names "wait" signals, not just steps. "Router: all status lights solid, ~2 minutes." "NAS: web UI answers — then start the app server." A step without a done-signal is a step people skip.
  • Include the "do NOT power up" line. After a storm: the UPS that logged a battery fault, the machine that smells warm, the outlet the electrician hasn't seen. One line prevents the heroic second failure.
  • Laminate it, photograph it, put it in the shared drive. The card in the drawer behind the NAS is a card that isn't there when the drawer is the thing that's dark.

4. After dirty power: the data-integrity sweep

  • Read the UPS event log before you trust the room. One sag, or six sags over four hours, or a full outage your "clean shutdown" politely declined to perform because the agent crashed in June — the log line goes into the incident notes verbatim.
  • Check the disks, not just the services. Dirty power shows up as SMART errors, RAID resync flags, and filesystem messages days later. Short SMART self-test on every Tier 1 disk the same day; eyeball the RAID status page. A volume that mounts is not a volume that's healthy.
  • Distrust any backup that was mid-flight when the lights went. A backup killed at 60% is worse than none if you restore from it unverified. Find the jobs whose window overlaps the outage, verify each, re-run the dead ones.
  • Watch for the brownout's delayed invoice. Undervoltage damages power supplies in ways that surface as "random" reboots a week later. Put the power event in the ticket history so the pattern is visible when someone connects week-two weirdness to week-one flicker.

5. The quarterly test and the evidence it produces

  • One thirty-second unplug drill, quarterly, in a window. Fifteen minutes per quarter is the entire cost of never finding out during a real blackout that the agent crashed in June. Stack the checks in the same window: UPS self-test, battery date, runtime re-measure, cable seated — one window, five checks, one log line.
  • The log line is the deliverable. "Q3 power drill: 30s unplug, agent fired at 8s, shutdown began, clean reboot, battery OK, runtime 14 min." Four of those lines is what "we take continuity seriously" looks like as evidence on the cyber-insurance questionnaire, the customer security questionnaire, and the annual review — none of which accept "we have UPSes" as an answer.
  • Escalate the findings, not the anxiety. The drill will find a battery two years past date and a server someone moved off the UPS "temporarily" in March. Each finding gets an owner and a date — a finding without an owner is a finding you'll re-read next quarter, unchanged.

Small-team honesty note: you do not need a generator, a transfer switch, or a fuel contract. What a ten-person office needs is four Tier 1 devices on UPS, a shutdown script that has fired once on purpose, a cold-start card on the wall, and a quarterly unplug drill logged in one line. The trap is green-light complacency: three years of green lights on a battery that died in year two, and an auto-shutdown script "configured" since 2024 that has never once actually run. Green lights are not evidence. The unplug drill is.

Related pages (all free, no signup):

From the HIVE80lab kit:

💬 Discussion: What are your first 3 steps in an incident?

Top comments (0)