A small office survives a Texas grid emergency by shutting down cleanly before the power drops and restarting in dependency order after it returns. The UPS only buys the minutes needed for a clean shutdown. Everything else, from when to act to what comes back first, has to be decided in advance and written down.
I'm the CTO at a managed IT company in Dallas, and this is the mechanical part of storm planning that's easiest to skip. Not the policy. The wiring.
The problem
A UPS is not a runtime plan. It's a shutdown plan.
The outages that actually hit DFW offices aren't short. After the windstorm on May 28, 2024, Oncor counted more than 650,000 customer outages, and the hardest-hit areas waited days. During Winter Storm Uri in February 2021, ERCOT's firm load shed ran nearly 3 consecutive days. No battery in a server closet covers that.
So the goal during an outage is simple. Get every server down cleanly while the battery still has charge, then bring things back in the right order once power is stable.
Step 1: make the UPS actually trigger the shutdown
A UPS that beeps during its self-test proves the battery works. It doesn't prove the server will shut down. Those are separate paths, and the second one is where plans fail.
The UPS needs a way to tell the server that the battery is low. There are 3 common paths:
A USB cable from the UPS to a single server
A network management card in the UPS that signals hosts over the LAN
A shutdown agent on the hypervisor, which can take down the guests before the host
Pick the one that fits your setup, set servers to shut down on low battery, and then test it the boring way. Pull the plug on a quiet afternoon and watch. Until you've seen it happen, assume it doesn't.
Step 2: tie shutdown actions to alerts, not to the outage
Waiting for the lights to flicker is too late. ERCOT publishes a notification ladder (called TXANS), and the useful trick is to attach an IT action to each rung.
ERCOT alert IT action
Weather Watch (about 3 to 5 days out) Confirm last night's backup finished, run a UPS self-test, check that remote logins work
Conservation Appeal Turn off nonessential gear, move scheduled reboots out of the appeal window, confirm low-battery shutdown is set
Energy Emergency Alert Level 1 or 2 Name who shuts down which system, then stand by
Energy Emergency Alert Level 3 Shut down noncritical servers cleanly before the power goes, keep the firewall and core switch on UPS, post a status note for staff
EEA Level 3 is the only rung where ERCOT says controlled outages are possible, and ERCOT is clear they'd hit every customer class, commercial included.
Keeping the firewall and core switch on UPS is deliberate. They're what lets the status note, and anyone working remotely, still reach the network.
Step 3: restart in dependency order
When power comes back, don't power everything on at once. Ignore the first flicker, give the power a little while to hold, and then bring systems up in the order they depend on each other.
Internet service and the firewall. Everything else needs somewhere to go.
Core switches and Wi-Fi.
Storage, NAS or SAN. Let it finish its own checks before anything leans on it.
Domain controllers, or whatever handles sign-in.
Application servers and line-of-business software.
Backups. Confirm the next scheduled job actually runs and finishes.
The logic is plain. Storage comes before servers because the servers lean on it. Sign-in comes before applications because the apps need something to authenticate against.
Step 4: post-boot checks
Anything that rebooted on its own deserves a look for 3 things:
The right IP address
The right clock time
The services it's supposed to be running
Then check the UPS units. They're probably drained. A second outage an hour later will find them empty and take everything down hard, which is the exact scenario you spent all that effort avoiding.
Top comments (0)