DEV Community

Cover image for Microsoft's Fix Didn't Work. Yours Won't Either. Here's How to Survive It.
KAMAL KISHOR
KAMAL KISHOR

Posted on

Microsoft's Fix Didn't Work. Yours Won't Either. Here's How to Survive It.

Outlook went down. Teams went down. Millions of people couldn't email, couldn't meet, couldn't "quickly hop on a call."

Microsoft shipped a fix.

Then they posted the update every on-call engineer has typed with shaking hands at 2 AM:

"It's not going as expected."

Let that sink in. One of the biggest engineering orgs on the planet. Elite SREs. Runbooks for their runbooks. And the first fix still didn't work.

If you've ever pushed a hotfix that made things worse, welcome. You're in very expensive company. 🍿


🔥 Hot take #1: Your "fix" is just another deploy

We treat hotfixes like magic. They're not. A hotfix is a deploy written under pressure, reviewed by tired people, and shipped into a system that's already broken.

That's the worst possible time to ship code. And it's exactly when we skip the safeguards.

What to do instead:

  • Hotfixes go through the same pipeline as normal deploys. Just faster, not skipped.
  • Roll them out to 1% first. Yes, even during an outage. Especially during an outage.
  • Before you ship, ask: "If this is wrong, how do I undo it in under 5 minutes?"

🔥 Hot take #2: Rollback is not a strategy if you've never tested it

Everyone says "we'll just roll back." Cool. When did you last actually do it?

Rollbacks fail for boring reasons:

  • A database migration that only goes one way
  • A cache full of data in the new format
  • Config that changed somewhere nobody remembered

Practice rollbacks like fire drills. If it's not rehearsed, it's not a plan. It's a prayer. 🙏


🔥 Hot take #3: Your company has a single point of failure. It's called "everything"

When email and chat go down together, most companies don't slow down. They stop.

Ask yourself honestly:

  • If Slack/Teams dies, where does your incident channel live?
  • If your SSO provider is down, can anyone log in to fix anything?
  • Is your status page hosted on the same infra that just went down? 💀

Have a boring backup: a phone tree, a WhatsApp group, a status page on a different provider. Boring saves lives.


🔥 Hot take #4: "It's not going as expected" is the most senior sentence in engineering

Juniors hide failed fixes. Seniors announce them.

Microsoft said publicly that their fix wasn't working. That's not weakness, that's how incident response is supposed to work:

  1. Say what you know.
  2. Say what you don't know.
  3. Say when you'll update next. And then actually update.

Silence during an outage is worse than bad news. Users can handle "we're still on it." They can't handle nothing.


🧠 The 60-second incident checklist

Save this. Print it. Tape it to your monitor.

  • [ ] Can I stop the bleeding before finding the root cause? (feature flag, traffic shift, rollback)
  • [ ] Is my fix going to a small slice of traffic first?
  • [ ] Do I have a tested way to undo this fix?
  • [ ] Have I posted a status update in the last 30 minutes?
  • [ ] Is one person clearly in charge? (Not five people all typing in the same channel.)
  • [ ] Am I writing down what I did and when, for the postmortem?

💬 The real lesson

Every system fails. Every team ships a bad fix eventually. The giants just do it in front of a bigger audience.

The difference between a bad day and a disaster isn't whether your first fix works.

It's whether you built a system that lets you try a second one safely.


Your turn: what's the worst hotfix you've ever shipped? 👇
Confessions in the comments. No judgment. We've all git push --force'd something we regret. 😅

Top comments (0)