Production outage lessons: we blamed our code. It was SSL.

Building a product is one job.
Keeping it alive is a completely different job.

Nobody warns you about the second one.
Or they do, and you don't listen.
I didn't listen.

This post is my production outage lessons, learnt the expensive way.
One backend split, one expired SSL certificate, four hours of blaming the wrong thing.
And what I'd do differently: checklists, rollbacks, monitoring and postmortems.

The clever idea: splitting the backend

This happened in 2026, not long after we incorporated TrackMy Tech.
We'd just become a private limited company.
And we had a nagging feeling we weren't doing things professionally enough.
We wanted to fix that.

We also had real problems.
Lots of payment-related issues.
A steady stream of API errors.
And the load wasn't balancing well.

Then an Instagram reel showed up.
I shared it with the team.
"This model is so capable. We can try some serious stuff with it."
That one reel kicked off the whole cycle.

The plan: split the backend into separate pieces.
We used an AI coding agent to do the heavy lifting.
To be fair to the agent, it did its job.
It was us who got things wrong.

On paper, splitting load sounds smart.
Spread the work. Feel like a real engineering team.
Put "distributed" in your head and feel tall.

Then everything went down

Not slowly. Not politely.
Everything went down.
For hours.

If you've never broken production, let me describe the feeling.
Your stomach drops to somewhere near your shoes.
Every message notification sounds like an accusation.

Real academies use Track My Academy.
Real coaches. Real parents. Real attendance.

None of them care about our clever architecture.
They care that the app works.

And it did not work.

Photo: Taylor Vick on Unsplash

The real culprit: an expired SSL certificate

We were sure we'd broken the code.
So we did what any panicking team does.
We checked the split.
We checked the code.
Then we checked it all again.

About four hours later, we found it.
It wasn't the code at all.
The SSL certificate on our backend domain had expired.
Right in the middle of the transition.

It was failing silently.
And we were busy blaming our own clever architecture.

When we finally saw it, we just stared at the screen.
Oh God.
This is why debugging is hard.

Here's why it's so sneaky.
A TLS certificate proves a server is who it claims to be.
Every certificate has a "Not After" date.
After that date, it's simply no longer valid.
Apps and browsers stop trusting the server, and requests fail.

Short certificate lifetimes are normal now.
Let's Encrypt says its default certificates are valid for 90 days.
It recommends renewing them every 60 days.
Miss a renewal and the clock just quietly runs out.

The timing is what fooled us.
Two things changed at once: our backend, and the certificate's validity.
So naturally we blamed the change we'd made ourselves.

Google's SRE book says roughly 70% of outages come from changes to a live system.
We were the change.
We just weren't the cause.

The longest night in the office

I stayed in the office all night.

Not because it looks dedicated on LinkedIn.
Because it was my mess, and I wanted it fixed.

There's a special kind of quiet in an office at 3 a.m.
It's just you, the monitors, and your mistakes.

The Monster cans on the wall were not decoration that night.
They were fuel.
(The full tour of that desk is in three monitors, one Monster can.)

Office nights are not rare for me, sadly.
That's a whole other post: exams, the office and a 1 a.m. gate.

When I'm stuck badly enough, I call Arun

Every founder needs one person who picks up at any hour.
For me, that's Arun. Our CTO.

No questions asked. He just picks up.
At any hour. Which is frankly suspicious.

If you want to know who he is, here's his portfolio.
He's the reason "stuck" never becomes "stuck forever".

The lesson here isn't technical.
Know who you'll call before the crisis, not during it.

A deployment checklist I'd hand my past self

I'm not going to pretend I have a grand theory.
But here's the checklist that night burned into my brain.

1. Check the boring stuff first. SSL certificates, DNS, environment variables. Then blame the code.
2. Know when every certificate expires. Write the dates down before you touch anything.
3. Change one big thing at a time. Two changes, one failure, endless guessing.
4. Deploy when people are awake. Ideally the people who can fix it.
5. Know your way back. Before you ship, not after.
6. Watch it after you ship. "Deployed" is not the same as "working".

Google's SRE book lists three practices for safer changes.
Progressive rollouts.
Quickly and accurately detecting problems.
Rolling back safely when problems arise.

That's basically my checklist, written by people with far more sleep.

Photo: Jakub Żerdzicki on Unsplash

Rollbacks: have a way back

If you can't undo a change quickly, you're not deploying.
You're gambling.

One classic pattern is blue-green deployment, as Martin Fowler describes it.
You keep two nearly identical production environments.
One is live. The other gets the new version.
You test the idle one, then switch the router over.

If it breaks, you switch the router back.
Fowler calls it "a rapid way to rollback".

You don't need a fancy setup to steal the idea.
Keep the old config. Keep the old build.
Know the exact command that puts things back.
Practise it once on a calm day.

Monitoring and alerting: find out before your users do

The worst way to discover an outage is from a user.
The second worst is from a friend asking "is your app down?"

The Google SRE chapter on monitoring distributed systems is a good start.
It names four golden signals: latency, traffic, errors and saturation.
Measure those and page a human when they look wrong.
I'd add one more for small teams: certificate expiry dates.
That gets you decently covered, according to the book.

It also says something I love: "Every page should be actionable."
An alert you ignore is just a very expensive ringtone.

Later, I built a "Pager System" for Track My Academy developers.
It's an internal real-time error and notification system.
The idea is simple: problems should find developers, not wait to be found.

Photo: Luke Chesser on Unsplash

Write the postmortem, and keep it blameless

After the fix, the temptation is to sleep and never speak of it again.
Don't.

Write an incident postmortem.
Google's chapter on postmortem culture puts it nicely.
"Writing a postmortem is not punishment."
It's a learning opportunity for the whole company.

Their triggers include user-visible downtime and an engineer having to roll back.
A multi-hour outage ticks both boxes with enthusiasm.

Atlassian's guide to incident postmortems is very practical.
It covers a timeline, root cause, impact, metrics and action items.
It suggests a review meeting within 24 to 48 hours of resolution.
Then draft the write-up straight after.
Wait longer and the details evaporate.

Blameless doesn't mean nobody made a mistake.
It means you fix the system so the next person can't make it.
Even if the next person is, once again, me.

Was it worth it?

Honestly? Yes.
Not the outage. The outage was terrible.
But I learned more in that one night than in a month of tutorials.

Mistakes are expensive teachers.
They're also very, very memorable.

I'd still prefer the tutorials, though.

FAQ

What are the most important production outage lessons for a small team?

Change one thing at a time.
Always have a tested way back.
Monitor the basics so you hear about problems before users do.

Can an expired SSL certificate take down a backend?

Yes, and quietly.
Once a certificate passes its expiry date, clients stop trusting the server.
Requests fail, and it can look exactly like a code bug.

What should go in an incident postmortem?

A timeline, the impact, the root cause and clear action items.
Keep it blameless.
Fix the system, not the person.

Sources

• Google SRE book: Introduction
• Google SRE book: Monitoring distributed systems
• Google SRE book: Postmortem culture
• Atlassian: Incident postmortems
• Martin Fowler: Blue-green deployment
• Wikipedia: Public key certificate
• Let's Encrypt FAQ

What next

Curious how the product got built in the first place? Read the app that arrived from 2006.
Or how I learned the tools: dozens of email accounts and zero shame.

Running an academy? Track My Academy handles attendance, fees, player analytics and parent communication.
It's built by TrackMy Tech.

Want to talk shipping, breaking and fixing things? Book a call.