DEV Community: Rahul Dhileep Kumar

Production outage lessons: we blamed our code. It was SSL.

Rahul Dhileep Kumar — Fri, 16 Oct 2026 04:30:00 +0000

Building a product is one job.
Keeping it alive is a completely different job.

Nobody warns you about the second one.
Or they do, and you don't listen.
I didn't listen.

This post is my production outage lessons, learnt the expensive way.
One backend split, one expired SSL certificate, four hours of blaming the wrong thing.
And what I'd do differently: checklists, rollbacks, monitoring and postmortems.

The clever idea: splitting the backend

This happened in 2026, not long after we incorporated TrackMy Tech.
We'd just become a private limited company.
And we had a nagging feeling we weren't doing things professionally enough.
We wanted to fix that.

We also had real problems.
Lots of payment-related issues.
A steady stream of API errors.
And the load wasn't balancing well.

Then an Instagram reel showed up.
I shared it with the team.
"This model is so capable. We can try some serious stuff with it."
That one reel kicked off the whole cycle.

The plan: split the backend into separate pieces.
We used an AI coding agent to do the heavy lifting.
To be fair to the agent, it did its job.
It was us who got things wrong.

On paper, splitting load sounds smart.
Spread the work. Feel like a real engineering team.
Put "distributed" in your head and feel tall.

Then everything went down

Not slowly. Not politely.
Everything went down.
For hours.

If you've never broken production, let me describe the feeling.
Your stomach drops to somewhere near your shoes.
Every message notification sounds like an accusation.

Real academies use Track My Academy.
Real coaches. Real parents. Real attendance.

None of them care about our clever architecture.
They care that the app works.

And it did not work.

Photo: Taylor Vick on Unsplash

The real culprit: an expired SSL certificate

We were sure we'd broken the code.
So we did what any panicking team does.
We checked the split.
We checked the code.
Then we checked it all again.

About four hours later, we found it.
It wasn't the code at all.
The SSL certificate on our backend domain had expired.
Right in the middle of the transition.

It was failing silently.
And we were busy blaming our own clever architecture.

When we finally saw it, we just stared at the screen.
Oh God.
This is why debugging is hard.

Here's why it's so sneaky.
A TLS certificate proves a server is who it claims to be.
Every certificate has a "Not After" date.
After that date, it's simply no longer valid.
Apps and browsers stop trusting the server, and requests fail.

Short certificate lifetimes are normal now.
Let's Encrypt says its default certificates are valid for 90 days.
It recommends renewing them every 60 days.
Miss a renewal and the clock just quietly runs out.

The timing is what fooled us.
Two things changed at once: our backend, and the certificate's validity.
So naturally we blamed the change we'd made ourselves.

Google's SRE book says roughly 70% of outages come from changes to a live system.
We were the change.
We just weren't the cause.

The longest night in the office

I stayed in the office all night.

Not because it looks dedicated on LinkedIn.
Because it was my mess, and I wanted it fixed.

There's a special kind of quiet in an office at 3 a.m.
It's just you, the monitors, and your mistakes.

The Monster cans on the wall were not decoration that night.
They were fuel.
(The full tour of that desk is in three monitors, one Monster can.)

Office nights are not rare for me, sadly.
That's a whole other post: exams, the office and a 1 a.m. gate.

When I'm stuck badly enough, I call Arun

Every founder needs one person who picks up at any hour.
For me, that's Arun. Our CTO.

No questions asked. He just picks up.
At any hour. Which is frankly suspicious.

If you want to know who he is, here's his portfolio.
He's the reason "stuck" never becomes "stuck forever".

The lesson here isn't technical.
Know who you'll call before the crisis, not during it.

A deployment checklist I'd hand my past self

I'm not going to pretend I have a grand theory.
But here's the checklist that night burned into my brain.

Check the boring stuff first. SSL certificates, DNS, environment variables. Then blame the code.
Know when every certificate expires. Write the dates down before you touch anything.
Change one big thing at a time. Two changes, one failure, endless guessing.
Deploy when people are awake. Ideally the people who can fix it.
Know your way back. Before you ship, not after.
Watch it after you ship. "Deployed" is not the same as "working".

Google's SRE book lists three practices for safer changes.
Progressive rollouts.
Quickly and accurately detecting problems.
Rolling back safely when problems arise.

That's basically my checklist, written by people with far more sleep.

Photo: Jakub Żerdzicki on Unsplash

Rollbacks: have a way back

If you can't undo a change quickly, you're not deploying.
You're gambling.

One classic pattern is blue-green deployment, as Martin Fowler describes it.
You keep two nearly identical production environments.
One is live. The other gets the new version.
You test the idle one, then switch the router over.

If it breaks, you switch the router back.
Fowler calls it "a rapid way to rollback".

You don't need a fancy setup to steal the idea.
Keep the old config. Keep the old build.
Know the exact command that puts things back.
Practise it once on a calm day.

Monitoring and alerting: find out before your users do

The worst way to discover an outage is from a user.
The second worst is from a friend asking "is your app down?"

The Google SRE chapter on monitoring distributed systems is a good start.
It names four golden signals: latency, traffic, errors and saturation.
Measure those and page a human when they look wrong.
I'd add one more for small teams: certificate expiry dates.
That gets you decently covered, according to the book.

It also says something I love: "Every page should be actionable."
An alert you ignore is just a very expensive ringtone.

Later, I built a "Pager System" for Track My Academy developers.
It's an internal real-time error and notification system.
The idea is simple: problems should find developers, not wait to be found.

Photo: Luke Chesser on Unsplash

Write the postmortem, and keep it blameless

After the fix, the temptation is to sleep and never speak of it again.
Don't.

Write an incident postmortem.
Google's chapter on postmortem culture puts it nicely.
"Writing a postmortem is not punishment."
It's a learning opportunity for the whole company.

Their triggers include user-visible downtime and an engineer having to roll back.
A multi-hour outage ticks both boxes with enthusiasm.

Atlassian's guide to incident postmortems is very practical.
It covers a timeline, root cause, impact, metrics and action items.
It suggests a review meeting within 24 to 48 hours of resolution.
Then draft the write-up straight after.
Wait longer and the details evaporate.

Blameless doesn't mean nobody made a mistake.
It means you fix the system so the next person can't make it.
Even if the next person is, once again, me.

Was it worth it?

Honestly? Yes.
Not the outage. The outage was terrible.
But I learned more in that one night than in a month of tutorials.

Mistakes are expensive teachers.
They're also very, very memorable.

I'd still prefer the tutorials, though.

FAQ

What are the most important production outage lessons for a small team?

Change one thing at a time.
Always have a tested way back.
Monitor the basics so you hear about problems before users do.

Can an expired SSL certificate take down a backend?

Yes, and quietly.
Once a certificate passes its expiry date, clients stop trusting the server.
Requests fail, and it can look exactly like a code bug.

What should go in an incident postmortem?

A timeline, the impact, the root cause and clear action items.
Keep it blameless.
Fix the system, not the person.

Sources

What next

Curious how the product got built in the first place? Read the app that arrived from 2006.
Or how I learned the tools: dozens of email accounts and zero shame.

Running an academy? Track My Academy handles attendance, fees, player analytics and parent communication.
It's built by TrackMy Tech.

Want to talk shipping, breaking and fixing things? Book a call.

How to win a hackathon, according to a tired judge

Rahul Dhileep Kumar — Mon, 12 Oct 2026 04:30:00 +0000

Cover photo: Evan Marvell on Unsplash

On 8 and 9 October, I sat on the other side of the table.

I was on the jury of a hackathon at VISTAS.
It was run by the Department of Computing Sciences and the Vels Innovation Council.
More than 80 teams turned up.
Over two days, I evaluated somewhere between 20 and 30 of them.

I've been a hackathon participant.
I've lost hackathons.
I've won a couple too, which my about page mentions with suspicious frequency.

Now I've judged one.
The personal story of that loop is in from hackathon loser to hackathon jury.

This post is the practical bit.
How to win a hackathon, from someone who just spent two days scoring them.

What hackathon judges actually see

Here's the part nobody tells participants.

Your judge isn't seeing your project in isolation.
They're seeing it as team number seven.
Or team number fourteen.

Most teams in a judge's group share the same or similar problem statements.
So the judge has already heard your problem explained.
Several times.
Usually in the same words.

Around the fifth team, focus starts slipping.
Not because judges don't care.
Because judges are human.

There's research on this, sort of.
Decision fatigue describes decisions getting worse after a long run of them.
The evidence for it is honestly mixed, so don't build a religion on it.
But anyone who has judged fifteen pitches in a row will nod politely.

Running order matters too.
A Carnegie Mellon study found later performers got higher marks.
That was Eurovision and figure skating, not hackathons.
You can't choose your slot anyway.
So the only thing you control is how fast you earn the judge's attention.

Photo: Herlambang Tinasih Gusti on Unsplash

The first 30 seconds of your hackathon pitch

Start like a person, not a slide deck.

Give the judge a firm handshake.
Ask them how their day is going.
They'll appreciate it.
It's probably been a long one.

Then introduce yourself and your team.
Clearly.
Confidently.
Names, and who built what.

Then say three things, fast:

The problem statement you picked, in one line.
The scope you chose to tackle within it.
Your solution, in one sentence a tired person can repeat.

That's it.
You now have the judge's attention, and you've spent maybe thirty seconds.

An elevator pitch usually runs thirty seconds to two minutes.
Your opening should sit at the short end.

Show the prototype early

The demo is the hackathon.
Everything else is decoration.

Plenty of hackathon rulebooks say this out loud.
The UH CodeRED rules, for example, strongly encourage a demo.
They say pitches and presentations are discouraged.
They even say teams aren't judged on the quality of their pitch.

Not every hackathon works that way.
But almost every judge prefers seeing a thing work to hearing about it.

So get to the prototype quickly.
Show the main flow, end to end.
One happy path that works beats five half-loaded features.

If something breaks, say so and move on.
Judges have seen broken demos before.
I've broken production, so I'm in no position to judge that part.

Photo: Shibraj Deb on Pexels

How to win a hackathon: stay on the problem statement

This one hurt to watch, because I used to do it.

Teams drift.
They start on the given problem statement.
Then they bolt on a chatbot, a dashboard and a dream.
By the end, the judge can't find the original problem.

Your score is usually tied to the problem you were given.
Solving a different, cooler problem rarely earns bonus points.
It mostly earns confused faces.

Before you build anything, write the problem statement on a sticky note.
Every feature must point back to it.
If it doesn't, it waits for version two.
I run my own life off sticky notes, as my desk setup post proves.

Don't over-engineer your hackathon project

The second classic mistake: building a cathedral in two days.

Microservices.
Three databases.
A custom login system nobody asked for.
And a demo that loads forever.

Judges don't score your architecture diagram.
They score what they can see working.

Pick the boring stack you already know.
Build the smallest thing that proves your idea.
Spend the saved hours polishing the one flow you'll demo.

When I started building Track My Academy, the first night ended with a landing page.
Not an architecture.
A landing page I actually liked.
Small, finished things build momentum.

What the jury doesn't want to hear

Some things quietly drain your score.

Facts the judge already knows. They've read the problem statement. Probably several times today.
Statistics read off a slide. Especially ones an AI tool produced five minutes earlier.
A tour of everything you made with AI. Use AI tools, absolutely. Just don't make them the pitch.
Trying to impress instead of explain. Big words, no working product. Judges spot it instantly.

On AI: I'm a fan.
I once chased free AI trials across dozens of email accounts.
But the judge is scoring your thinking, not your prompt history.

Photo: Phil Hearing on Unsplash

A simple hackathon pitch structure

If you want a template, steal this one.

Hello (10 seconds). Handshake, how's your day, team intro.
Problem and scope (20 seconds). One line each. No essays.
Solution (15 seconds). One sentence, plain words.
Demo (most of your time). One flow, start to finish, working.
How you built it (30 seconds). Your stack, and one hard thing you solved.
What's next (10 seconds). One honest line. Then stop talking.

Then let the judge ask questions.
Answer them directly.
"I don't know yet" is a perfectly good answer.
Making something up is not.

Hackathon tips for the questions round

Questions are where good teams pull ahead.

Listen to the whole question.
Answer the question that was asked, not the one you rehearsed.
Let the person who built that part answer it.
Keep answers to a few sentences.

If a judge points out a flaw, agree if it's true.
Then say how you'd fix it.
That shows judgement, which is exactly what they're measuring.

FAQ

How do hackathon judges score projects?

It depends on the event, so read the rules first.
Common criteria are technical difficulty, design, completion and impact.
Most judges weigh what they see working more than what they hear.

How long should a hackathon pitch be?

As short as the rules allow.
Aim for under a minute before the demo starts.
Then let the working prototype do the talking.

Do you need a perfect product to win a hackathon?

No.
You need one flow that clearly works and clearly fits the problem statement.
Polish that, and be honest about the rest.

Go build something

If you're heading into a hackathon, good luck.
Keep it short, keep it focused, and show the thing.

I build products at TrackMy Tech, and I'm always happy to talk shop.
Book a call if you want a second opinion on your idea.
Or read about the app that arrived from 2006, my favourite cautionary tale.