DEV Community

Cover image for Why "Just Restart It" Stopped Working
Sreekanth Kuruba
Sreekanth Kuruba

Posted on

Why "Just Restart It" Stopped Working

#devops #kubernetes #sre

Why "Just Restart It" Stopped Working

A eulogy for the universal debugging technique

The Universal Truth

Every engineer has said it.

Every engineer has heard it.

Three words that have debugged more systems than all monitoring tools combined:

"Have you tried restarting it?"

It worked for decades. So well we turned it into a meme. A joke. A badge of honor.

"Did you turn it off and on again?"

We laughed because it was true.

When Restarting Made Sense

Once upon a time, a server was a physical thing.

One machine. One process. One problem.

When something broke:

Service stops responding
→ SSH into the box
→ ps aux | grep myapp
→ PID still there? Process hung?
→ kill -9 PID
→ ./start-myapp.sh
→ Everything works again

Total time: 2 minutes
Total stress: Minimal
Total sleep lost: None
Enter fullscreen mode Exit fullscreen mode

Why did this work?

Because the problem was usually temporary.

A memory leak. A deadlock. A bad connection that timed out wrong.

The code had a bug, sure. But restarting reset the state to before the bug happened.

It wasn't elegant. It wasn't permanent.

But at 3 AM, that's all anyone cared about.

The First Sign of Trouble

Then we got more servers.

One box became ten.

Ten became a hundred.

Restarting stopped being a single command.

It became a deployment.

for server in $(cat servers.txt); do
    ssh $server "systemctl restart myapp"
done
Enter fullscreen mode Exit fullscreen mode

This worked. Mostly.

Until the day it didn't.

The Cascade

I watched this happen once.

02:15 - Pager: "Database connections failing"
Enter fullscreen mode Exit fullscreen mode

The on-call engineer checks the logs.

Database is overwhelmed. Too many connections.

The solution, burned into muscle memory from years of single-server debugging:

"Restart the database."

One command. One mistake.

systemctl restart postgresql
Enter fullscreen mode Exit fullscreen mode

The database came back in 45 seconds.

In those 45 seconds:

  • All 200 application servers lost their connection pools
  • All 200 retried simultaneously, using identical retry logic
  • All 200 failed their health checks
  • The load balancer marked them all unhealthy
  • The site went down

The database was fine.

The app servers were fine.

The connections were gone.

The restart fixed nothing and broke everything.

One restart.

47 minutes of downtime.

Why Restarting Broke

Restarting worked when:

  • State lived in one place
  • Dependencies were simple
  • Recovery was faster than finding root cause

Restarting broke when:

  • State moved to databases, caches, message queues
  • Services started calling other services
  • "Just restart it" became "restart everything in the right order with the right delays and pray"

A restart is no longer a local action.

It's a distributed event.

You don't restart one thing.

You restart a graph of dependencies.

What Happens When You Restart Now 

You restart Service A
↓
Service A disconnects from database
↓
Database releases locks
↓
Service B loses connection to Service A
↓
Service B retries aggressively
↓
Retries overwhelm Service C
↓
Service C crashes
↓
Everything is on fire
Enter fullscreen mode Exit fullscreen mode

All because you restarted "just one thing."

The Lie We Tell Ourselves

"Restarting is harmless."

It isn't.

Every restart is:

  • A forced state reset
  • A connection teardown
  • A potential cascade trigger
  • A temporary partial outage (even if small)

We accepted restarts as "free" because the cost was invisible.

Until it wasn't.

What Replaced Restarting

The industry didn't ban restarts.

It made them unnecessary.

Health checks

Detect problems before users do.

# Kubernetes liveness probe example
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
Enter fullscreen mode Exit fullscreen mode 
If service unhealthy, don't send traffic
Let it recover or replace it
Users never see the failure
Enter fullscreen mode Exit fullscreen mode

Graceful degradation

Fail partially, not completely.

Cache down? Serve stale data
Database slow? Queue writes, serve reads
Something broke? Everything else keeps running
Enter fullscreen mode Exit fullscreen mode

Automatic replacement

Never restart. Always replace.

Pod dies? New one starts
Node fails? Pods move
Same binary. Clean state. No cascade
Enter fullscreen mode Exit fullscreen mode

Rolling restarts

One at a time, with verification.

Restart server 1 of 10
Wait for health check
Restart server 2 of 10
Never lose capacity
Enter fullscreen mode Exit fullscreen mode

The Systems That Don't Need Restarts

Netflix doesn't restart. It terminates and replaces.

Google doesn't restart. It shifts load and repairs.

Your bank doesn't restart. It fails over to another region.

These aren't magic.

They're design choices.

They assumed from day one that "restart" was not a strategy.

The Honest Confession

I still say "have you tried restarting it?"

Sometimes it's the fastest path to it works now.

But I don't pretend it's a fix anymore.

It's a diagnostic.

A temporary patch.

A way to buy time until the real problem reveals itself.

The difference is:

I know the difference now.

What You Can Do Monday

For your most critical service:

  • Find the last time it was restarted
  • Ask: "Why did that restart happen?"
  • Ask: "Could we have avoided it?"

If yes, build the automation.

If no, document why (so next time you know).

For your next outage:

  • Resist the restart reflex
  • Check dependencies first
  • Check connections second
  • Check logs third
  • Restart only when you understand what you're about to break

The Question

When was the last time you restarted something

and didn't know exactly what would happen when it came back?

Be honest.

This is part of a series on operations in the age of distributed systems. Next up: "The Pager Should Not Exist."

Top comments (0)