How you doin'?
Let's talk about the Iconic thing we heard a lot: "server down, try later." Your daddy said it while trying to book a Tatkal ticket. Your cousin said it the day JEE results dropped and the portal turned into a spinning wheel of despair. It's become our national way of shrugging at technology — like the internet is weather, and servers just... go down sometimes, nobody's fault, act of god, try later na.
Except it's not weather. Every single time a site goes down, there is a specific, findable reason, sitting in a log or a trace somewhere, and almost nobody ever looks at it because looking at it is annoying and "try later" is right there, free, zero effort. So for a hackathon, I decided to stop saying "server down hai" and start actually finding out what "down" means. I built a fake exam-results website, gave myself the power to break it on command, and then made myself watch — using an observability tool called SigNoz — exactly what "down" looks like from the inside, every single time.
Turns out "server down" is not one thing. It's at least four different things wearing the same trench coat.
Suspect #1: The database that forgot how to hurry
This is the boring one and also the most common one. Somewhere behind your "check result" button, there's a database being asked a question, and sometimes that question takes way longer to answer than it should — too many people asking at once, a badly written query, whatever. The site isn't "down." It's just... waiting. Politely. Forever.
I simulated this by literally telling my backend to nap for 3 seconds before touching the database:
with tracer.start_as_current_span("db.query") as db_span:
if state.db_slowdown:
db_span.set_attribute("chaos.triggered", "db_slowdown")
time.sleep(state.db_slowdown_seconds)
Then I opened SigNoz's trace explorer, sorted by duration, and there it was — a fat, unmissable span sitting right at the top labeled db.query, 3 seconds wide, with an attribute literally saying chaos.triggered: db_slowdown. "Server down hai" is, in this case, a lie. The server is very much up. It's just stuck talking to a database that's having a bad day.
Suspect #2: The cache that quietly clocked out
Most fast websites cheat. They keep a copy of common answers in something like Redis so they don't have to ask the real database every single time. When that cache goes down, nobody tells you — the site doesn't crash, it just gets slow and grumpy, because now every single request has to go the long way round.
I killed my Redis connection on purpose and watched the cache.lookup span in SigNoz change from occasionally saying cache.hit: true to saying cache.hit: false on literally every request, back to back to back. That pattern — a flat line of misses where there used to be a healthy mix — is the fingerprint of a dead cache. Nobody's server "went down." Its shortcut did.
Suspect #3: The deploy somebody pushed on a Friday
Sometimes it's not infrastructure at all. It's a person. Someone shipped a change, and now 1 in every 2 requests throws an error that has nothing to do with traffic or databases — it's just broken code, freshly broken, by a human, this morning.
if state.bad_deploy and random.random() < state.bad_deploy_error_rate:
raise HTTPException(status_code=500, detail="unhandled exception in results-v2")
This one shows up completely differently in SigNoz — not a slow trace, a failed one, and if you're watching an error-rate panel, it looks like a cliff. One minute it's a flat 0%, the next it's spiking past 10%, and I actually had an alert configured to catch this exact thing and it fired, unprompted, the moment I flipped this scenario on. That's the difference between a dashboard and an alert: a dashboard waits for you to look. An alert taps you on the shoulder.
Suspect #4: Everyone showing up at once
And then there's the one that isn't really anyone's fault: results day, ticket booking day, sale day — a hundred thousand people click the same button in the same ten minutes, and infrastructure sized for a normal Tuesday just... isn't. I wrote a tiny script that ramps requests-per-second up over a minute, and watching SigNoz's own "requests/sec" panel climb while latency climbs right alongside it is the closest thing I've felt in this project to actually watching a results day happen.
The honest truth about this one: traffic spikes usually aren't the root cause by themselves. They're a stress test that reveals whichever of the first three suspects was already weakest.
I tried installing SigNoz with Foundry pretty cool:
curl -fsSL https://signoz.io/foundry.sh | bash
mkdir signoz-install && cd signoz-install
foundryctl cast -f casting.yaml
If your dashboards aren't showing anything, half the time it's not a code bug — it's that your telemetry is quietly being sent nowhere, because a docs page you trusted is a version behind reality. Worth checking before you assume your app is the problem.
What "try later" actually means, once you can see it
Here's the thing that changed for me after a weekend of doing this on purpose: "server down" isn't a diagnosis, it's a way of not looking. Every one of the four suspects above leaves a completely different, completely specific fingerprint the moment you have real tracing and metrics in place — a fat span, a flatlined cache-hit rate, a spike in one specific error code, a climbing RPS graph. None of them require guessing. They require someone to have wired up the telemetry before the bad day happens, not during it.
That's the actual pitch for observability tools like SigNoz, minus the sales voice: it's not about looking impressive on a dashboard nobody checks. It's about being able to say, out loud, in the middle of an incident, "it's the database" — and being right, in under a minute, instead of "server down hai, try later" and hoping it fixes itself by dinner.
Wrap-up
Next time a site goes down on you, it's not the weather. It's one of four suspects, and somewhere, someone either has the telemetry to catch it in real time or they don't. I decided to be the person who does, at least for one fake exam-results portal over one very long weekend — and honestly, watching a slow trace light up in real time is more satisfying than it has any right to be.
My Hackathon- project:
My project puts you in the shoes of the on-call SRE for a national exam-results portal. A Student Portal lets "students" check results with a roll number, DOB, and captcha. A separate Mission Control panel gives the SRE four rocker-switch toggles to inject real production incidents into the live backend:
Database Latency — injects a multi-second delay into DB spans
Redis / Cache Failure — forces every request through to Postgres directly
Deployment Bug — randomly throws 500s on ~50% of requests
Traffic Spike — paired with a load-generator script that ramps requests/sec
Flip a switch, watch the portal degrade, then open SigNoz to find the exact failing span, confirm it against a live dashboard and a firing alert, flip the switch back off, and watch it recover — all in real time.
Built for the SigNoz hackathon.
Arigato !!!!
Top comments (1)
There's a sneaky fifth suspect that hides behind suspect #1, connection pool exhaustion. The db.query span itself can look perfectly healthy, but every pooled connection is held by a slow query, so new requests never even reach that span and just queue waiting to check one out. That's the most confusing flavor of "the DB looks fine but the site is down," and it's exactly what sends people back to "try later." Would make a good fifth switch if you keep building on this.