DEV Community: Sheryar Ahmed

When something breaks across 20 microservices, one question decides how fast you recover

Sheryar Ahmed — Mon, 03 Aug 2026 17:15:15 +0000

It's 2 A.M. and Your Dashboard Just Says "Something Is Wrong"

A field guide to observability — logs, metrics, and traces — for everyone who has ever debugged twenty microservices with grep and prayer.

It's 2:14 a.m. Your phone is doing the thing.

Caption: "on-call, living the dream."

Checkout is failing. Not for everyone, not all the time, but enough that support is drowning and someone has already tweeted a screenshot with the words "is this app down or is it just me." You open your laptop, one eye still asleep, and pull up the dashboard.

A big red number is climbing. Error rate: up. Something is wrong.

And that is all it tells you. Something.

Caption: "Dashboard: 'Something is wrong.' Me: great, love that for us, very specific."*

Here's the trap you're standing in. Your system isn't one program anymore. It's twenty services, spread across a cluster, written by different people in different languages, calling each other in a chain you couldn't draw from memory if your life depended on it. One "place order" click quietly fans out into a dozen network hops. One of them is failing. Maybe two. Your dashboard knows the house is on fire. It has absolutely no idea which room.

So you do what everyone does at 2 a.m. You start SSH-ing into boxes. You grep logs. You guess. "Probably the payment service." You read a thousand lines and find nothing. "The database?" Fine. The clock ticks, users keep failing to check out, and somewhere in that chain of twenty services the actual problem is sitting there, arms crossed, invisible, waiting for you to trip over it.

This — this exact moment — is what observability is for. Not to look impressive on a screen. To answer one question, fast: where is the problem, and why?

Let me actually teach you how it works, start to finish, so that nights like this stop being three-hour horror movies and start being ninety-second fixes. No prior knowledge needed. Just bring the caffeine.

First, a distinction that everything else hangs on

People throw around "monitoring" and "observability" like they mean the same thing. They don't, and the gap between them is the whole story.

Monitoring is the smoke alarm. You decided in advance what to watch — error rate, CPU, latency — and built a dashboard for it. It's brilliant at one job: screaming when a known thing goes wrong. "There is smoke." But a smoke alarm cannot tell you that the wiring in the upstairs bathroom shorted out. It only knows: smoke.

Observability is being able to walk through the burning house and ask questions you never planned for. "Why are only Android users, only in one region, only after 6 p.m., only when paying by card, getting slow checkouts?" Nobody builds a dashboard that specific ahead of time. But if your system is constantly describing itself — emitting enough signal — you can start at the alarm and follow the smoke all the way to the one shorted wire, after the fire starts, without shipping new code to investigate.

Monitoring tells you the house is on fire. Observability tells you which room. You need both, but at 2 a.m., the alarm was never the hard part. Finding the room is.

And to find the room, you need three different kinds of signal. Each answers a different question, and the entire skill is knowing which one to grab.

The three signals (a.k.a. your detective kit)

Treat your broken checkout like a crime scene. You've got three tools, and they are not interchangeable.

Metrics — the vital-signs monitor

Metrics are numbers over time: requests per second, error rate, latency. Cheap, always running, glanceable. Metrics are what spiked and woke you up. Their job is to answer is something wrong, and how bad?

What metrics cannot do is tell you which patient or why. A metric is an aggregate — the heart-rate line jumping — and it will never name the one request that died, because to stay cheap it blended that request into a million others. That's not a bug. That's the deal: you traded per-request detail for a signal that runs 24/7 and costs almost nothing.

If you only ever watch a handful of metrics, watch these four — the industry calls them the Four Golden Signals: latency (how slow), traffic (how much demand), errors (how many are failing), and saturation (how full your system is). Those four are the vital signs of basically any service. Get them on a dashboard and you have honest, always-on vitals.

Traces — the detective tailing the suspect through the building

A trace is the story of one single request as it travels across all your services, broken into timed steps called spans. The gateway received it (span one). The gateway called orders (span two). Orders called payments (span three). Each span has a start, a duration, and a parent. Lay them out and you get a waterfall — a visual timeline of exactly where that one request spent its time, and exactly where it fell over.

When your metrics scream "checkout is slow," the trace is what walks you down the hallway, past the five services that were perfectly fine, and stops you at the door of the one that wasn't. Traces answer the question metrics can't: where?

Logs — the security-camera footage

When the trace stops you at the payment service's door, the log tells you what actually happened inside the room: "payment processor declined: connection timed out after 5000ms." Logs are the detailed, event-level story of one specific thing. Richest signal you have, and the most expensive to keep — which is why they're only useful if they're written like data, not diary entries.

Here's the difference, because it matters. This is a log written like a diary entry:

2026-08-02 02:14  payment failed for bob

Useless. To find every failed payment you're back to grep and regex. Now here's the same event written like data:

{"level":"error","service":"payments","trace_id":"a1b2c3…","msg":"processor declined","reason":"timeout"}

That you can query: "give me every error from the payments service in the last hour." The rule for logs, then, is short: log JSON, one event per line, and put a trace ID in every single one. (Hold onto that trace ID. It's about to become the hero of the story.)

So: metrics say is it broken. Traces say where. Logs say what. Three tools, three questions, one investigation.

Now for the part almost every tutorial skips — the three ideas that separate people who merely own dashboards from people who can actually use them. They're all a little counterintuitive, and once they click, you will never look at a system the same way.

Truth #1: Averages are filthy liars

Say your checkout handles a hundred requests. Ninety-nine come back in 100 milliseconds. One takes ten full seconds.

The average? About 200 milliseconds. You glance at it and think: healthy. Ship it.

But one in every hundred customers just waited ten seconds for a page. At scale that's thousands of furious people a day, and your dashboard smiled the entire time. The average didn't lie by being wrong — it lied by being an average. It smeared one catastrophe across ninety-nine good experiences until it vanished.

There's an old line: a statistician drowned crossing a river that was, on average, three feet deep.

Caption: "the average response time is fine, sir."

This is why people who run big systems almost never look at averages. They look at percentiles. The p50 (median) is your typical experience. The p95 is what most users feel — only one in twenty is slower. The p99 is the painful tail, the one-in-a-hundred where the angry tweets are born. When an engineer says "our p99 blew up," they mean "the average looks fine, but our worst-served users are suffering, and I can see it." That visibility is the whole difference between a system you think is healthy and one you know is.

Truth #2: The thing that makes metrics cheap can also blow them to pieces

Metrics are cheap because they're aggregated. But there's a landmine buried in that word, and it takes down real systems.

Every metric can be sliced by labels: error rate by service, by status code, by region. Each unique combination of labels is a separate line the database must store and hold in memory. Slice by service and status? Maybe sixty lines. Fine. But the second someone adds a label like user_id, and you have a million users, you've just asked your metrics database to track sixty million lines. It falls over and dies. This has a name — cardinality explosion — and it's one of the most popular ways teams accidentally nuke their own monitoring.

Caption: "me, after adding user_id as a metric label."*

The lesson is a rule that keeps you alive: labels are for things with a small, bounded set of values — service, status code, region. They are never for unbounded things — user IDs, emails, request IDs. And this is exactly where the three signals snap together into a system: when you want per-user, per-request detail, you don't cram it into a metric. That's what logs and traces are for. Every signal has a job. The whole art is not forcing one to do another's.

Truth #3: The real magic is the thread that ties all three together

Here's the idea that turns this from "three separate tools" into a superpower.

Remember that trace ID I told you to keep in every log line? It also rides along on every metric and every span. One string, following one request across every service it touches. And when all three signals carry the same one, something genuinely magical happens.

You're staring at a metric spike. You click it — and you're looking at an example trace from that exact moment. You see the payment span glowing red. You click that — and you're reading the exact log lines that request produced: "processor timeout." You never opened a terminal. You never guessed. You followed one thread from "something's wrong" all the way to "here's the precise line and the precise reason," across a system written in three languages, in about ninety seconds.

Caption: "…all carrying the same trace_id."*

For that thread to survive, one thing has to work. When service A calls service B, it has to pass the trace ID along, tucked inside the request. There's an open standard for this — a header literally called traceparent — and every service in the chain reads it and continues the same story instead of starting a new one. Break that hand-off anywhere (an ancient library, a queue that drops the header) and your one beautiful trace shatters into disconnected fragments and the magic dies.

The good news is you don't wire this by hand per vendor. OpenTelemetry is the open, vendor-neutral standard for producing all three signals and passing that thread along. You instrument your code once against it, and you can send the data to any backend you like without rewriting anything. Instrument once, send anywhere, never get locked in. So the entire discipline, in one breath: instrument everything with OpenTelemetry, and never drop the thread.

Truth #4 (bonus): Alert on what your users feel, not what your servers feel

This one saves your sleep, so it earns a spot.

You could alert on a thousand things. Most of them will wake you at 2 a.m. for absolutely nothing. The rule that rescues your sanity: alert on symptoms your users feel, not on internal causes.

High error rate is a symptom — someone can't check out right now. Wake me. Slow p95 is a symptom — the app feels broken. Wake me. But CPU at 90%? That is not a symptom. A server pinned at 90% while happily serving fast, successful responses is a server doing its literal job. Page a human for that and you've taught your team that alerts are noise — so the day a real one fires, everyone scrolls past it. CPU and memory are for diagnosis, something you check after a symptom already pulled you in. They should never be the thing that pulls you in.

And give every alert a little patience: "error rate above 5% for two minutes," not "for one second." That short wait absorbs the meaningless blips so the only things that ever wake you are real. An alert that isn't worth acting on isn't an alert. It's a notification you'll learn to hate.

What it looks like when it's all actually running

Concepts are slippery until you watch them move, so let me paint the finished picture.

Imagine a small but real distributed system — three services, and on purpose written in two languages: a gateway in Node.js, an orders service in Python, and a payments service back in Node.js. A single request to the gateway fans out through all three, exactly like your checkout. Around them sits the modern stack: Prometheus collecting metrics, Loki holding logs, Tempo storing traces, an OpenTelemetry Collector routing all of it, and Grafana on top as one single pane of glass.

Send one request, and you watch it become all three signals at once. It shows up as a spike on the metrics dashboard. It shows up as structured log lines. And it shows up as one distributed trace whose waterfall runs from Node, into Python, and back into Node — living proof that the thread survived the trip across languages. Click the trace, jump to its logs. Click a log line, jump back to its trace. That round trip, in one tool, is the entire idea made physical. Watching a single request stitch three languages into one story is the moment observability stops being a buzzword and becomes obvious.

Now rewind to 2 a.m.

Same night. Same fire. But this time your system can explain itself.

Your phone buzzes. Error rate climbing. You open the dashboard — but instead of shrugging at "something is wrong," you click into an example failing trace from that exact minute and watch the waterfall light up the payment span in angry red. You click it. The logs are right there: "payment processor declined: connection timed out." You check the processor's status page — they're having an incident. You flip a feature flag to fail over to your backup provider, checkout recovers, and you are back in bed before the coffee would have finished brewing.

Caption: "found it in 90 seconds. good night."

Same fire. Same twenty services. The difference isn't that you got smarter at 2 a.m. It's that your system could finally tell you where it hurt.

Where to start, if you're starting from zero

Don't try to boil the ocean. Do these three, in this order:

First, put the Four Golden Signals — latency, traffic, errors, saturation — on one dashboard, so you have honest vital signs. Second, make your logs structured: JSON, one event per line, a trace ID in every one, so they're queryable instead of grep-bait. Third, add tracing with OpenTelemetry, so you can follow one request across every service and jump between all three signals.

Do that, and the next time something breaks across your services, you won't be the person SSH-ing into boxes and guessing at 2 a.m. You'll be the person who clicks three times, points at the exact failing service, says "found it," and goes back to sleep while everyone else is still reaching for the terminal.

That, in the end, is all observability really is: giving your system a voice, so that when it's hurting, it can finally just tell you where.

If you'd like to see all of this running — the three services, the cross-language trace, every signal lighting up in Grafana — I built the whole thing as a hands-on demo and walk through it live. Even better, it runs with one command, so you can break it on purpose and watch it tell on itself. Link's in the comments.

"an AI-interview experience with zero AI and zero cost." Opens with the three bills you didn't pay (TTS, Whisper, proctoring vendor)

Sheryar Ahmed — Tue, 07 Jul 2026 20:57:07 +0000

A mock interviewer that speaks, listens, and proctors with no AI bill

Demo: https://drive.google.com/file/d/13WEiJXqwq7O6XzouhUR0LpNfgz67znd9/view?usp=sharing
Mentors on my platform wanted to run practice interviews with their mentees: a voice
asks a question, the candidate answers out loud, it gets recorded, and — because it's
practice for the real thing the session is proctored. Every off-the-shelf version of
this hands you three metered bills: a TTS service to read the question, an STT
service (usually Whisper) to transcribe the answer, and a proctoring vendor to watch the
room. Three subscriptions for a feature people use in short bursts.

The naive plan is to just pay them. Wire up a cloud TTS voice, POST each audio answer to
a speech-to-text endpoint, embed a proctoring SDK, and eat the per-minute cost forever.
It works, but every one of those is a recurring charge, a data-egress path, and a privacy
footprint — for something that's fundamentally the candidate's own browser talking to
the candidate's own microphone. I didn't want a network round-trip sitting between a
mentee and a practice question.

The decision

I asked how much of the stack the browser already ships, and the answer was: nearly all
of it. The whole "AI interviewer" experience is three native Web APIs in a trench coat.

TTS — the interviewer speaks. window.speechSynthesis renders the question on-device.
The mentor even tunes the pitch/rate so the interviewer ("Aria") has a personality. No
audio is generated on a server; nothing is downloaded.

STT — the answer transcribes itself. SpeechRecognition (webkitSpeechRecognition
in Chromium) does live, streaming transcription on-device — interimResults means
words land on screen as they're spoken, the same engine behind your phone keyboard's mic
button. No upload, no Whisper invoice.

Audio streaming — the recording is the ground truth. MediaRecorder captures the mic
to a chunked audio/webm blob. That blob is the only byte that touches my server, and
it goes to storage I already pay for.

// TTS: the interviewer reads the prompt (mentor-tuned voice)
const u = new SpeechSynthesisUtterance(prompt);
u.rate = interviewer.rate; u.pitch = interviewer.pitch;
speechSynthesis.cancel();            // never overlap two questions
speechSynthesis.speak(u);

// STT: live, streaming transcript — no server in the loop
const rec = new (window.SpeechRecognition || window.webkitSpeechRecognition)();
rec.continuous = true; rec.interimResults = true;
rec.onresult = (e) => setTranscript([...e.results].map(r => r[0].transcript).join(''));

// Audio streaming: MediaRecorder is the ground truth if STT is unsupported
const mr = new MediaRecorder(stream, { mimeType: 'audio/webm' });
mr.ondataavailable = (e) => e.data.size && chunks.push(e.data);

The key judgment call is layering, not just "use the free thing." STT support is uneven
(Firefox notably lags), so transcription is never the source of truth — the recording
is. STT degrades to a nicety; the feature never breaks. And a green-room mic check runs a
lightweight VAD-style level meter off the Web Audio AnalyserNode (RMS on the time-
domain data) so a candidate can see the mic is live before the clock starts — again,
zero network, pure getByteTimeDomainData.

Tally: TTS $0, STT $0, proctoring $0. The only recurring cost is the audio blob storage I
already had.

The gotcha

Proctoring is where "free browser signals" nearly bit me. The harness itself is cheap and
honest — fullscreen lock, tab/blur focus-loss counting, copy/paste/right-click blocking,
periodic webcam snapshots, all buffered client-side and flushed in batches. The trap was a
kit option I was quietly proud of: "require camera."

I shipped it, then during testing found the guarantee was hollow. A candidate could allow
the camera at the start, then turn it off mid-interview from the browser's site
controls. The session sailed on — and worse, my snapshot loop kept firing, dutifully
uploading black frames. On the surface everything looked compliant. "Require camera"
that you can switch off after ten seconds isn't a requirement; it's decoration.

The fix is one signal: a MediaStreamTrack's health. When a camera is turned off, covered,
or unplugged, its track goes muted/ended and readyState stops being 'live'. Poll
that, and two behaviors fall out of the same check.

const track = (video.srcObject as MediaStream)?.getVideoTracks?.()[0];
const live = !!track && track.readyState === 'live' && track.enabled && !track.muted;

// (1) block instead of silently continuing
if (!live) { setCameraLive(false); log('camera_off'); }   // → full-screen "turn it back on" overlay

// (2) never upload a black frame from a dead camera
if (!track || track.readyState !== 'live' || track.muted) return; // skip the snapshot

Now when the camera drops, the runner throws a full-screen block — "your camera is off,
turn it back on to continue" with a one-click Retry camera — and the candidate can't
answer or advance until it's back. The clock keeps running and the off→on gap is logged,
so turning the camera off to buy thinking time is a recorded act, not a free one. And the
mentor's proctor gallery gets a clean "Camera turned off" flag instead of a wall of
black rectangles. Same track state, both jobs.

What I'd do next

STT confidence + a manual-correction pass. On-device recognition trails Whisper on accents and noise. I'd surface a confidence score and let the candidate fix the transcript, keeping the audio as the authority.
Real VAD, not just a level meter. The RMS reading is enough for a "mic is live" cue; proper voice-activity detection could auto-trim dead air and auto-advance on silence.
Honest limits on proctoring. Client-side signals raise the cost of cheating; they don't make it impossible (a second device still exists). The goal is deterrence and an auditable timeline for a mentor — not a courtroom — and the UI should say so.

The meta-lesson: before you add an AI line-item, check what window already does for free.
The browser quietly grew a whole speech-and-media stack — TTS, streaming STT, MediaRecorder,
Web Audio VAD, MediaStreamTrack health — and most of us keep reaching for an API instead.

What's the last feature you paid a vendor for that the platform could already do TTS,
STT, or something else?

I added dark mode by editing CSS variables not 100 components

Sheryar Ahmed — Tue, 30 Jun 2026 13:21:06 +0000

I added dark mode by editing CSS variables - not 100 components

My app had ~1,200 hardcoded indigo-* and ~2,600 hardcoded slate-* class usages across
130-odd components. The brief was small to say and brutal to do: "ship dark mode, and
let me rebrand the accent color whenever I want."

The naive plan is a sweep: open every component, add dark: variants, swap colors. That's
weeks of work, and worse, it's permanent work - every new component re-incurs it, and
the design drifts the moment two people touch it. I wanted dark mode to be a property of
the theme, not of every component.

The decision

Two ideas, both "edit the tokens, not the markup":

1. One brand token. Instead of components knowing the color indigo, they reference a
semantic brand scale. I did a one-time codemod - indigo- → brand- across app/,
components/, lib/ (~1,230 mechanical edits, no logic change) - and defined the brand
scale once as CSS variables. Now recoloring the entire app, globally or per-user, is a
variable swap. No component ever needs editing again.

:root {
  --brand-50:#EEF4FF; /* … */ --brand-500:#0066FF; --brand-600:#0052D6; /* … */
}
/* Tailwind v4: expose them so bg-/text-/border-/ring-brand-* all resolve to the vars */
@theme inline {
  --color-brand-500: var(--brand-500);
  --color-brand-600: var(--brand-600);
  /* …the full scale… */
}

Per-user accent "vibes" then become trivial - each preset just re-defines that scale under
an attribute selector, so picking one recolors the app instantly with zero component churn:

html[data-accent="emerald"] { --brand-500:#059669; --brand-600:#047857; /* … */ }

2. Dark mode as a token remap, not a component sweep. This is the part that surprised
me.

The mechanism (and the wrong turn I took first)

My first attempt failed silently. I tried mapping Tailwind's slate scale through my own
indirection in @theme inline - and text stayed dark in dark mode. Tailwind's default
slate-* utilities weren't picking up my override.

Then I read the compiled CSS, and the trick fell out of it. Tailwind v4 emits its own
palette as real CSS variables, and the utilities reference them:

.text-slate-900 { color: var(--color-slate-900); }

So I don't need to touch a single utility. I just redefine the slate ramp inside the
.dark selector - reversed, so "dark text" becomes light and "light background" becomes
dark:

.dark {
  --color-slate-900: #f1f5f9;  /* was near-black → now near-white text */
  --color-slate-700: #cbd5e1;
  --color-slate-200: #3a4659;  /* was light border → now a dark border */
  --color-slate-50:  #273345;  /* was page-white → now an elevated surface */
  color-scheme: dark;          /* native scrollbars/date pickers follow too */
}

Every text-slate-900, bg-slate-50, border-slate-200 in the codebase flips
automatically. Thousands of usages, one block of CSS.

The gotcha

A reversed ramp isn't a straight inversion - surface hierarchy breaks if you're naive.
My first ramp made slate-50 darker than my card color, so every "subtle gray" inner box
(inside drawers and cards) suddenly looked like the page bleeding through the card. The
fix was to treat the low slate shades as elevated surfaces that sit above the card,
not below it: canvas (#0b1120) < card (#1e293b) < slate-50/100/200. Dark mode isn't
"flip the colors" - it's "preserve the depth ordering with a dark palette." Miss that and
everything looks flat and muddy.

A second, smaller one: native <select> option lists. Components set a dark background on
the control, but the option popup inherited the dark bg without a light text color -
dark-on-dark, invisible. One global rule fixed all of them at once:

select option { background-color: var(--card); color: var(--card-foreground); }

What I'd do next

A no-flash inline script in <head> to set the theme before first paint (right now a provider gates render, which avoids the flash but costs a frame).
Audit the handful of intentionally dark elements (a dark CTA, a tooltip) that I had to pin to fixed colors so the remap couldn't invert them - those are the exceptions that prove the rule.
A contrast-ratio check in CI so a future accent preset can't ship an unreadable pair.

The meta-lesson: if changing a visual property means editing many files, the property is
in the wrong layer. Push color and theme into tokens, and "rebrand the app" or "add dark
mode" turns from a sprint into a diff.

What's your dark-mode strategy - dark: everywhere, CSS variables, or something else?