DEV Community: Kushagra Bhalla

The Outage That Wasn't in the Runbook: A Timezone Bug Postmortem

Kushagra Bhalla — Wed, 08 Jul 2026 03:57:22 +0000

A production incident, broken down using a five-step failure framework: symptom, root cause, blind spot, fix, principle.

The symptom
A scheduled batch job completed successfully. No errors, no failed retries, no alerts. The next morning, downstream reports were wrong not missing data, just incorrect values, in a way that initially looked like a data quality issue rather than a code issue.

Where we looked first (and why it was wrong)
Because the job showed as successful, the first hour of investigation focused on the reporting layer and the data pipeline. That was the wrong layer. The job hadn't failed it had run correctly, just against the wrong time window.

Root cause
A hardcoded UTC offset had been written into the job's config years earlier, before daylight saving was ever a relevant factor for this process. It had never been revisited since.

The blind spot
This is the step most postmortems skip. The real question isn't just "what broke," it's "why did nobody catch this." In this case: the config hadn't changed in years, so it had quietly fallen out of everyone's mental model of "things that could break." Stability had been mistaken for safety.

The fix
Two parts. The immediate fix replaced the hardcoded offset with a proper timezone-aware calculation. The more important fix added a synthetic check that verifies the job executed within the correct local time window, not just that it executed at all closing the gap between "ran" and "ran correctly."

The principle
If a system's correctness depends on time, don't just trust the clock. Verify the calendar too. Time-based assumptions in long-lived scheduled jobs are one of the most common invisible failure classes in enterprise systems, precisely because they're stable for years right up until they aren't.
This is the first post in a recurring series where I break down real production incidents using this same five-step structure. If you've hit a similar time-based bug, I'd genuinely like to hear how it surfaced for you.

One-line fix after 4 hours of debugging a concurrency bug

Kushagra Bhalla — Mon, 06 Jul 2026 04:15:09 +0000

I once spent 4 hours debugging a race condition...

The fix was one line.

The bug only happened under load. Locally: "Nothing". In staging: "Nothing". In production, every time traffic spiked past a threshold, two API responses would swap their data.

User A would get User B’s policy details. Silently.

This was not a fun bug to discover.

I started the usual way: added logs everywhere, ran load tests locally, replayed requests, and traced every code path involved.

Nothing.

Then I started going deeper: thread dumps, heap analysis, and reading through every line of that service twice.

Hour 4: I finally found it.

A shared StringBuilder instance was being reused across threads without synchronization. Fast for single requests. A ticking time bomb under concurrent load.

The fix: make it a local variable inside the method instead of a class-level field.

One. Line. Change.

What I learned that day isn’t just about StringBuilder:

Shared mutable state is the source of most hard bugs.

“It works in testing” means nothing if your test doesn’t simulate real concurrency.

Thread dumps are your best friend learn to read them before you need them.

4 hours of debugging. 1 line fix. 100% worth it for what it taught me.

If you’ve never had a concurrency bug humble you, you haven’t shipped enough to production yet.

What’s the longest you’ve spent debugging something with a tiny fix?

How Netflix Serves 270 Million Users With Almost Zero DB Writes

Kushagra Bhalla — Fri, 03 Jul 2026 04:30:02 +0000

The Core Insight: Separate Read Paths from Write Paths

When you press Play, here's what actually happens:

Your request hits a CDN edge node (99% of the time, the video is already cached near you)
Your watch history and preferences are fetched from a read replica — not the primary DB
Playback events (pauses, seeks, buffer stats) are queued asynchronously, not written immediately
Only after you stop watching does a batch job sync your actual watch state

The Secret Ingredient: Apache Cassandra

Netflix uses Cassandra for user activity data. Here's why:

Designed for write-heavy, distributed workloads
Uses eventual consistency your "continue watching" list might be 2 seconds stale, but nobody cares
Scales horizontally across data centers without complex sharding logic

The Tradeoff They Made

Strong consistency was traded for performance.

Your "recently watched" might not update instantly. But 270 million people can stream simultaneously without a database bottleneck.

What This Means for You

Not every data write needs to be synchronous. Not every read needs to come from primary.

Understanding where you can afford eventual consistency is one of the most underrated distributed systems skills.

What's a system design tradeoff you've had to make at work? Let's discuss in the comments 👇

I Wrote Code to Impress My Reviewer for 2 Years. Here is What It Cost Me.

Kushagra Bhalla — Mon, 29 Jun 2026 06:46:00 +0000

For the first two years of my backend engineering career, I had a habit I didn't even know was a problem.

Every time I wrote code, there was an invisible audience in my head. Not the engineers who'd inherit the system. Not the on-call person who'd debug it at midnight. Not the junior who'd join the team six months later.

Just my code reviewer.

I optimized everything for that one person reading my PR. And I told myself it was good engineering.

It wasn't.

What "Writing for the Reviewer" Actually Looked Like

I was obsessed with looking senior in a diff view. Here is what that looked like in practice.

Instead of a simple, obvious method name like this:

public List<Order> getActiveOrders(String userId) {
    return orderRepository.findByUserIdAndStatus(userId, OrderStatus.ACTIVE);
}

I'd write something like this because it looked more "architectural":

public <T extends BaseEntity> List<T> fetchEntitiesByPredicate(
    Class<T> entityClass,
    Predicate<T> filterPredicate,
    Comparator<T> sortStrategy
) {
    return entityRegistry.stream()
        .filter(e -> entityClass.isInstance(e))
        .map(e -> entityClass.cast(e))
        .filter(filterPredicate)
        .sorted(sortStrategy)
        .collect(Collectors.toList());
}

Reviewer loved it. Called it "elegant generics usage" in the review.

Six months later that method caused a production incident because nobody could figure out what it was actually supposed to do.

The Friday Night That Changed Everything

A service went down on a Friday evening.

The on-call engineer wasn't me. It was a teammate who had joined three months earlier. He had to debug my code, under pressure, at 9 PM, without me available.

He called me after an hour.

He couldn't follow it.

Not because it was messy. Because it was written for an audience of one: the person approving it, not the person fixing it in the dark.

That call changed how I think about code permanently.

4 Questions I Ask Now Before Every PR

After that Friday night, I stopped asking "will this look good in review?" and started asking:

1. Can someone who didn't write this understand it at 2 AM?
If the answer is "probably not," the abstraction is wrong. Not the person reading it.

2. If this fails, will the error explain why or just that it failed?
Silent failures and generic exceptions are not clever engineering. They are traps with no map out.

3. Is this clever or is this clear?
Clear wins. Every single time. Clever code gets praised in review and cursed in production.

4. Would I be comfortable debugging this without the author next to me?
If not, it's not ready.

What Production-Ready Code Actually Looks Like

Here is what I changed that over-engineered method to:

public List<Order> getActiveOrders(String userId) {
    log.info("Fetching active orders for userId={}", userId);
    List<Order> orders = orderRepository.findByUserIdAndStatus(userId, OrderStatus.ACTIVE);
    log.info("Found {} active orders for userId={}", orders.size(), userId);
    return orders;
}

Boring in a PR? Absolutely.

Debuggable at 2 AM by someone who's never seen this code before? Yes.

That's the point.

The Mindset Shift Nobody Tells You About

Impressing your reviewer is a junior mindset. Not because reviewers don't matter. They do. But because the real audience for your code is production: the system running it, the logs trying to explain it, and the engineer at midnight trying to fix it.

The engineers I respect most write code that looks obvious. Every variable name says exactly what it is. Every method does exactly what it says. Every failure message tells you exactly where to look next.

Nobody taught me this explicitly. I had to get a 9 PM phone call to figure it out.

If you are a junior or mid-level engineer reading this: the goal is not to write code that makes you look smart. The goal is to write code that makes problems obvious and solutions clear.

That is what seniors actually trust. That is what gets you into production-critical systems.

What changed your approach to writing code: a code review, or a production incident? Drop it in the comments.