DEV Community: Mubashir Taj

Circuit Breakers: Stop Calling the Dependency That Is on Fire

Mubashir Taj — Wed, 21 Oct 2026 13:49:20 +0000

Every backend team learns the same lesson, usually at 3 AM: your service did not fail because of your code. It failed because a dependency got slow, and your code kept calling it.

The payments API slows to thirty seconds. Your threads block waiting. Incoming requests queue behind those threads. Memory climbs, health checks fail, and the orchestrator restarts you in the middle of the backlog. The dependency never went down. It just got slow, and slowness is contagious.

Then your own retries finish the job. Every timed-out call gets retried, so a thousand requests become four thousand against an API that is already struggling. The dependency never gets a quiet moment to recover, because your good intentions keep hammering it. This is a retry storm, and it turns a partial outage into a full one.

Circuit breakers exist to stop this exact cascade. Here is how they work, and how to tune them so they protect you instead of annoying you.

The idea, stolen from electricians

An electrical breaker opens when too much current flows, and the house does not burn down. A software circuit breaker does the same for network calls: when failures cross a threshold, calls stop going out. The failing dependency gets room to breathe. Your service keeps its threads for work that can actually succeed.

The breaker sits on the client side, wrapped around every outbound call: databases, APIs, queues, all of them. It has three states.

The three states

Closed is normal operation, and the confusing name just means traffic flows. Calls pass through while the breaker counts failures in a rolling window. The classic rule: if half the calls fail over the last twenty, the breaker decides this is not a blip and trips open. The window matters. Five failures in a quiet hour should not trip anything, and a good breaker knows the difference between a blip and a pattern.

Open is the whole point. Every call to that dependency now fails instantly, without touching the network. No thread waits thirty seconds on a doomed call. Those threads stay free to serve the healthy parts of your system, and the fallback path takes over. Failing fast is the feature: you spend zero resources on work that cannot succeed.

Half-open is how recovery gets tested instead of assumed. After a cooldown period, the breaker lets exactly one probe call through. Success closes the circuit and traffic resumes. Failure opens it again with a longer wait. You never dump full traffic onto a dependency that just coughed, which is what stops the flapping.

The fallback is the product

An open circuit does not mean a broken page. It means the fallback takes over, and the fallback deserves real design work.

Serve the cached price list instead of the live one. Queue the write for later instead of failing the checkout. Show a sensible default instead of an error page. Users tolerate slightly stale data. They do not tolerate a down page.

Design the fallback first, and the breaker becomes invisible. Design it as an afterthought, and every trip is a user-visible incident. Most teams skip this step and then blame the pattern.

Tuning: where most teams get it wrong

Trip on failure rate, not raw counts. Ten failures mean nothing at midnight and everything at noon. Rates normalize across traffic; counts do not.

Count slow calls as failures. A call that takes thirty seconds is a failure even if it eventually returns. Use your p99 latency as the slow-call threshold, not the average, because averages hide the tail that is killing you.

Let the cooldown grow. A fixed five-second wait after every trip flaps against a dependency that needs a minute. Back off: seconds first, longer with each consecutive failure.

Tune per dependency, never global. The search service and the payment gateway have nothing in common: different latency profiles, different error budgets, different business cost of a fallback. One global config protects nothing well.

The supporting cast

Circuit breakers work best with two companions. Timeouts bound how long any single call can wait, so a closed breaker still cannot hang forever. Bulkheads isolate thread pools per dependency, so one slow downstream cannot eat every thread even before the breaker trips.

And you do not have to build any of this. Resilience4j in Java, Polly in .NET, gobreaker in Go, pybreaker in Python. The pattern is identical everywhere, so learn it once and recognize it in every stack.

The takeaway

Slow dependencies are inevitable. Retry storms are optional. Break the circuit before it breaks you: closed counts, open fails fast, half-open probes. Build the fallback before you need it, tune per dependency, and the next outage stays someone else's problem.

If you take one thing from this: wrap your next outbound call in a breaker this week. The incident you prevent will not make noise, and that is exactly the point.

Database Indexes: From B-Trees to Production

Mubashir Taj — Thu, 08 Oct 2026 19:12:11 +0000

Every backend eventually meets the same slow query. The table grows, the page that used to load in 50 milliseconds now takes 3 seconds, and someone says "just add an index." Sometimes that fixes it. Often it does not, and now you have slower writes too. This article covers how indexes actually work, when they help, when they hurt, and how to read a query plan before you touch anything, with Postgres examples throughout.

1. What an index really is

An index is a separate data structure that lets the database find rows without scanning the whole table. Postgres's default, the B-tree, keeps keys in sorted order across pages, so a lookup walks down the tree in logarithmic time instead of reading every row. Think of it like the index at the back of a textbook: you look up the term, get the page numbers, and jump straight there instead of reading the book cover to cover.

The key insight: an index speeds up reads by paying a price on writes. Every INSERT, UPDATE, and DELETE must also maintain each index on the table. That is the trade you are making, every time, with every index.

2. Read the plan before you add anything

The fastest optimization is information. Run EXPLAIN ANALYZE on the slow query against production-like data:

EXPLAIN ANALYZE
SELECT id, email FROM users WHERE email = 'a@example.com';

Estimated plans can lie, so look at the actual row counts. The gap between "rows estimated" and "rows actual" is where most performance mysteries live. Then check the scan type:

Seq Scan on a large table: the usual suspect. The planner read every row.
Index Scan or Bitmap Heap Scan: the index is being used. If it is still slow, check selectivity below.
Index Only Scan: the best case. The query never touched the table at all.

3. Selectivity decides whether the index matters

An index on a boolean column with two values is nearly useless: the planner still reads half the table. Indexes win when they are selective, when the lookup narrows millions of rows down to a handful. A unique email column is the perfect candidate. A status column with three values is usually not, unless it is combined with something else.

This is why "add an index on the WHERE column" is incomplete advice. Ask first: how many rows does this filter actually eliminate?

4. Composite indexes: order matters

When a query filters on two columns, one composite index beats two single-column indexes:

CREATE INDEX idx_orders_customer_date
ON orders (customer_id, created_at);

Column order follows selectivity and usage: the most selective, most frequently filtered column first. And remember the leftmost-prefix rule: this index serves queries filtering on customer_id alone, or on both columns, but not queries filtering only on created_at.

5. Covering indexes: skip the table entirely

Sometimes the query uses the index but still goes back to the table for one more column per row. Those heap fetches add up. A covering index includes the extra columns so the query is answered from the index alone:

CREATE INDEX idx_users_email_covering
ON users (email) INCLUDE (id, name);

That Index Only Scan in the plan is the payoff. Use INCLUDE for columns you read but never filter on; keep the key itself narrow.

6. The invisible index killers

These show up in plans as sequential scans that "should" have used an index:

Functions on the indexed column. WHERE lower(email) = ... disables a plain B-tree on email. Fix it with an expression index: CREATE INDEX ... ON users (lower(email)).
Type mismatches. Comparing a text column to a number forces a cast on every row. Match the types.
Leading wildcards. LIKE '%smith' cannot use a B-tree. Use trigram indexes (pg_trgm) if you need that pattern.
Stale statistics. If the planner's row estimates are wildly off, run ANALYZE on the table before blaming the index.

7. When indexes hurt

Every index costs on the write path and in storage. Tables with heavy INSERT loads and many indexes will show it in write latency. Bloated or unused indexes are pure cost: find them with pg_stat_user_indexes and drop the ones with near-zero scans. The engineers who add few deliberate indexes outperform the ones who add many hopeful ones.

Production checklist

EXPLAIN ANALYZE on production-like data before adding anything.
One composite index, ordered by selectivity, over several single-column ones.
Covering indexes (INCLUDE) for hot read paths.
Expression indexes where functions wrap filtered columns.
Monitor unused indexes with pg_stat_user_indexes; drop the dead ones.
Re-check after major data growth: selectivity changes, and so should your indexes.

Indexes are the highest-leverage tool in query performance, and the easiest to misuse. Measure first, add deliberately, and re-verify. Your write path will thank you.

What is the most surprising thing a query plan ever showed you?

Caching from Zero to Production

Mubashir Taj — Tue, 29 Sep 2026 13:19:41 +0000

Every backend eventually hits the same wall: the database cannot keep up, but the data barely changes. That is the moment caching stops being an optimization and becomes a requirement. This article walks through the full path, from the basic pattern to the failure modes that take down production systems, with Go examples throughout.

1. Why cache at all

It comes down to orders of magnitude. Reading from memory takes nanoseconds. Reading from disk takes milliseconds. A network round trip to the database takes longer still. When the same data is requested thousands of times per second and changes rarely, serving it from the database every time is pure waste.

The two numbers that justify a cache: request volume and data stability. High volume plus stable data means caching will pay for itself. Low volume or constantly changing data means you are adding operational complexity for little gain. Cache deliberately, not by default.

2. The cache-aside pattern

Cache-aside, also called lazy loading, is the most common pattern and the right starting point. The application owns the logic:

Check the cache for the key.
On a hit, return the value.
On a miss, read from the database, write the result into the cache with a TTL, then return it.

Here is a compact Go example using Redis:

func GetUser(ctx context.Context, rdb *redis.Client, db *sql.DB, id string) (*User, error) {
    key := "user:" + id

    // 1. Try the cache first
    if cached, err := rdb.Get(ctx, key).Bytes(); err == nil {
        var u User
        if json.Unmarshal(cached, &u) == nil {
            return &u, nil
        }
    }

    // 2. Miss: fall through to the database
    u, err := fetchUserFromDB(ctx, db, id)
    if err != nil {
        return nil, err
    }

    // 3. Populate the cache for next time (TTL: 5 minutes)
    if data, err := json.Marshal(u); err == nil {
        rdb.Set(ctx, key, data, 5*time.Minute)
    }
    return u, nil
}

The cache stays simple because it never talks to the database directly. The trade-off is the first request after every expiry pays the full database cost, and stale data is possible between the write to the database and the next cache refresh. For most read-heavy workloads, that trade-off is correct.

3. TTLs, and why fixed TTLs are dangerous

The TTL is the most consequential number in your caching layer, and teams often set it once and forget it. A fixed TTL on every key creates a subtle trap: keys written at the same time expire at the same time.

Consider a hot key absorbing 5,000 requests per second. Its TTL expires. In the next instant, all 5,000 requests fall through to the database simultaneously. CPU climbs from 40% to 100% in seconds, the connection pool saturates, and latencies spike. This is the thundering herd, also called cache stampede. The cache did not fail. The expiry pattern synchronized thousands of requests into a single attack on your own database.

The cheapest fix is jitter. Instead of a fixed 300-second TTL, use 300 seconds plus a random offset of up to 60 seconds. Expirations spread across time, and the synchronized cliff disappears:

// Jittered TTL: spreads expirations so hot keys never die together
ttl := 5*time.Minute + time.Duration(rand.Intn(60))*time.Second
rdb.Set(ctx, key, data, ttl)

One line of randomness removes an entire class of incident. This belongs in every caching layer from day one.

4. Stampede protections

Jitter handles the synchronized case, but a genuinely cold key under heavy load still funnels every request to the database. Three protections cover the remaining scenarios.

Request coalescing. When 5,000 requests need the same missing key, only one should query the database. In Go, golang.org/x/sync/singleflight does exactly this: concurrent callers for the same key share a single in-flight execution.

var group singleflight.Group

func GetProduct(ctx context.Context, id string) (*Product, error) {
    v, err, _ := group.Do("product:"+id, func() (interface{}, error) {
        return GetProductCached(ctx, id) // cache-aside logic from section 2
    })
    if err != nil {
        return nil, err
    }
    return v.(*Product), nil
}

5,000 concurrent requests become one database query. The rest wait microseconds for the shared result.

Stale-while-revalidate. Serve the slightly old value instantly and refresh it in the background. Implementation: on a miss, check for a stale copy kept under a separate key with a longer TTL. If present, return it immediately and trigger an async refresh. Users get a fast response, the database gets one query instead of thousands. This is the right choice when slightly stale data is acceptable, which covers far more endpoints than teams assume.

Probabilistic early refresh. For the highest-traffic keys, refresh the cache before the TTL expires, with probability increasing as expiry approaches. This keeps hot keys warm permanently and removes the miss path entirely for your most critical data.

Use all three together and stampedes become a non-issue rather than an incident category.

5. Eviction policies

Memory is finite, so Redis must decide what to remove when it fills up. The maxmemory-policy setting controls this, and the default (no eviction, writes start failing) surprises people.

The policies worth knowing:

allkeys-lru: evict the least recently used keys across everything. The best default for a general-purpose cache, because it automatically keeps hot data.
volatile-lru: evict least recently used, but only among keys that have a TTL. Use this when some keys must never be evicted.
allkeys-lfu: evict the least frequently used keys. Better than LRU when access patterns have long-term hot keys mixed with one-off scans that would pollute an LRU.
volatile-ttl: evict keys with the shortest remaining TTL first. A reasonable choice when TTLs genuinely reflect data priority.

Set it explicitly in your Redis configuration:

maxmemory 2gb
maxmemory-policy allkeys-lru

An unset eviction policy is a latent outage. Decide this before production, not during it.

6. Write strategies

Reads are only half the story. When data changes, the cache and the database must agree, or at least disagree briefly and predictably.

Cache-aside with invalidation (the common path): on write, update the database, then delete the cache key. The next read repopulates it. Simple and correct for most cases. The small risk is a read repopulating stale data between the database write and the delete, which the delete-then-verify patterns address if you need them.

Write-through: on write, update the cache and the database together. Reads never miss, but every write pays both costs, and you cache data nobody may read.

Write-behind: on write, update the cache immediately and the database asynchronously. Lowest write latency, but a crash before the async write means data loss. Only for data you can afford to lose or reconstruct.

For most systems: cache-aside with invalidation on writes. Reach for the others when a measured requirement demands it.

7. Observability: the metrics that matter

A cache you cannot observe is a liability. Four metrics cover nearly everything:

Hit ratio (hits / total requests). Below 80% on a read-heavy endpoint means the cache is not earning its keep. Investigate key design, TTLs, or whether the data is even cacheable.
p99 latency with and without cache. The gap between cache hits and database fallbacks tells you the real value of the layer, and a shrinking gap means the database path needs attention.
Eviction rate. A sudden rise means working set exceeded memory. Either grow the instance or tighten what you cache.
Stampede signals: watch for correlated spikes in database connections and cache misses on the same key pattern. That correlation is the fingerprint of a herd forming.

Alert on eviction rate and on miss-rate spikes for your top keys. Those two catch most caching incidents before users notice.

8. Production checklist

Before calling your caching layer production-ready:

TTLs are jittered, not fixed.
Request coalescing (singleflight or equivalent) guards every hot path.
An eviction policy is set explicitly and matches the workload.
Write invalidation is tested, including the failure case where the delete fails.
Hit ratio, eviction rate, and p99 latency are dashboarded and alerted.
The cache failing does not take down the application. Every cache call has a timeout and a fallback to the database. A cache outage should degrade latency, never cause an outage.

That last point deserves emphasis. The cache is an optimization layer. If its absence breaks you, the architecture has the dependency backwards.

Closing

Caching is simple to start and easy to get wrong at scale. The pattern fits in thirty lines of code, but production demands jittered TTLs, coalesced requests, explicit eviction, and real observability. Get those right and the cache becomes the quietest, most reliable part of your stack.

What is your go-to stampede protection in production?

hello developerws

Mubashir Taj — Thu, 24 Sep 2026 19:33:20 +0000

hello developers