DEV Community: Krishnam Murarka

Our Database Was at 15% CPU While the API Was Timing Out

Krishnam Murarka — Mon, 10 Aug 2026 06:06:01 +0000

A few months ago we spent most of a morning chasing a latency problem that had every symptom of database overload and none of the causes.

The alert was on our main API's p95, which had climbed from about 180ms to over 9 seconds during an ordinary weekday afternoon. Not a traffic spike — request volume was within 10% of the previous day. Requests weren't failing outright at first, they were just queuing somewhere and eventually hitting our 10-second gateway timeout.

The first place we looked was the database, because that's where "slow" always seems to live. Postgres was at 15% CPU. The slow query log was empty. Replication lag was under a second. Every individual query we ran by hand came back in single-digit milliseconds. The database was, by every metric we had, bored.

Where the time was actually going

We had request timing instrumentation, but it measured the wrong boundary. It started the clock when a handler issued a query and stopped when the rows came back. That interval was fine — 4ms, 7ms, 11ms. What it didn't measure was the gap between "the handler wants a connection" and "the handler has one."

Once we put a span around connection acquisition specifically, the picture inverted immediately. Query execution: 6ms. Waiting for a connection from the pool: 8.4 seconds.

Our pool had a max size of 20. That number had been set roughly two years earlier by copying a default, and had never been revisited. It was fine for a long time, because our handlers held connections for a few milliseconds each. What changed wasn't the traffic — it was what the handlers did while holding a connection.

The actual bug: holding a connection across a network call

A feature we'd shipped a few weeks earlier did this, in effect:

conn = pool.acquire()
record = conn.query("select ... from jobs where id = $1", id)
enriched = http.get(vendor_api, record.external_id)   # 300-800ms, external
conn.execute("update jobs set ... where id = $1", id)
conn.release()

The database work here is trivial. But the connection stays checked out across an external HTTP call that averaged around 400ms and had a p99 near two seconds. With 20 connections and a handler occupying one for ~400ms, we could serve roughly 50 of those requests per second before the pool became the ceiling — and every other endpoint in the service, including ones that touched no jobs at all, queued behind them.

That's the part that makes this class of bug nasty. The failure isn't localized to the slow feature. A pool is a shared resource, so one handler with a bad hold time degrades every endpoint sharing it. Our health check endpoint was timing out, which is exactly what made this look like an infrastructure problem rather than a code problem.

What we changed

Three things, in order of how much they mattered.

We stopped holding connections across anything that isn't a query. The fix in the code above is to release the connection before the HTTP call and acquire a second one for the update — two short holds instead of one long one. That alone took pool wait from 8.4 seconds back to sub-millisecond.

We added a hold-time metric and alerted on it. Not pool utilization — hold time per checkout, at p99. Utilization tells you the pool is full; hold time tells you why. We alert when p99 hold time exceeds 100ms, which is generous for a pool that should only be serving queries, and it catches this class of regression the day it ships instead of weeks later.

We set an explicit acquisition timeout. Previously a handler would wait indefinitely for a connection, which is how a saturated pool turns into an unbounded queue and then into memory pressure. Now acquisition fails after 2 seconds and the request returns a 503. Failing fast isn't a fix, but it keeps one degraded dependency from consuming the entire service's capacity.

We did also raise the pool size, from 20 to 40 — and that was the least important change of the three. We were deliberately cautious about it, because a bigger pool would have masked the real problem for a few more weeks and then pushed the eventual failure down onto the database itself, where it would have been considerably harder to recover from.

The general lesson

Connection pools fail in a way that points at the wrong component. Every symptom — timeouts, queuing, latency that climbs with load — looks like a database problem, while the database sits at 15% CPU insisting it isn't. The only way to see it is to measure the wait for a resource separately from the use of that resource.

We now do that for every pooled resource we have: database connections, HTTP client pools, worker slots. It's about ten lines of instrumentation, and it's the difference between a twenty-minute diagnosis and a four-hour one.

If a pool has a max size, something will eventually sit at that ceiling. Better to hear about it from a metric than from a pager.

We write these up as we run into them. We're the engineering team at Edilec, where we build and maintain backend systems for growing products.

The Retry That Charged a Customer Twice, and What We Learned About Idempotency

Krishnam Murarka — Sun, 09 Aug 2026 09:34:50 +0000

We found out about it from a support ticket, not a dashboard. A customer had been billed $340 twice for the same order, four seconds apart, and wanted to know if we'd started double-charging people. We hadn't — or rather, we hadn't meant to. What we'd actually built was a payment flow that assumed retries were free, and that assumption cost us a very uncomfortable afternoon of pulling logs and a much longer week rebuilding the write path to be safe by default.

The setup was ordinary. Our checkout service called out to a payment processor, waited for a response, and marked the order paid. Like any network call, it could time out — and like any reasonable engineer, whoever wrote the original client had added a retry: if the request didn't come back in 5 seconds, try again. That's normal, defensible advice for GET requests and most reads. The problem is that "charge this card $340" is not a read, and a timeout doesn't mean the request failed. It just means we stopped waiting for the answer. The first call had actually gone through on the processor's side; the client's retry was a second, fully independent charge. Same card, same amount, no error on either end — just two successful transactions where we intended one.

The uncomfortable part of debugging it was realizing how much of our system quietly relied on "retry it and see" as a correctness strategy. Webhook handlers retried. Background jobs retried. The checkout client retried. All of that is fine for idempotent operations — re-running a read, or an update that sets a field to the same value, doesn't change the outcome. It's only unsafe for operations with side effects that aren't naturally idempotent, and a payment charge is the sharpest possible example of that category. We had been treating "make it retry-safe" as a network-layer concern instead of an operation-design concern, and it had worked fine right up until it didn't.

The fix was idempotency keys, and the part that took longest wasn't the concept — it's well documented, most payment processors support it natively — it was finding every place in our own code that needed one and didn't have it. The pattern: every write operation that could plausibly be retried gets a client-generated idempotency key, a UUID created once at the start of the user action and reused across every retry of that same action. On our side, before executing the charge, we check a dedupe table keyed on that UUID. If we've seen it before, we return the stored result of the original request instead of re-executing anything. If we haven't, we execute it and store the result, with a TTL long enough to cover realistic retry windows — we settled on 24 hours after finding a few edge cases where mobile clients retried failed requests on next app launch, sometimes the following day.

We didn't stop at payments. Once we went looking, we found three other endpoints — refund issuance, subscription upgrades, and a bulk-invite endpoint — that had the identical shape: a mutating call, a client that retried on timeout, and no dedication mechanism to tell a legitimate retry from an accidental duplicate. Refund issuance was the scariest one to find, because unlike a duplicate charge, a duplicate refund doesn't generate a customer complaint — it just quietly loses the company money until finance notices the numbers don't reconcile.

Since shipping idempotency keys across those four endpoints, duplicate-charge and duplicate-refund tickets have gone from a recurring monthly occurrence to zero. More usefully, we stopped having the argument about whether a given endpoint "needs" one — the checklist for any new mutating endpoint now includes idempotency key support as a default, the same way auth and input validation are defaults, not something you bolt on after the first incident report.

The broader lesson wasn't really about payments. It was that retry logic and idempotency are a matched pair — you can't safely add one without the other, and code that retries a non-idempotent write is a bug waiting for the right timeout to trigger it. We write about the operational lessons like this one as we run into them building Edilec's backend — if you want more of the same, edilec.com has the rest.

Automated Deploy Pipelines: Cutting Release Time From Three Days to Forty Minutes

Krishnam Murarka — Fri, 31 Jul 2026 07:34:35 +0000

For the first year of Edilec, shipping a release meant blocking out an afternoon and hoping nothing else came up. A deploy was: SSH into the box, git pull, stop the service, run migrations by hand, restart, tail the logs for ten minutes, and cross your fingers. When it worked, it took maybe forty minutes of actual effort. When it didn't, it ate the rest of the day and usually part of the next one too.

We didn't set out to fix this. We got forced into it by a release that went from "should be quick" to a three-day incident.

The release that broke the calendar

We were shipping a schema change alongside an API update — nothing exotic, just a new column and a service that needed to read from it. The migration ran fine in staging. In production, it ran fine too, technically — it just ran for eleven minutes on a table nobody had checked the size of in months, holding a lock the whole time. Requests backed up, timeouts cascaded into our queue workers, and by the time someone noticed, we had a partial deploy: half the fleet on the new code, half on the old, both talking to a database mid-migration.

Rolling back wasn't clean, because "rollback" meant someone manually reversing steps they'd typed by hand twenty minutes earlier under pressure, from memory, while alerts were firing. We got the system stable within a few hours. We spent the next two and a half days on cleanup: reconciling data written by mixed-version code, re-running migrations that had partially applied, and writing an incident doc nobody wanted to write.

The postmortem's real finding wasn't about the migration. It was that our entire release process had no repeatable shape. Every deploy was a slightly different set of manual steps performed by whoever was on point that day, which meant every deploy carried the risk of a step skipped, reordered, or misremembered. Speed wasn't really the problem. Reproducibility was.

What we built instead

We rebuilt the pipeline around one rule: nothing in a release should depend on a person remembering to do it correctly under time pressure.

One artifact, tested once. CI builds a single versioned, immutable artifact per commit. That artifact — not source, not a branch — is what moves through every environment. If it passed staging, it's bit-for-bit what reaches production. We'd previously had environment-specific build steps that meant "tested in staging" and "running in production" weren't always the same code.

Migrations run and gate separately from deploy. This was the direct fix for the incident. Migrations now run as their own pipeline stage, against a replica first to estimate lock time and row count, and anything projected to hold a lock past a threshold fails the pipeline instead of running blind in production. Schema changes and code changes are sequenced, not bundled — expand, deploy, contract, each as its own step with its own gate.

Progressive rollout with automatic health checks. New versions go to a small slice of traffic first, watched against error rate, latency, and a couple of business-specific metrics for a fixed window. If those stay within bounds, rollout proceeds automatically in increments. If they don't, the pipeline halts and rolls back the slice on its own — no one has to notice and decide, the decision is already encoded.

Rollback is a pipeline stage, not an improvisation. Every release keeps the prior artifact and its migration state addressable, so reverting is running the same pipeline backward against a known-good version, not someone reconstructing steps from memory. We tested this by rehearsing rollbacks on purpose, on a schedule, so the first real rollback wasn't also the first time anyone had run one.

Humans approve intent, not mechanics. A person still decides whether to ship and when. Nobody decides how — how is the same every time, which is the entire point.

What it bought us

Release time went from an afternoon, best case, to about forty minutes end to end, most of which is intentional soak time during progressive rollout rather than anyone doing manual work. That's the headline number, but it undersells the real change: releases stopped being events. We ship several times a day now instead of batching changes into a dreaded weekly window, which paradoxically made each individual release lower-risk, because smaller diffs are easier to reason about and easier to roll back cleanly.

The three-day incident cost us trust in our own process. The fix wasn't more caution — more caution just makes deploys slower and rarer, which concentrates risk instead of removing it. The fix was making the safe way the only way, automatically, every time.

We build this kind of infrastructure discipline into every system we ship at Edilec — if a release process depends on someone remembering the right order of steps, it's not a process yet, it's a hope.

A Dependency That's Slow Is Worse Than One That's Down: What Building Circuit Breakers Taught Us

Krishnam Murarka — Wed, 29 Jul 2026 04:43:53 +0000

We used to think the worst failure mode for a downstream service was an outage. It isn't. An outage is clean — the connection refuses, the request fails fast, your code hits a catch block and moves on. What actually took us down was a payment-verification service that didn't fail. It just got slow.

It started as a P2 ticket: a handful of customers reporting that checkout was "spinning." Our on-call engineer checked the payment service's health dashboard — green across the board, CPU normal, error rate at zero. The service was up. It just took 8 seconds to answer instead of 80 milliseconds, and about one request in twenty took 30+ seconds before timing out on our side.

Here's the part that turned a minor annoyance into a full outage: every request into checkout held a thread (and a database connection from our pool) for the entire time it waited on that call. We had a connection pool sized for a world where dependencies answer quickly. When the payment service degraded, our pool filled with connections stuck waiting on it. Once the pool was exhausted, unrelated requests — order lookups, cart updates, anything touching that same pool — started queuing behind them. Within 11 minutes, a slow dependency for one feature had turned into a site-wide outage for everything.

That's the core lesson: a hard failure is contained by definition. A slow failure spreads, because nothing in a typical request path is designed to give up early. Threads wait. Connections hold. Retries pile on top of requests that are already struggling. The system doesn't crash, it just gets quieter and quieter until it stops answering anything.

We fixed the immediate incident by manually killing connections to the payment service and restarting the pool, which is not a strategy, it's a stopgap. The actual fix was a circuit breaker in front of every external dependency call — payment verification included.

The idea is simple, but it took discipline to implement well: track success/failure and, crucially, latency on every call to a dependency, per dependency, in a rolling window. If the failure rate or the slow-response rate crosses a threshold, "trip" the breaker — stop even trying the call for a cooldown period, and fail fast with an explicit error instead. After the cooldown, let a small number of test requests through; if they succeed, close the breaker; if not, stay open. The mechanics aren't novel — we didn't reinvent anything Hystrix or resilience4j hadn't already solved — but wiring it in correctly mattered more than the algorithm.

Three details made the difference between a breaker that helped and one that just moved the problem:

We tripped on latency, not just errors. A dependency returning 200s at 10x normal latency will never trip a breaker that only counts failures — it looks "healthy" the whole time it's strangling your thread pool. We added a p95-latency threshold per dependency as a trip condition, not just an error-rate one.

We scoped breakers per dependency, not globally. An early draft used one shared breaker for "external calls," which meant a degraded shipping-rate API could trip protection for payment calls that were working fine. Each external call now gets its own breaker with thresholds tuned to that dependency's normal behavior, not a one-size-fits-all number.

We made the open state loud, not silent. A tripped breaker that just returns a generic 500 pushes the failure downstream to whatever's easiest — usually the customer, staring at a spinner. Ours returns a specific, typed error the caller can act on: for payment verification, checkout falls back to "we'll confirm and email you," rather than hanging or hard-failing the whole order.

None of this eliminates the underlying problem — the payment service still gets slow sometimes, for reasons outside our control. What changed is the blast radius. A slow dependency now costs us one degraded feature for a few minutes, not a site-wide outage. That distinction — degraded versus down — is the entire point of a circuit breaker, and it's cheap insurance once you've been paged for the alternative.

We think about resilience patterns like this constantly at Edilec (edilec.com), because most of the incidents that actually hurt aren't the ones where something breaks loudly — they're the ones where something just gets a little too slow to notice until it's too late.

How We Took Checkout From 4.2s to 220ms by Moving Almost Everything Off the Request Path

Krishnam Murarka — Tue, 28 Jul 2026 05:58:44 +0000

Our checkout endpoint had a p95 latency of 4.2 seconds. The actual work of authorizing a payment and creating an order took a small fraction of that. The rest was inventory updates, a confirmation email, an analytics event, and a couple of notifications to downstream systems — all running synchronously, all inside the same request the customer was sitting there waiting on.

None of that extra work needed to block the response. The customer's contract with us at that moment is "tell me my order went through." It is not "tell me my order went through and also confirm that six other systems have been updated." We had built the endpoint as if every one of those steps was equally urgent, when in reality exactly one of them was: authorize the payment and record the order. Everything else could happen a second later without the customer ever knowing the difference.

The fix was conceptually simple — move everything except payment authorization and order creation into a job queue, and let it run after the response goes out. In practice, most of the work wasn't in wiring up a queue. It was reworking the failure handling, because a synchronous chain of steps and an asynchronous set of independent jobs fail in different ways, and pretending otherwise is how you end up with subtler bugs than the ones you started with.

In the old synchronous version, a failure anywhere in the chain was visible immediately and (usually) rolled back correctly, because it was all one request. Move the same steps into background jobs, and you lose that for free — a failed confirmation email doesn't roll back an order, and it shouldn't, but now you need to decide, explicitly, what happens when the email job fails: does it retry, how many times, does anyone get paged, does the customer eventually get a way to know their order succeeded even if the email never lands. We ended up giving each job its own retry policy and its own success/failure visibility instead of treating "post-checkout work" as a single unit — a slow analytics pipeline shouldn't share a failure mode with a broken notification integration just because they both used to run in the same request.

The latency result was almost entirely mechanical: p95 dropped from 4.2 seconds to about 220 milliseconds because the request path itself got dramatically shorter, not because any individual step got faster. The background work still takes roughly as long as it always did. It's just no longer something a paying customer is staring at a spinner for.

The part worth remembering isn't "use a job queue," that's not exactly a novel idea. It's that moving work off the request path is only half the job — the harder half is deciding, deliberately, what each piece of that work is allowed to fail like once it's no longer wrapped in the same transaction as everything else. Skip that part and you've just traded a slow endpoint for a fast endpoint with quietly-broken side effects.

We ended up applying the same pattern to a few other endpoints afterward, once we had the retry/visibility scaffolding in place — it's become a fairly standard tool in how we build backend systems at Edilec. More on how we think about that kind of infrastructure work at edilec.com.

The Cache Key That Was Missing One Parameter, and What It Cost Us

Krishnam Murarka — Tue, 28 Jul 2026 05:57:29 +0000

A support ticket came in that made no sense at first: a customer swore they'd briefly seen someone else's search results. Not an error, not a crash — just, for a few seconds, data that wasn't theirs. Then it "fixed itself" on a page refresh, and they moved on with their day more confused than angry. We should have been a lot more alarmed than we were.

We had a shared cache layer in front of a search endpoint, keyed on the query string and the account's pricing tier. That had worked fine for a long time, because in practice most cache keys collided safely — two customers on the same tier running a similarly-shaped query usually got results that happened to match, or close enough that nobody noticed. The cache key didn't include the dimension that actually mattered: the account's specific data scope. Two different customers could produce the exact same cache key while expecting to see completely different result sets, and whichever one populated the cache first would end up serving the other one's request for the life of that cache entry.

The bug had probably existed since the caching layer was first added. It just needed two things to line up to become visible: two customers on the same tier, with overlapping-enough queries, whose underlying data had diverged enough that a shared result would actually look wrong to one of them. Early on, with few customers and sparse data, the odds of that were low. As the customer base grew, the odds went up, and the ticket we got was really just the first time we got unlucky enough for someone to notice.

The fix, once we found it, was almost anticlimactic: audit every parameter that could change what the "correct" response looks like for a given request, and make sure all of them are part of the cache key — not just the ones that seemed obviously relevant when the cache was first built. We ended up adding an automated check as much as a code fix: a test that generates two different account contexts with the same visible query parameters and asserts that they cannot produce the same cache key. That test is the thing that actually prevents this class of bug from coming back, because it doesn't rely on a person remembering to think about it during a future refactor.

What made this bug uncomfortable wasn't the complexity of the fix, it was the shape of the failure. This wasn't a performance bug or a UI glitch — it was two customers' data touching in a way that neither of them chose and neither of them could have detected reliably from their side. That's the kind of bug that's cheap to prevent up front and expensive to explain after the fact, both in engineering time and in the harder-to-quantify cost of a customer wondering what else they can't see.

The broader lesson we took from it: a cache key isn't just a performance knob, it's a security boundary as soon as any two callers can share a cache entry non-consensually. Every time we design a new cache layer now, "what could two different callers legitimately have in common, and what must always keep them apart" is a question we answer explicitly before the cache goes live, not something we patch in after a ticket forces the question.

This is the kind of correctness work that's easy to skip when a system is small and obvious in the moment, and expensive to retrofit later — it's part of why we treat this stuff as a first-class concern in how we build backend systems at Edilec, more on that at edilec.com.

We Fixed the Same Auth Bug Three Times Before We Admitted It Was One Bug

Krishnam Murarka — Mon, 27 Jul 2026 06:42:37 +0000

The third time a customer reported "I'm logged in but your API says I'm not," we finally stopped patching and asked the question we should have asked after the first report: why does this keep happening in a different service every time?

The answer was uncomfortable. We had three services — the public API, the partner integrations service, and an internal admin tool — and each one had its own copy of the JWT validation logic. Not a shared library with three call sites. Three separate implementations, written by three people, at three different points in the company's history, each one a slightly different interpretation of "check the token." One validated the exp claim with a 30-second leeway for clock skew. One didn't allow any leeway at all. One checked token revocation against a cache that refreshed every five minutes; the other two checked it on every request. None of this was documented anywhere — you only found out the behavior differed by hitting the edge case.

The bug itself was almost mundane: a token issued right at the boundary of a permission change would pass validation in one service and fail in another, because the revocation check timing didn't line up. A customer's session would look valid to the endpoint they'd used a minute ago and invalid to the one they hit next. We'd "fixed" this twice already, each time by patching the specific service that was reported broken. Both fixes were correct and both fixes were incomplete, because we were treating three symptoms of a shared design problem as three unrelated bugs.

Centralizing was the obvious answer, and the one we'd been avoiding

Centralizing auth was the obvious answer and also the part we'd been avoiding, because it touched every service and none of us wanted to own that migration. We ended up building it as a gateway-level concern rather than a shared library, for a reason that mattered more in practice than in theory: a shared library still gets forked. Someone vendors an old version, someone patches their copy under deadline pressure, and eighteen months later you have three implementations again, just with more shared git history. Pulling validation out to the edge — one gateway service that terminates auth before a request ever reaches application code — meant there was structurally only one place it could happen.

Shadow mode caught the disagreements before they were incidents

The migration took about five weeks, most of which was not writing the gateway itself but making the cutover safe. We ran the gateway in shadow mode first: it validated every request and logged what it would have decided, without actually blocking anything, while the three existing services kept doing their own checks. That surfaced real disagreements — cases where the gateway would have rejected a token one of the old services was still accepting — before any of them affected a real request. We fixed those discrepancies one at a time, in the gateway's logic, until shadow mode ran clean for a full week. Only then did we start cutting services over, one at a time, oldest and lowest-traffic first, with the option to fail open back to the service's own check if the gateway had an outage.

Consolidating the decision also consolidates the failure mode

The part that surprised us was the operational side, not the security side. Centralizing auth meant centralizing its failure mode too — a bug or outage in the gateway now affects every service at once instead of one. We treated that as a cost worth paying deliberately: the gateway runs on its own scaling policy, gets paged on its own error budget separate from any downstream service, and has a circuit breaker that fails open to a short-lived cached decision rather than blocking everything if its own dependency (the revocation store) is slow. Consolidating a decision doesn't remove the risk of getting it wrong; it just moves the risk somewhere you can actually watch it.

None of this is exotic — "auth belongs at the edge, not reimplemented per service" is close to conventional wisdom. What's easy to miss is that you don't usually get there by architecture review. You get there because the same bug keeps showing up wearing a different service's name, and eventually someone notices the pattern instead of just filing the third ticket.

We went through a similar consolidation on our own platform at Edilec after the same pattern showed up in our services, and it's since become the default we reach for whenever auth logic starts spreading across more than one place — more on how we think about that kind of infrastructure work at edilec.com.

We Killed 400 Clients' 5-Second Polling Loop and Cut Server Load by 90%

Krishnam Murarka — Sun, 26 Jul 2026 23:39:06 +0000

Our status dashboard feature started life as the simplest thing that could work: every connected client polled a status endpoint every five seconds. It was easy to build, easy to reason about, and for the first year, with maybe 30 concurrent clients, it was completely invisible as a cost. By the time we had around 400 clients polling continuously, that endpoint was our single highest-traffic route by a wide margin, and most of those requests were doing nothing but confirming that nothing had changed.

The math is what finally forced the conversation. Four hundred clients polling every five seconds is 80 requests per second sustained, every second of every day, regardless of whether any status ever changed — and in practice, status changes were rare, maybe a few per minute across the whole client base. We were running a database query and serializing a response 80 times a second to deliver, on average, almost no new information. The server cost of that endpoint had become disproportionate to the actual amount of data flowing through it.

Moving to WebSocket push meant rethinking who initiates the update, not just changing the transport

The naive migration is "same data, pushed instead of pulled," but that undersells what actually changes: with polling, the client decides when to check. With push, the server decides when something is worth telling anyone about, which meant we had to go find every code path that mutated status and make sure it fired an event into our pub/sub layer at the moment of the change, not just leave that logic implicit in "the next poll will pick it up." That surfaced two places where status changes weren't clearly demarcated — cleanup jobs that updated records in bulk without a clear single point to hook an event into. We had to add that instrumentation before push was viable at all, which took longer than the WebSocket plumbing itself.

We kept a polling fallback rather than making WebSocket mandatory

Some of our clients sit behind corporate proxies that don't handle long-lived WebSocket connections well, and we didn't want the migration to trade one failure mode (server load) for another (silently broken clients behind a hostile proxy). Clients now attempt a WebSocket connection on load; if it fails or drops, they fall back to a much slower poll — every 60 seconds instead of 5 — as a safety net, not a primary channel. In practice under 3% of clients end up on the fallback path, but that 3% would have been a very bad debugging session if we hadn't planned for it.

Reconnection logic mattered more than we expected. A WebSocket connection dying — Wi-Fi hiccup, laptop sleep, load balancer cycling — is normal, constant, background noise at 400-client scale, and our first version handled reconnects by just re-establishing the socket and waiting for the next push. That has a real gap: any status change that happened during the disconnected window is silently missed, because push, unlike polling, doesn't naturally "catch up" a client that wasn't listening. We fixed this by having clients request a state snapshot immediately on reconnect, before resuming push, so a dropped connection costs at most one extra request instead of a permanently stale UI until the next unrelated change happens to arrive.

The load drop was almost exactly what the math predicted, which was oddly satisfying. Sustained request volume on the old polling endpoint dropped by about 90% once the majority of clients moved to push, with the remaining load being connection heartbeats (much cheaper than a full status query) plus that small fallback-polling cohort. Just as important as the raw server load number: status changes now reach clients within a second or two of happening instead of up to five seconds later, which for a monitoring dashboard is a meaningfully different product experience, not just a cost optimization.

The lesson that stuck with us wasn't really about WebSockets specifically — it was that polling intervals are a tax you pay whether or not anything happened, and that tax scales with client count in a way push architectures don't. Five seconds felt harmless when we picked it. It stopped being harmless somewhere around client number 200, and we didn't notice until the bill did.

Rethinking how a system delivers updates, rather than just how fast it delivers them, is a pattern we come back to a lot in the systems we build at Edilec — more on our approach at edilec.com.

How Routing Reads to Replicas Took Our Primary From 90% CPU to 20%

Krishnam Murarka — Sun, 26 Jul 2026 23:32:09 +0000

For most of a year, every query our application made — reads and writes alike — went to the same primary Postgres instance. It worked fine until it didn't: our primary was sitting at a sustained 85-90% CPU during business hours, query latency was creeping up across the board, and every time we ran a reporting query for an internal dashboard, customer-facing endpoints got measurably slower for the few seconds it took to run. We'd been putting off the obvious fix — read replicas — because it sounded like it would touch a lot of application code. It touched less than we expected, and the part that actually mattered wasn't the infrastructure, it was deciding which reads were safe to move.

The infrastructure part was almost boring. We stood up two read replicas using standard streaming replication, put them behind a connection string our ORM could route to separately from the primary, and confirmed replication lag stayed under 100ms under normal load. That part took an afternoon. The part that took two weeks was going through every code path that touched the database and classifying each query as either "must go to the primary" or "safe on a replica."

Read-your-own-writes is where this gets hard

The naive version of read replicas — send all writes to primary, send all reads to replicas — breaks the moment a user does something like update their profile and then immediately view their profile page, and the view happens to route to a replica that hasn't caught up yet. They see their old data and assume the save failed. We didn't want to solve this with a blanket "wait for replication" because that reintroduces the latency problem we were trying to fix. Instead we tagged specific request flows — anything immediately following a write in the same user session — to pin to the primary for a short window (we used 5 seconds, which covered our observed replication lag with a comfortable margin), while everything else defaulted to replicas.

We built this as a decorator on our query layer, not a config toggle scattered through the codebase. Every database call already went through a thin query-execution wrapper, so adding a read_preference parameter there — defaulting to replica, explicitly set to primary only where a caller needed strict consistency — meant we didn't have to hunt down every call site by hand. We did still have to review each one, but the review was "does this specific query need to be primary" rather than "how do I even route this differently."

The categories that had to stay on the primary surprised us a little

Obviously anything in a write transaction. Less obviously: any read used to make an authorization decision (we did not want a stale replica to be the reason a permission check passed when it shouldn't have), and any read feeding into a financial calculation, even one that felt low-stakes, because a few seconds of staleness compounding across a report was a correctness bug we didn't want to explain to anyone. Analytics queries, search, activity feeds, and the vast majority of page-load reads all moved to replicas without any observable behavior change for users.

Monitoring replication lag became a first-class metric, not an afterthought. Before this project we weren't watching lag at all because it didn't affect anything. Once application correctness depended on it staying low, we added alerting at 2 seconds (warning) and 10 seconds (page), and we load-tested what happens to our primary-pinning window if lag spikes beyond our 5-second assumption during, say, a bulk data migration. It does happen occasionally during large batch jobs, and our on-call runbook now includes "check replication lag" as a standard step whenever someone reports seeing stale-looking data.

The result was primary CPU dropping from the high 80s/low 90s down to around 20% under the same traffic, and query latency on customer-facing endpoints becoming noticeably more consistent, since it was no longer sharing capacity with every internal reporting query in the company. The infrastructure change was the easy part. Knowing exactly which reads could tolerate a few seconds of staleness and which absolutely could not — that's the part that actually determines whether a read-replica migration is safe or is a very slow-motion correctness incident.

We think through this same read/write consistency tradeoff on most systems we build at Edilec, since it comes up anywhere read volume outgrows what a single primary should carry — more on how we approach it at edilec.com.

The 40x Traffic Spike From One Partner: What Token-Bucket Rate Limiting Actually Buys You

Krishnam Murarka — Sun, 26 Jul 2026 23:29:26 +0000

One of our integration partners pushed a firmware update to their fleet of devices, and every device in that fleet decided to sync at once. Our API request volume from that single partner went from roughly 40 requests a second to just over 1,600 in the space of about ninety seconds. Nothing was malicious about it — it was a retry storm caused by a config change on their side, the kind of thing that happens to every API provider eventually. What made it interesting for us wasn't the spike itself, it was watching which parts of our rate limiting held and which parts didn't.

We'd had rate limiting in production for over a year, but it was a fixed-window counter: X requests per client per 60-second window, reset on the minute. Fixed windows have a well-known flaw that we'd read about and shrugged off as theoretical — you can get up to 2x your limit in a short burst if it straddles the window boundary, because a client can spend its full quota in the last second of one window and its full quota again in the first second of the next. What we hadn't accounted for was what happens when a client isn't trying to burst but genuinely has 40x normal traffic sustained for over a minute. The fixed window doesn't smooth that out at all — it just rejects everything past the limit with a hard 429, uniformly, the instant the counter fills. For a partner with real, if misconfigured, traffic, that meant near-total request failure for the worst two minutes of the incident, plus a wall of retries hitting us right as we were trying to recover.

Why token buckets fit this problem better

A token bucket holds a maximum number of tokens, refills at a steady rate, and every request costs one token. The bucket depth (max tokens) controls how much burst you tolerate; the refill rate controls your actual sustained throughput limit. We sized ours so normal traffic never touches the ceiling, short bursts (the kind real clients produce constantly — a page load firing five API calls at once) drain the bucket but don't empty it, and sustained overload drains it and then throttles smoothly at the refill rate instead of hard-cutting at a window boundary. The partner's spike still got rate limited — that's the point — but it degraded gracefully into a steady trickle of allowed requests instead of alternating between "everything succeeds" and "everything 429s."

Per-client buckets, not one global bucket

Our first implementation used a single bucket per API key, which was right, but we'd initially discussed a simpler global bucket per endpoint to save on Redis calls. That would have meant one partner's spike could throttle every other partner hitting the same endpoint. We kept per-key buckets and ate the extra Redis round-trip; a shared cluster with pipelining made this a non-issue in practice, and it meant the incident was fully contained to the one partner who caused it. Nobody else even saw elevated error rates.

429s need a Retry-After header, or your rate limiting fights your clients' retry logic instead of cooperating with it. We were already setting this, but the incident showed us our retry-after values were too conservative — we were telling clients to back off longer than the bucket actually needed to refill, which meant well-behaved clients were retrying later than necessary and (ironically) sometimes bunching up their retries into new bursts. Tightening the header to match the real refill math smoothed out a secondary wave of thundering-herd retries we hadn't anticipated.

We also learned we needed a way to talk to the partner directly, not just rate limit them. Rate limiting protected us, but the partner's own dashboards were showing them a wall of failed syncs with no obvious explanation. We added a lightweight webhook we can fire at a partner's registered ops contact when they're being sustained-rate-limited for more than a few minutes, with the current bucket state and refill rate included. It turned what used to be a confused support ticket into a five-minute conversation where their engineer just fixed the retry config on their end.

None of this stopped the spike from happening — that's not really rate limiting's job. What it did was turn a 40x traffic surge from an incident that degraded the platform for everyone into one that was contained, visible, and self-correcting within a couple of minutes. That containment is really the whole value proposition of rate limiting done right: not preventing bad traffic, but making sure it can't take down the good traffic sitting next to it.

We treat this kind of isolation as a baseline requirement now for anything at Edilec that serves multiple external partners on shared infrastructure — you can read more about how we think about API reliability at edilec.com.

Webhook Retries Aren't Optional: The 6-Hour Silent Failure That Changed How We Build Them

Krishnam Murarka — Sun, 26 Jul 2026 21:52:17 +0000

We found out our webhook delivery was broken the same way most teams do: a customer told us. Not an alert, not a dashboard going red — an email asking why a payment status update from three hours earlier had never shown up in their system. By the time we'd pulled the logs, we counted just over six hours where a downstream endpoint had been failing silently, and every event meant for it was gone. Not queued, not retried. Gone.

The endpoint in question belonged to a mid-sized integration partner whose ops team had rotated a TLS cert and, in the process, briefly served a handshake our client library didn't like. Our webhook sender treated that as a delivery failure, logged it at a level nobody was watching, and moved on. There was no retry logic — we'd built a fire-and-forget publisher because in two years of running it, deliveries had basically always succeeded. That streak was the problem. It meant we'd never had to think about what "basically always" leaves out.

The fix looked simple on a whiteboard and took us about three weeks to get right in production, mostly because the interesting failure modes only show up under real load and real partner flakiness.

Exponential backoff, not fixed intervals

Our first instinct was to retry every 30 seconds for five minutes. That's exactly the pattern that turns one struggling downstream service into a thundering herd — every failed webhook from every affected customer retries in lockstep, which is a great way to keep a recovering endpoint down. We moved to backoff starting at 10 seconds and doubling up to a ceiling of about 30 minutes, with jitter of ±20% on every attempt so retries from different events don't stack on the same tick. Six attempts over roughly two hours, then the event moves to a dead-letter queue instead of disappearing.

The dead-letter queue is the actual fix, not a nice-to-have

The bug wasn't "we don't retry enough times" — it was "when retries are exhausted, the event vanishes." Our DLQ is a plain durable table: event ID, destination, payload, attempt history, and last error. Nothing fancy. What made it useful was building the replay tool at the same time as the queue, not months later. An event sitting in a DLQ that nobody can inspect or resubmit is just a slower way to lose data. We wired ours to page on-call once an endpoint accumulates more than 20 dead-lettered events in an hour, because that's a much stronger signal of "this partner's endpoint is actually down" than any single failed request.

Idempotency keys had to come first, not after

Retries only work if replaying a webhook is safe. We'd been lucky that most of our event handlers were naturally idempotent, but "most" isn't a guarantee, and a payment-status webhook is exactly the kind of event where a duplicate delivery causing a duplicate downstream action is worse than the original failure. Every event we send now carries a UUID in the payload and header, and we document — loudly, in the integration guide — that consumers are expected to dedupe on it. That's a contract change, not just an internal one, and we spent real time making sure existing partners knew before we shipped it.

Distinguishing "retry this" from "don't bother" mattered more than we expected

A 500 from a downstream service is worth retrying. A 400 because the payload is malformed is not — retrying it six times over two hours just delays the DLQ entry and the alert that would have caught the real bug faster. We now classify failures at send time and only apply backoff to the response codes that are actually likely to resolve on their own.

None of this is novel — retry-with-backoff-and-DLQ is a well-worn pattern. What's easy to underestimate is how much of the value is in the boring parts: the replay tooling, the alerting thresholds, the idempotency contract with partners. The queue itself is a day of work. Making failures visible and recoverable instead of silent is the part that actually prevents the next 2 a.m. email from a customer telling you something broke hours ago.

We ended up rebuilding a good chunk of our event-delivery pipeline around these lessons at Edilec, and it's now the default pattern we reach for anytime a service needs to guarantee delivery to something outside our control — you can see more of how we approach this kind of reliability work at edilec.com.

Zero Trust for Business Applications

Krishnam Murarka — Thu, 23 Jul 2026 13:16:19 +0000

Originally published on the Edilec blog: https://edilec.com/blog/sec-4108/zero-trust-for-business-applications/

Zero trust replaces implicit confidence based on network location with explicit, risk-informed decisions about each resource request. For a business application, that means authenticating people and workloads, authorizing the requested action on the specific resource, limiting session and credential scope, and using telemetry to reassess access. It does not mean distrusting employees as people, nor does it describe one product.

Define scope in business and risk terms

Bring together application owners, identity teams, endpoint operations, data stewards and support to map high-value resources and current access paths. Trace real requests from a person or workload through authentication, authorization and data access, including remote and partner scenarios.

Inventory protected resources, data sensitivity, users, workloads and existing access paths.
Express authorization in business actions and objects, not only network zones or broad roles.
Strengthen identity, lifecycle and recovery according to risk.
Give services verifiable identities and protect service-to-service calls.
Collect policy, device, session and resource telemetry without making one signal absolute.
Migrate one high-value journey with fallback and support before expanding.

Design the lifecycle, decisions and boundaries

Represent access as a decision over subject, workload, device, resource, action and context. Enforcement must occur at the resource-facing layer for each meaningful request, not only at network entry.

Resolve implementation details early

Assign owners for resource classification, identity lifecycle, device evidence, authorization policy, enforcement and incident response. Document token lifetime, revocation, policy distribution, telemetry freshness, regional dependencies, privacy limits and emergency access.

Roll out with observable gates

Baseline the current outcome, delay, failure demand and risk before changing the process.
Build a thin end-to-end path and test contracts, permissions, telemetry and recovery.
Run in simulation, shadow or limited-production mode where the control permits it.
Release to a named cohort with an owner, support coverage, stop conditions and rollback steps.
Review technical signals and business outcomes together; investigate segment differences.
Expand only when exceptions are handled reliably and operating documentation matches reality.
Retire the previous path, credentials, jobs and access after evidence and retention needs are satisfied.

Operate the capability after launch

Establish governance around protected resources and access policies, not merely the zero-trust tooling. Review privileged roles, policy exceptions, service identities, signal quality and resources lacking enforcement. Workload identity is essential because modern applications call one another more often than people call each service directly. Give each deployable workload a distinct identity, authenticate both ends where feasible and authorize the specific API operation.

Key takeaways

Start from a complete business outcome and the evidence needed to trust it.
Make identity, authority, state, exception handling and ownership explicit.
Design failure, recovery and reconciliation before expanding volume.
Roll out to controlled cohorts with measurable gates and practiced rollback.
Treat configuration, policy, access and retirement as continuing product work.

More from Edilec: RBAC design for internal tools (edilec.com/blog/sec-9002) and Edilec's cybersecurity services (edilec.com/services/cybersecurity).