DEV Community: Krishnam Murarka

A Handoff Is a Transfer of Authority, Not a Transfer of Context

Krishnam Murarka — Mon, 31 Aug 2026 06:22:57 +0000

The easiest way to build a multi-agent system is to let one agent call another and pass along the conversation so far. It works in a demo. It fails in production for a reason that has nothing to do with model quality: the transcript is the one thing that does not need to cross the boundary, and the two things that do need to cross it are usually missing.

What has to transfer is a task the receiver can accept or reject, and a bounded set of actions it is permitted to take. Everything else is either recoverable from a system of record or should never have left one.

So we ask a blunt question before writing any handoff: after this transfer, who is responsible for advancing the task? If the answer is the receiver, it is a handoff and it needs a contract. If the coordinator keeps the plan and only wants an answer back, that is a tool call wearing a handoff's clothes, and naming it a handoff hides where accountability actually lives.

The envelope we send is deliberately small. Objective, with a test for what counts as done. Current authoritative status. Evidence by reference: record identifiers and a retrieval path, not copied rows. Permitted actions, with limits and an expiry. A return schema. What we do not send is the transcript, the receiver's persona prompt, or the transferor's credentials. The receiver gets its own audience-bound credentials from the platform. A bearer token sitting in a handoff payload is a permission escalation with extra steps.

Separating facts from inference inside that envelope matters more than we expected. A summary that flattens "the customer said" and "the billing system returned" and "the previous agent concluded" into one paragraph of prose lets a guess get promoted to a fact at every hop, and nothing downstream can tell that it happened. We carry the class of each claim next to the claim, along with when it was retrieved, so a receiver can refresh anything stale before acting on it.

The failure that costs the most rework is subtler: treating natural language as a terminal state. An agent that reports "I've taken care of that" has not given the workflow anything it can branch on. Terminal states have to be enumerated and structured: completed, failed, cancelled, input required, approval required, timed out. A failure has to say whether retrying is safe and which effects may already have landed. Otherwise the coordinator's only recovery move is to run the work again and hope every step of it was idempotent.

Two more things we now write down at the start. Non-delegable decisions, such as releasing money above a threshold or accepting legal terms, where the specialist returns an approval request describing the proposed effect and the server re-checks that request after approval, because an approval is evidence about one specific action rather than a standing permission. And ownership of the user relationship during the transfer, because if that is not stated, both parties assume the other one sent the delay notice.

None of this makes the agents smarter. It is ordinary distributed-systems hygiene applied to a boundary that happens to have a model on either side of it, and the tests are the familiar ones: missing fields, expired authority, unsupported schema version, duplicate delivery, and a handoff chain that bounces between specialists because nobody owns the terminal decision.

The full write-up on our own site covers the rest: the field-by-field contract table, what acceptance and rejection look like, the failure matrix for stale evidence and duplicate handoff IDs, and how to trace a handoff as a linked responsibility span.

Agent Handoff Contracts: What State, Evidence, and Authority Should Transfer

Written by the engineering team at Edilec.

A CVE Is Not a Unit of Work

Krishnam Murarka — Thu, 27 Aug 2026 22:07:00 +0000

Scanners hand us CVEs, and a CVE is not something anyone can fix. What actually gets patched is a particular version of a particular component on a particular host with a particular route to it. One advisory can land on an internet-facing gateway, an isolated lab appliance, a library that ships in the image but is never invoked, and a container that has not been scheduled in months. Same identifier, four different answers to whether anybody should be paged.

So the row in our queue is a pair: the vulnerability and the instance. Everything downstream depends on that choice, because a clock and an owner can be attached to an instance and cannot honestly be attached to an identifier.

The CISA KEV catalog is the best available evidence that a vulnerability is being exploited in the wild, and it is published as machine-readable data with a schema, so it belongs in a pipeline rather than in a spreadsheet someone refreshes monthly. What it does not tell us is whether the vulnerable function is reachable here. KEV is the threat half of the join. Exposure is the half we have to prove ourselves, from an external vantage point, through the load balancers, proxies, partner links and management planes that actually carry traffic.

CVSS is useful in the same partial way. Version 4.0 separates Base, Threat, Environmental and Supplemental metric groups, and Base is explicitly intrinsic severity under standardized assumptions. A queue built on the Base number alone keeps the one part of that structure which knows nothing about our environment and discards the vector that explains the attack path. We store the vector, the version and the source. Two findings with an identical score can require completely different sequencing, and the vector is where that shows up.

CISA's BOD 26-04, issued in June 2026, moved federal prioritization off the earlier flat KEV-centered model onto asset exposure, KEV status, exploit automation and post-exploitation technical impact. It binds covered federal agencies and not us. We borrowed the decision variables anyway, because those four are the questions triage was already arguing about without naming them.

Two things fail quietly after that.

The first is the claim that something is not reachable. A package can be installed without its vulnerable code path ever being called, and VEX exists to say exactly that. We treat it as a control with an expiry rather than as a deletion. The record carries the path that was tested and the time it was tested, because a feature flag, a dependency bump or a routing change can make yesterday's analysis false without anyone touching the vulnerable code.

The second is closing a finding on a mitigation. A firewall rule can remove the route today and leave the vulnerable software in place for the next topology change or the next internal compromise to reach. Mitigation lowers urgency and does not close the row. The row closes when the changed state is independently verified on every affected instance, which means a rescan, an image inspection or a configuration check, not a ticket transition.

NIST SP 800-40 Rev. 4 frames patching as enterprise preventive maintenance rather than emergency response, and that framing is the part most teams under-invest in. The way to survive a compressed clock is to make ordinary change safe, observable and reversible long before the clock starts.

The full write-up on our own site covers the rest: the decision bands and the evidence each one requires, the treatment table from patch through isolation to time-bound acceptance, when the clock is allowed to start, and what closure verification looks like on each path.

Prioritize Patching With CISA KEV and Local Exposure, Not CVSS Alone

Written by the engineering team at Edilec.

An Outbox Makes the Write Atomic. It Does Not Make Delivery Exactly-Once.

Krishnam Murarka — Mon, 24 Aug 2026 06:47:59 +0000

The transactional outbox exists to close one gap. A service has to change its own database and tell other services about it, and a database transaction cannot commit to a broker. Write state first and the event can be lost. Publish first and you can announce a transaction that later rolls back. So you insert an event row in the same local transaction as the business mutation, and change data capture relays the committed row afterwards.

That part is genuinely solved, and it is why we reach for the pattern. What we have learned to say out loud in design review is what it does not solve.

The relay is still at-least-once. A connector can read a row, publish it, and lose the broker acknowledgement; on restart it publishes again. So the event ID has to be generated inside the business transaction and survive every hop unchanged, and the consumer has to be built to see it twice. An inbox row keyed by source plus event ID, written in the same local transaction as the effect, is the version of this we trust for local effects. Remote calls do not get that for free. They need their own idempotency identity, or their own outbox.

Two decisions then fail quietly.

The first is the broker key. Ordering is guaranteed within a partition, so the key has to be the boundary the invariant actually lives on. Key on event type and events for one order scatter across partitions. Key on tenant and a large tenant serializes behind itself. Neither is visible in a test that runs one aggregate at a time.

The second is cleanup. Outbox tables grow indefinitely, so something deletes from them, and that job can outrun CDC. A row sitting behind the connector's offset is not proof that every downstream system accepted it. We treat the deletion watermark as a separate control from connector progress, leave margin, and test a connector rebuild against it. The related failure is log retention: a long enough connector outage and the database log segments the connector still needs are gone, which turns a delayed relay into a resnapshot. That deserves an alert well before the horizon rather than at it.

The last thing we push on is lag. Outbox publication is asynchronous, which makes connector lag a product behavior and not an infrastructure metric. Someone has to answer how long after commit search, notifications, fulfillment or analytics may lag. If the honest answer is that they cannot, the outbox is the wrong shape for that workflow and polling harder will not fix it.

The full write-up on our own site covers the rest: what belongs in the event payload versus a versioned reference, why the relay must never reread current tables, the ownership split between application and platform teams, and the failure-to-recovery table we work from.

Transactional Outbox with CDC: Close the Database-to-Broker Gap

Written by the engineering team at Edilec.

A Trace ID in Your Metrics Is Not a Link to a Trace

Krishnam Murarka — Fri, 21 Aug 2026 05:53:13 +0000

Prometheus exemplars are the shortest path we know from "the slow tail on this route moved at 14:10" to one specific request. The histogram sample carries a bounded reference, usually a trace ID, and the dashboard turns the point on the graph into a link. It removes the step where an on-call engineer guesses labels and scrolls a trace search by timestamp.

What surprises teams is how many independent systems have to agree before that click resolves.

Emitting a trace ID is one hop. The latency observation has to be recorded while the request's span context is still current and valid, and that is exactly where wrapper metrics, asynchronous callbacks, thread hops and queue consumers lose it. Automatic instrumentation covering a service is not evidence that it covers those paths. The exposition format and the scraper then have to preserve the exemplar, and so does remote write, and so does the long-term store: exemplar support and retention can differ from sample support, so forwarding is worth testing on its own rather than assumed from a working metric. The dashboard has to map the ID to the correct trace data source and tenant, carrying a time range with margin for clock skew. And the trace backend has to still hold the trace.

That last one is the failure we would look for first. Metrics commonly live for months and traces for days, so a link on a two-week-old graph is predictably dead, and nothing about the graph says so.

The subtler version is a sampling mismatch. If the metric SDK records an exemplar from a trace context that tail sampling later drops, the reference was written for something that no longer exists. The reservoir policy and the trace sampling policy have to be coordinated, or two systems make independent decisions about which request mattered and then disagree.

So the rule we work to is that correlation is a path, and each hop gets tested. Measure link resolution as a rate and split the failures by cause: no active context, reservoir selection, exposition loss, storage loss, trace sampling, expiry, tenant mapping, dashboard configuration. Without that split, "exemplars sometimes don't work" turns into raising trace retention when the fault was in metric ingestion.

One thing worth saying to whoever carries the pager: an exemplar is one representative observation from a bucket. It is not the worst request in that bucket and it is not proof of cause.

The full path is written up on our own site: choosing bucket boundaries around the alert threshold, keeping trace IDs out of ordinary metric labels, what each reservoir policy costs, and the failure-injection drill (collector delay, trace-backend rejection, remote-write retry, clock skew) that tells you which hop is actually broken.

Prometheus Exemplars: Move from a Bad Metric to the Exact Trace

Written by the engineering team at Edilec.

Queue Depth Doesn't Tell You Whether You Have a Problem

Krishnam Murarka — Thu, 13 Aug 2026 19:33:56 +0000

Most autoscaling rules for worker queues are wired to depth. When the backlog grows, add consumers. That rule is right exactly when the bottleneck is consumer capacity, and wrong the rest of the time.

Depth doesn't separate the two cases you actually care about. A queue holding forty thousand short-lived cache-refresh events can be completely healthy: it drains in seconds and nothing waiting on it has a deadline. A queue holding twelve payment commands that are an hour old is an incident, and it will never trip a depth alarm.

What separates them is the age of the oldest useful message, read against what that class of work is allowed to owe. Depth is a second-order fact about arrival rate. Age is a direct statement about whether you are meeting an obligation.

Then there is what happens when the depth rule fires against the wrong bottleneck. Elastic consumers scale in seconds; the thing behind them does not. The database has a connection ceiling, the third-party API has a quota, and the review queue has a person in it. Adding workers against a saturated dependency raises latency, which triggers timeouts, which produce retries, which is more load. Invocation count goes up and completed business effects go down. The dashboard shows a system working harder while less useful work leaves it.

So the number that should govern consumer concurrency is not the backlog. It is the slowest protected dependency's sustainable throughput, divided by the worst-case number of calls a single invocation makes. That second term is the one that gets skipped. A concurrency cap of 100 on a handler that fans out to five downstream calls is a cap of 500 on the thing you were trying to protect.

We wrote up the full control policy on our own site: how to set the dependency budget, where admission belongs relative to the queue, how event-source limits and function concurrency interact, and why a retry budget has to be bounded by a deadline instead of an attempt count.

Backpressure for Serverless Systems: Concurrency, Queues, and Overload Control

It includes the signal table we work from (what each metric reveals, which control it should move, and the misreading it invites) and a failure-mode table covering connection saturation, throttled third parties, poison messages and the recovery ramp, which is the phase where a released backlog turns into the second incident.

Written by the engineering team at Edilec.

Our Database Was at 15% CPU While the API Was Timing Out

Krishnam Murarka — Mon, 10 Aug 2026 06:06:01 +0000

A few months ago we spent most of a morning chasing a latency problem that had every symptom of database overload and none of the causes.

The alert was on our main API's p95, which had climbed from about 180ms to over 9 seconds during an ordinary weekday afternoon. Not a traffic spike — request volume was within 10% of the previous day. Requests weren't failing outright at first, they were just queuing somewhere and eventually hitting our 10-second gateway timeout.

The first place we looked was the database, because that's where "slow" always seems to live. Postgres was at 15% CPU. The slow query log was empty. Replication lag was under a second. Every individual query we ran by hand came back in single-digit milliseconds. The database was, by every metric we had, bored.

Where the time was actually going

We had request timing instrumentation, but it measured the wrong boundary. It started the clock when a handler issued a query and stopped when the rows came back. That interval was fine — 4ms, 7ms, 11ms. What it didn't measure was the gap between "the handler wants a connection" and "the handler has one."

Once we put a span around connection acquisition specifically, the picture inverted immediately. Query execution: 6ms. Waiting for a connection from the pool: 8.4 seconds.

Our pool had a max size of 20. That number had been set roughly two years earlier by copying a default, and had never been revisited. It was fine for a long time, because our handlers held connections for a few milliseconds each. What changed wasn't the traffic — it was what the handlers did while holding a connection.

The actual bug: holding a connection across a network call

A feature we'd shipped a few weeks earlier did this, in effect:

conn = pool.acquire()
record = conn.query("select ... from jobs where id = $1", id)
enriched = http.get(vendor_api, record.external_id)   # 300-800ms, external
conn.execute("update jobs set ... where id = $1", id)
conn.release()

The database work here is trivial. But the connection stays checked out across an external HTTP call that averaged around 400ms and had a p99 near two seconds. With 20 connections and a handler occupying one for ~400ms, we could serve roughly 50 of those requests per second before the pool became the ceiling — and every other endpoint in the service, including ones that touched no jobs at all, queued behind them.

That's the part that makes this class of bug nasty. The failure isn't localized to the slow feature. A pool is a shared resource, so one handler with a bad hold time degrades every endpoint sharing it. Our health check endpoint was timing out, which is exactly what made this look like an infrastructure problem rather than a code problem.

What we changed

Three things, in order of how much they mattered.

We stopped holding connections across anything that isn't a query. The fix in the code above is to release the connection before the HTTP call and acquire a second one for the update — two short holds instead of one long one. That alone took pool wait from 8.4 seconds back to sub-millisecond.

We added a hold-time metric and alerted on it. Not pool utilization — hold time per checkout, at p99. Utilization tells you the pool is full; hold time tells you why. We alert when p99 hold time exceeds 100ms, which is generous for a pool that should only be serving queries, and it catches this class of regression the day it ships instead of weeks later.

We set an explicit acquisition timeout. Previously a handler would wait indefinitely for a connection, which is how a saturated pool turns into an unbounded queue and then into memory pressure. Now acquisition fails after 2 seconds and the request returns a 503. Failing fast isn't a fix, but it keeps one degraded dependency from consuming the entire service's capacity.

We did also raise the pool size, from 20 to 40 — and that was the least important change of the three. We were deliberately cautious about it, because a bigger pool would have masked the real problem for a few more weeks and then pushed the eventual failure down onto the database itself, where it would have been considerably harder to recover from.

The general lesson

Connection pools fail in a way that points at the wrong component. Every symptom — timeouts, queuing, latency that climbs with load — looks like a database problem, while the database sits at 15% CPU insisting it isn't. The only way to see it is to measure the wait for a resource separately from the use of that resource.

We now do that for every pooled resource we have: database connections, HTTP client pools, worker slots. It's about ten lines of instrumentation, and it's the difference between a twenty-minute diagnosis and a four-hour one.

If a pool has a max size, something will eventually sit at that ceiling. Better to hear about it from a metric than from a pager.

We write these up as we run into them. We're the engineering team at Edilec, where we build and maintain backend systems for growing products.

The Retry That Charged a Customer Twice, and What We Learned About Idempotency

Krishnam Murarka — Sun, 09 Aug 2026 09:34:50 +0000

We found out about it from a support ticket, not a dashboard. A customer had been billed $340 twice for the same order, four seconds apart, and wanted to know if we'd started double-charging people. We hadn't — or rather, we hadn't meant to. What we'd actually built was a payment flow that assumed retries were free, and that assumption cost us a very uncomfortable afternoon of pulling logs and a much longer week rebuilding the write path to be safe by default.

The setup was ordinary. Our checkout service called out to a payment processor, waited for a response, and marked the order paid. Like any network call, it could time out — and like any reasonable engineer, whoever wrote the original client had added a retry: if the request didn't come back in 5 seconds, try again. That's normal, defensible advice for GET requests and most reads. The problem is that "charge this card $340" is not a read, and a timeout doesn't mean the request failed. It just means we stopped waiting for the answer. The first call had actually gone through on the processor's side; the client's retry was a second, fully independent charge. Same card, same amount, no error on either end — just two successful transactions where we intended one.

The uncomfortable part of debugging it was realizing how much of our system quietly relied on "retry it and see" as a correctness strategy. Webhook handlers retried. Background jobs retried. The checkout client retried. All of that is fine for idempotent operations — re-running a read, or an update that sets a field to the same value, doesn't change the outcome. It's only unsafe for operations with side effects that aren't naturally idempotent, and a payment charge is the sharpest possible example of that category. We had been treating "make it retry-safe" as a network-layer concern instead of an operation-design concern, and it had worked fine right up until it didn't.

The fix was idempotency keys, and the part that took longest wasn't the concept — it's well documented, most payment processors support it natively — it was finding every place in our own code that needed one and didn't have it. The pattern: every write operation that could plausibly be retried gets a client-generated idempotency key, a UUID created once at the start of the user action and reused across every retry of that same action. On our side, before executing the charge, we check a dedupe table keyed on that UUID. If we've seen it before, we return the stored result of the original request instead of re-executing anything. If we haven't, we execute it and store the result, with a TTL long enough to cover realistic retry windows — we settled on 24 hours after finding a few edge cases where mobile clients retried failed requests on next app launch, sometimes the following day.

We didn't stop at payments. Once we went looking, we found three other endpoints — refund issuance, subscription upgrades, and a bulk-invite endpoint — that had the identical shape: a mutating call, a client that retried on timeout, and no dedication mechanism to tell a legitimate retry from an accidental duplicate. Refund issuance was the scariest one to find, because unlike a duplicate charge, a duplicate refund doesn't generate a customer complaint — it just quietly loses the company money until finance notices the numbers don't reconcile.

Since shipping idempotency keys across those four endpoints, duplicate-charge and duplicate-refund tickets have gone from a recurring monthly occurrence to zero. More usefully, we stopped having the argument about whether a given endpoint "needs" one — the checklist for any new mutating endpoint now includes idempotency key support as a default, the same way auth and input validation are defaults, not something you bolt on after the first incident report.

The broader lesson wasn't really about payments. It was that retry logic and idempotency are a matched pair — you can't safely add one without the other, and code that retries a non-idempotent write is a bug waiting for the right timeout to trigger it. We write about the operational lessons like this one as we run into them building Edilec's backend — if you want more of the same, edilec.com has the rest.

Automated Deploy Pipelines: Cutting Release Time From Three Days to Forty Minutes

Krishnam Murarka — Fri, 31 Jul 2026 07:34:35 +0000

For the first year of Edilec, shipping a release meant blocking out an afternoon and hoping nothing else came up. A deploy was: SSH into the box, git pull, stop the service, run migrations by hand, restart, tail the logs for ten minutes, and cross your fingers. When it worked, it took maybe forty minutes of actual effort. When it didn't, it ate the rest of the day and usually part of the next one too.

We didn't set out to fix this. We got forced into it by a release that went from "should be quick" to a three-day incident.

The release that broke the calendar

We were shipping a schema change alongside an API update — nothing exotic, just a new column and a service that needed to read from it. The migration ran fine in staging. In production, it ran fine too, technically — it just ran for eleven minutes on a table nobody had checked the size of in months, holding a lock the whole time. Requests backed up, timeouts cascaded into our queue workers, and by the time someone noticed, we had a partial deploy: half the fleet on the new code, half on the old, both talking to a database mid-migration.

Rolling back wasn't clean, because "rollback" meant someone manually reversing steps they'd typed by hand twenty minutes earlier under pressure, from memory, while alerts were firing. We got the system stable within a few hours. We spent the next two and a half days on cleanup: reconciling data written by mixed-version code, re-running migrations that had partially applied, and writing an incident doc nobody wanted to write.

The postmortem's real finding wasn't about the migration. It was that our entire release process had no repeatable shape. Every deploy was a slightly different set of manual steps performed by whoever was on point that day, which meant every deploy carried the risk of a step skipped, reordered, or misremembered. Speed wasn't really the problem. Reproducibility was.

What we built instead

We rebuilt the pipeline around one rule: nothing in a release should depend on a person remembering to do it correctly under time pressure.

One artifact, tested once. CI builds a single versioned, immutable artifact per commit. That artifact — not source, not a branch — is what moves through every environment. If it passed staging, it's bit-for-bit what reaches production. We'd previously had environment-specific build steps that meant "tested in staging" and "running in production" weren't always the same code.

Migrations run and gate separately from deploy. This was the direct fix for the incident. Migrations now run as their own pipeline stage, against a replica first to estimate lock time and row count, and anything projected to hold a lock past a threshold fails the pipeline instead of running blind in production. Schema changes and code changes are sequenced, not bundled — expand, deploy, contract, each as its own step with its own gate.

Progressive rollout with automatic health checks. New versions go to a small slice of traffic first, watched against error rate, latency, and a couple of business-specific metrics for a fixed window. If those stay within bounds, rollout proceeds automatically in increments. If they don't, the pipeline halts and rolls back the slice on its own — no one has to notice and decide, the decision is already encoded.

Rollback is a pipeline stage, not an improvisation. Every release keeps the prior artifact and its migration state addressable, so reverting is running the same pipeline backward against a known-good version, not someone reconstructing steps from memory. We tested this by rehearsing rollbacks on purpose, on a schedule, so the first real rollback wasn't also the first time anyone had run one.

Humans approve intent, not mechanics. A person still decides whether to ship and when. Nobody decides how — how is the same every time, which is the entire point.

What it bought us

Release time went from an afternoon, best case, to about forty minutes end to end, most of which is intentional soak time during progressive rollout rather than anyone doing manual work. That's the headline number, but it undersells the real change: releases stopped being events. We ship several times a day now instead of batching changes into a dreaded weekly window, which paradoxically made each individual release lower-risk, because smaller diffs are easier to reason about and easier to roll back cleanly.

The three-day incident cost us trust in our own process. The fix wasn't more caution — more caution just makes deploys slower and rarer, which concentrates risk instead of removing it. The fix was making the safe way the only way, automatically, every time.

We build this kind of infrastructure discipline into every system we ship at Edilec — if a release process depends on someone remembering the right order of steps, it's not a process yet, it's a hope.

A Dependency That's Slow Is Worse Than One That's Down: What Building Circuit Breakers Taught Us

Krishnam Murarka — Wed, 29 Jul 2026 04:43:53 +0000

We used to think the worst failure mode for a downstream service was an outage. It isn't. An outage is clean — the connection refuses, the request fails fast, your code hits a catch block and moves on. What actually took us down was a payment-verification service that didn't fail. It just got slow.

It started as a P2 ticket: a handful of customers reporting that checkout was "spinning." Our on-call engineer checked the payment service's health dashboard — green across the board, CPU normal, error rate at zero. The service was up. It just took 8 seconds to answer instead of 80 milliseconds, and about one request in twenty took 30+ seconds before timing out on our side.

Here's the part that turned a minor annoyance into a full outage: every request into checkout held a thread (and a database connection from our pool) for the entire time it waited on that call. We had a connection pool sized for a world where dependencies answer quickly. When the payment service degraded, our pool filled with connections stuck waiting on it. Once the pool was exhausted, unrelated requests — order lookups, cart updates, anything touching that same pool — started queuing behind them. Within 11 minutes, a slow dependency for one feature had turned into a site-wide outage for everything.

That's the core lesson: a hard failure is contained by definition. A slow failure spreads, because nothing in a typical request path is designed to give up early. Threads wait. Connections hold. Retries pile on top of requests that are already struggling. The system doesn't crash, it just gets quieter and quieter until it stops answering anything.

We fixed the immediate incident by manually killing connections to the payment service and restarting the pool, which is not a strategy, it's a stopgap. The actual fix was a circuit breaker in front of every external dependency call — payment verification included.

The idea is simple, but it took discipline to implement well: track success/failure and, crucially, latency on every call to a dependency, per dependency, in a rolling window. If the failure rate or the slow-response rate crosses a threshold, "trip" the breaker — stop even trying the call for a cooldown period, and fail fast with an explicit error instead. After the cooldown, let a small number of test requests through; if they succeed, close the breaker; if not, stay open. The mechanics aren't novel — we didn't reinvent anything Hystrix or resilience4j hadn't already solved — but wiring it in correctly mattered more than the algorithm.

Three details made the difference between a breaker that helped and one that just moved the problem:

We tripped on latency, not just errors. A dependency returning 200s at 10x normal latency will never trip a breaker that only counts failures — it looks "healthy" the whole time it's strangling your thread pool. We added a p95-latency threshold per dependency as a trip condition, not just an error-rate one.

We scoped breakers per dependency, not globally. An early draft used one shared breaker for "external calls," which meant a degraded shipping-rate API could trip protection for payment calls that were working fine. Each external call now gets its own breaker with thresholds tuned to that dependency's normal behavior, not a one-size-fits-all number.

We made the open state loud, not silent. A tripped breaker that just returns a generic 500 pushes the failure downstream to whatever's easiest — usually the customer, staring at a spinner. Ours returns a specific, typed error the caller can act on: for payment verification, checkout falls back to "we'll confirm and email you," rather than hanging or hard-failing the whole order.

None of this eliminates the underlying problem — the payment service still gets slow sometimes, for reasons outside our control. What changed is the blast radius. A slow dependency now costs us one degraded feature for a few minutes, not a site-wide outage. That distinction — degraded versus down — is the entire point of a circuit breaker, and it's cheap insurance once you've been paged for the alternative.

We think about resilience patterns like this constantly at Edilec (edilec.com), because most of the incidents that actually hurt aren't the ones where something breaks loudly — they're the ones where something just gets a little too slow to notice until it's too late.

How We Took Checkout From 4.2s to 220ms by Moving Almost Everything Off the Request Path

Krishnam Murarka — Tue, 28 Jul 2026 05:58:44 +0000

Our checkout endpoint had a p95 latency of 4.2 seconds. The actual work of authorizing a payment and creating an order took a small fraction of that. The rest was inventory updates, a confirmation email, an analytics event, and a couple of notifications to downstream systems — all running synchronously, all inside the same request the customer was sitting there waiting on.

None of that extra work needed to block the response. The customer's contract with us at that moment is "tell me my order went through." It is not "tell me my order went through and also confirm that six other systems have been updated." We had built the endpoint as if every one of those steps was equally urgent, when in reality exactly one of them was: authorize the payment and record the order. Everything else could happen a second later without the customer ever knowing the difference.

The fix was conceptually simple — move everything except payment authorization and order creation into a job queue, and let it run after the response goes out. In practice, most of the work wasn't in wiring up a queue. It was reworking the failure handling, because a synchronous chain of steps and an asynchronous set of independent jobs fail in different ways, and pretending otherwise is how you end up with subtler bugs than the ones you started with.

In the old synchronous version, a failure anywhere in the chain was visible immediately and (usually) rolled back correctly, because it was all one request. Move the same steps into background jobs, and you lose that for free — a failed confirmation email doesn't roll back an order, and it shouldn't, but now you need to decide, explicitly, what happens when the email job fails: does it retry, how many times, does anyone get paged, does the customer eventually get a way to know their order succeeded even if the email never lands. We ended up giving each job its own retry policy and its own success/failure visibility instead of treating "post-checkout work" as a single unit — a slow analytics pipeline shouldn't share a failure mode with a broken notification integration just because they both used to run in the same request.

The latency result was almost entirely mechanical: p95 dropped from 4.2 seconds to about 220 milliseconds because the request path itself got dramatically shorter, not because any individual step got faster. The background work still takes roughly as long as it always did. It's just no longer something a paying customer is staring at a spinner for.

The part worth remembering isn't "use a job queue," that's not exactly a novel idea. It's that moving work off the request path is only half the job — the harder half is deciding, deliberately, what each piece of that work is allowed to fail like once it's no longer wrapped in the same transaction as everything else. Skip that part and you've just traded a slow endpoint for a fast endpoint with quietly-broken side effects.

We ended up applying the same pattern to a few other endpoints afterward, once we had the retry/visibility scaffolding in place — it's become a fairly standard tool in how we build backend systems at Edilec. More on how we think about that kind of infrastructure work at edilec.com.

The Cache Key That Was Missing One Parameter, and What It Cost Us

Krishnam Murarka — Tue, 28 Jul 2026 05:57:29 +0000

A support ticket came in that made no sense at first: a customer swore they'd briefly seen someone else's search results. Not an error, not a crash — just, for a few seconds, data that wasn't theirs. Then it "fixed itself" on a page refresh, and they moved on with their day more confused than angry. We should have been a lot more alarmed than we were.

We had a shared cache layer in front of a search endpoint, keyed on the query string and the account's pricing tier. That had worked fine for a long time, because in practice most cache keys collided safely — two customers on the same tier running a similarly-shaped query usually got results that happened to match, or close enough that nobody noticed. The cache key didn't include the dimension that actually mattered: the account's specific data scope. Two different customers could produce the exact same cache key while expecting to see completely different result sets, and whichever one populated the cache first would end up serving the other one's request for the life of that cache entry.

The bug had probably existed since the caching layer was first added. It just needed two things to line up to become visible: two customers on the same tier, with overlapping-enough queries, whose underlying data had diverged enough that a shared result would actually look wrong to one of them. Early on, with few customers and sparse data, the odds of that were low. As the customer base grew, the odds went up, and the ticket we got was really just the first time we got unlucky enough for someone to notice.

The fix, once we found it, was almost anticlimactic: audit every parameter that could change what the "correct" response looks like for a given request, and make sure all of them are part of the cache key — not just the ones that seemed obviously relevant when the cache was first built. We ended up adding an automated check as much as a code fix: a test that generates two different account contexts with the same visible query parameters and asserts that they cannot produce the same cache key. That test is the thing that actually prevents this class of bug from coming back, because it doesn't rely on a person remembering to think about it during a future refactor.

What made this bug uncomfortable wasn't the complexity of the fix, it was the shape of the failure. This wasn't a performance bug or a UI glitch — it was two customers' data touching in a way that neither of them chose and neither of them could have detected reliably from their side. That's the kind of bug that's cheap to prevent up front and expensive to explain after the fact, both in engineering time and in the harder-to-quantify cost of a customer wondering what else they can't see.

The broader lesson we took from it: a cache key isn't just a performance knob, it's a security boundary as soon as any two callers can share a cache entry non-consensually. Every time we design a new cache layer now, "what could two different callers legitimately have in common, and what must always keep them apart" is a question we answer explicitly before the cache goes live, not something we patch in after a ticket forces the question.

This is the kind of correctness work that's easy to skip when a system is small and obvious in the moment, and expensive to retrofit later — it's part of why we treat this stuff as a first-class concern in how we build backend systems at Edilec, more on that at edilec.com.

We Fixed the Same Auth Bug Three Times Before We Admitted It Was One Bug

Krishnam Murarka — Mon, 27 Jul 2026 06:42:37 +0000

The third time a customer reported "I'm logged in but your API says I'm not," we finally stopped patching and asked the question we should have asked after the first report: why does this keep happening in a different service every time?

The answer was uncomfortable. We had three services — the public API, the partner integrations service, and an internal admin tool — and each one had its own copy of the JWT validation logic. Not a shared library with three call sites. Three separate implementations, written by three people, at three different points in the company's history, each one a slightly different interpretation of "check the token." One validated the exp claim with a 30-second leeway for clock skew. One didn't allow any leeway at all. One checked token revocation against a cache that refreshed every five minutes; the other two checked it on every request. None of this was documented anywhere — you only found out the behavior differed by hitting the edge case.

The bug itself was almost mundane: a token issued right at the boundary of a permission change would pass validation in one service and fail in another, because the revocation check timing didn't line up. A customer's session would look valid to the endpoint they'd used a minute ago and invalid to the one they hit next. We'd "fixed" this twice already, each time by patching the specific service that was reported broken. Both fixes were correct and both fixes were incomplete, because we were treating three symptoms of a shared design problem as three unrelated bugs.

Centralizing was the obvious answer, and the one we'd been avoiding

Centralizing auth was the obvious answer and also the part we'd been avoiding, because it touched every service and none of us wanted to own that migration. We ended up building it as a gateway-level concern rather than a shared library, for a reason that mattered more in practice than in theory: a shared library still gets forked. Someone vendors an old version, someone patches their copy under deadline pressure, and eighteen months later you have three implementations again, just with more shared git history. Pulling validation out to the edge — one gateway service that terminates auth before a request ever reaches application code — meant there was structurally only one place it could happen.

Shadow mode caught the disagreements before they were incidents

The migration took about five weeks, most of which was not writing the gateway itself but making the cutover safe. We ran the gateway in shadow mode first: it validated every request and logged what it would have decided, without actually blocking anything, while the three existing services kept doing their own checks. That surfaced real disagreements — cases where the gateway would have rejected a token one of the old services was still accepting — before any of them affected a real request. We fixed those discrepancies one at a time, in the gateway's logic, until shadow mode ran clean for a full week. Only then did we start cutting services over, one at a time, oldest and lowest-traffic first, with the option to fail open back to the service's own check if the gateway had an outage.

Consolidating the decision also consolidates the failure mode

The part that surprised us was the operational side, not the security side. Centralizing auth meant centralizing its failure mode too — a bug or outage in the gateway now affects every service at once instead of one. We treated that as a cost worth paying deliberately: the gateway runs on its own scaling policy, gets paged on its own error budget separate from any downstream service, and has a circuit breaker that fails open to a short-lived cached decision rather than blocking everything if its own dependency (the revocation store) is slow. Consolidating a decision doesn't remove the risk of getting it wrong; it just moves the risk somewhere you can actually watch it.

None of this is exotic — "auth belongs at the edge, not reimplemented per service" is close to conventional wisdom. What's easy to miss is that you don't usually get there by architecture review. You get there because the same bug keeps showing up wearing a different service's name, and eventually someone notices the pattern instead of just filing the third ticket.

We went through a similar consolidation on our own platform at Edilec after the same pattern showed up in our services, and it's since become the default we reach for whenever auth logic starts spreading across more than one place — more on how we think about that kind of infrastructure work at edilec.com.