Our checkout endpoint had a p95 latency of 4.2 seconds. The actual work of authorizing a payment and creating an order took a small fraction of that. The rest was inventory updates, a confirmation email, an analytics event, and a couple of notifications to downstream systems — all running synchronously, all inside the same request the customer was sitting there waiting on.
None of that extra work needed to block the response. The customer's contract with us at that moment is "tell me my order went through." It is not "tell me my order went through and also confirm that six other systems have been updated." We had built the endpoint as if every one of those steps was equally urgent, when in reality exactly one of them was: authorize the payment and record the order. Everything else could happen a second later without the customer ever knowing the difference.
The fix was conceptually simple — move everything except payment authorization and order creation into a job queue, and let it run after the response goes out. In practice, most of the work wasn't in wiring up a queue. It was reworking the failure handling, because a synchronous chain of steps and an asynchronous set of independent jobs fail in different ways, and pretending otherwise is how you end up with subtler bugs than the ones you started with.
In the old synchronous version, a failure anywhere in the chain was visible immediately and (usually) rolled back correctly, because it was all one request. Move the same steps into background jobs, and you lose that for free — a failed confirmation email doesn't roll back an order, and it shouldn't, but now you need to decide, explicitly, what happens when the email job fails: does it retry, how many times, does anyone get paged, does the customer eventually get a way to know their order succeeded even if the email never lands. We ended up giving each job its own retry policy and its own success/failure visibility instead of treating "post-checkout work" as a single unit — a slow analytics pipeline shouldn't share a failure mode with a broken notification integration just because they both used to run in the same request.
The latency result was almost entirely mechanical: p95 dropped from 4.2 seconds to about 220 milliseconds because the request path itself got dramatically shorter, not because any individual step got faster. The background work still takes roughly as long as it always did. It's just no longer something a paying customer is staring at a spinner for.
The part worth remembering isn't "use a job queue," that's not exactly a novel idea. It's that moving work off the request path is only half the job — the harder half is deciding, deliberately, what each piece of that work is allowed to fail like once it's no longer wrapped in the same transaction as everything else. Skip that part and you've just traded a slow endpoint for a fast endpoint with quietly-broken side effects.
We ended up applying the same pattern to a few other endpoints afterward, once we had the retry/visibility scaffolding in place — it's become a fairly standard tool in how we build backend systems at Edilec. More on how we think about that kind of infrastructure work at edilec.com.
Top comments (0)