Most of the event-driven architecture I've built started the same way: a synchronous call chain that worked fine in a demo and fell over under real load. On a multi-tenant B2B SaaS backend I worked on, an ingestion service took in batches of records from a customer's connected system and processed them the obvious way: validate the batch, write it, call a processing service synchronously to derive whatever depended on it, then call a notification service to tell whoever cared. One request in, three services deep before anything went back to the caller.
It worked fine until ingestion volume grew past what the slowest service in that chain could handle. Everyone already knows the fix is "put a queue in the middle." What's less talked about is what you actually inherit once you do that: ordering questions, duplicate delivery, backpressure, poison messages. And those same problems show up again, almost unchanged, in how an AI agent runs a sequence of tool calls. That second part is the one people get wrong, because it doesn't look like a queueing problem at first glance.
Why the chain breaks, and it's never the average case
The naive pipeline calls processing and notifications inline, in the request path. It survives every load test built on average-case numbers, because averages are exactly what hide this failure. What actually kills it is tail latency compounding across hops: processing might run 200ms at p50 and 4 seconds at p99 (a slow query, a lock contention spike, a downstream dependency having a bad afternoon), and ingestion inherits that p99 whether it wants to or not. Every additional hop adds its own tail, and the whole chain ends up as slow as its worst-behaved member, not its average one.
A few things break specifically as volume climbs:
A slow downstream service backpressures everything ahead of it. If notifications is degraded, ingestion requests start timing out too, even though ingestion did nothing wrong. There's no shared database or shared deploy causing that; it's just one service holding a call open while it waits on another.
A failure partway through the chain is ambiguous. If the call to processing times out, did it finish the work and respond slowly, or never get the request at all? The caller has no way to know, so it can't safely decide whether to retry: retry a request that already landed and you risk double-processing, skip the retry on one that never landed and you silently lose work.
And scaling one stage tends to force you to provision for all of them together, because they're sized as one synchronous unit even when their real load patterns look nothing alike — ingestion spikes at particular hours, processing load doesn't necessarily follow it.
None of that is a bug in any single service. It's what synchronous coupling costs once the services on either end stop sharing the same performance and failure profile, which past a certain point they always do.
The fix, and what it actually costs you
The standard move: ingestion writes the batch, publishes an event ("record batch ingested for account X, batch Y") to a queue or topic, and returns immediately. Processing and notifications each pick that event up on their own schedule, and neither being slow or down blocks ingestion from taking the next batch. That decoupling is the part everyone talks about. The part that actually matters more is what you sign up for the moment a queue enters the picture: at-least-once delivery, not exactly-once.
Nearly every message queue in production — SQS, Kafka running typical at-least-once consumer semantics, RabbitMQ under normal config — will redeliver a message under some failure condition: a consumer crashing after processing but before acknowledging, a network blip, a redelivery timeout firing early. That's not a misconfigured queue. It's the honest tradeoff a distributed queue makes, because exactly-once delivery over a network is either impossible or expensive enough that it stops resembling a queue. Which means: if your consumer isn't idempotent, adding a queue didn't fix your duplicate-processing bug, it just moved it somewhere less visible.
This is where a pattern I've written about before — deriving a natural key straight from the request data instead of minting a client-side idempotency token — pays off again in a different context. A consumer reacting to "record batch ingested" needs the same protection any synchronous handler needs against a retried request: a deterministic key built from the event (account_id + batch_id), a uniqueness constraint the database enforces, and a path that catches the violation and returns the existing result instead of erroring out. The mechanism doesn't care whether the duplicate arrived as an HTTP retry or a queue redelivery — the underlying problem is identical either way.
@KafkaListener(topics = "record-batch-ingested")
public void onRecordBatchIngested(RecordBatchEvent event) {
String idempotencyKey = event.accountId() + ":" + event.batchId();
try {
processedBatchRepository.insertProcessedMarker(idempotencyKey, event);
recordProcessingService.process(event);
} catch (DuplicateKeyException alreadyProcessed) {
log.info("Duplicate delivery for {}, skipping processing", idempotencyKey);
// Ack and move on. Expected, not an error.
}
}
Ordering is the tradeoff without a tidy answer. A single Kafka partition (or an SQS FIFO queue) gives ordering within that partition, at the cost of throughput — one consumer, one message at a time, per partition. Partition on something arbitrary and you get parallelism with no ordering guarantee across related events. Partition by account_id and you get ordering for everything that matters to that account specifically — usually the real requirement is "did this account's events land in the order they happened," not global ordering across the whole system — while still parallelizing across accounts. Get this wrong and it rarely shows up as an obvious bug. It shows up three weeks later as a config update applied before the data event it was supposed to modify, and nobody's quite sure why one account's numbers looked wrong for exactly one day.
Agent orchestration is the same problem, minus the broker
Here's the connection that made this worth writing up. An AI agent working through a multi-step task — "find the records that failed validation last month and retry them" — is structurally an event-driven system whether or not anyone building it thought of it that way. The agent decides to call a tool, the call goes out (often asynchronously: a queued job, a webhook callback, a long-running request), a result comes back, and that result feeds the next decision. Decide, dispatch, wait, decide again. That's a consumer loop. Every tradeoff from above shows up again here, usually with no broker enforcing any discipline around it.
Retries are redelivery, and need the same treatment. If an agent's call to a retryRecord API times out, the agent has no better way of knowing whether the retry landed than the synchronous caller did earlier. A sensibly built agent framework retries the call anyway. If retryRecord isn't idempotent, that retry is a duplicate-processing attempt for the exact same reason a queue redelivery causes one: the caller can't tell "no response" apart from "processed, response lost." The fix doesn't change either. The API needs the natural-key-plus-constraint treatment; telling the agent to "be careful" doesn't do anything, because an agent has no way to be careful about a race condition it can't see. Only the receiving system can close that gap.
Backpressure matters more here, not less, because nothing throttles the caller. A person clicking retry three times has friction built in — attention, patience, the time it takes to click something. An agent deciding to retry a failed step, or fanning a task out across a dozen tool calls because that looked like a reasonable plan, has none of that. It can throw call volume at a downstream API that no human workflow would ever generate, and if that API has no rate limiting or backpressure signal of its own (a 429 with a Retry-After, not just a bare timeout), the loop and the service can drive each other into a failure worse than anything a person would trigger by hand: the agent retries faster than the service recovers, the retries themselves prevent recovery, and it compounds from there. Same lesson as sizing the slowest stage of a synchronous chain — except now the "chain" gets decided at runtime by the agent instead of being fixed in code.
And the authorization scope needs to travel with the tool call the way a partition key travels with an event. An agent's tool-calling loop is, underneath, a sequence of delegated actions, which is the same actor/scope pattern from the identity piece I wrote earlier. Each call in the loop should carry its own narrow, task-scoped credential rather than a blanket token good for anything the loop might eventually decide to do. That's not a separate concern from the queueing discussion above, it's the same "don't let one component's misbehavior have unbounded blast radius" idea, just applied to permissions instead of retries.
What's actually unsolved here
Ordering across a multi-step agent plan doesn't have a clean partition-key answer, because the "correct order" isn't fixed ahead of time. It's whatever the agent decides while it's running, and two runs of the same task can legitimately take different paths through it. You can constrain a specific workflow (step C can't fire until step B returns), but there's no general mechanism yet that gives you Kafka-style partition ordering for a sequence an autonomous agent invented on the fly.
Compensating actions are the other rough edge. A failed event-driven pipeline usually has a defined recovery path: a dead-letter queue, a saga with explicit compensating transactions, a human reviewing a failed-message backlog. An agent that finishes 3 of 5 planned tool calls before something breaks doesn't get any of that automatically. Rolling back those three steps requires each one to expose a defined, idempotent undo, and almost nothing does today. It's the same partial-completion problem the identity piece raised about mid-task revocation, seen from a different angle: stopping the agent is easy, figuring out what it left behind isn't.
Testing this
The test that actually matters for an idempotent tool-call handler is the same shape as the database-constraint test from the idempotent-APIs piece, just adapted to a queue: publish the same event twice, actually twice, not "call the handler function twice in a unit test," and check that exactly one side effect happened and both deliveries got a consistent result back. For the agent-loop side, write a forced-timeout test: let a tool call actually complete server-side, then force a timeout on that first attempt so the agent's retry logic fires, and check the downstream state shows one record retried, not two. That test only catches real bugs if the timeout gets injected after real completion, not before — a mock that fails outright never exercises the ambiguous case that's actually dangerous.
Takeaways
- A synchronous call chain fails on tail latency, not average latency. It inherits its worst-performing hop's p99, and that compounds with every hop after it.
- Adding a queue buys decoupling but charges you at-least-once delivery. An idempotent consumer is what makes that trade worth it; without one, you've just relocated the duplicate-processing bug somewhere harder to see.
- The natural-key-plus-uniqueness-constraint approach to idempotency works the same for async consumers as for synchronous APIs — the underlying problem doesn't care about the transport.
- Partition by whatever needs ordering relative to itself, not by something arbitrary. Global ordering and full parallelism pull against each other, and most systems only need ordering scoped to something narrower than "everything."
- An agent's tool-calling loop is an event-driven consumer loop with no broker enforcing discipline on it. Its retries are redelivery, its fan-out is a backpressure risk with none of the friction a human adds, and its tool calls need the same idempotency and scoped-credential handling as any other retryable, delegated action.
- Ordering and compensation for autonomous multi-step plans are still open. There's no general equivalent yet of partition-key ordering or saga-style rollback for a sequence the agent invents at runtime instead of one that's specified in advance.
- Test the ambiguous-timeout case, not just the clean failure. Inject the timeout after real completion, not before, or the test won't touch the scenario that actually produces duplicate side effects.
Top comments (0)