DEV Community: Sameer Khare

The Three Events That Turn Your Logs Into an SLO Dashboard

Sameer Khare — Mon, 21 Sep 2026 19:44:53 +0000

A few years into running a JVM backend in production, I stopped trusting dashboards just because they were green. The numbers weren't wrong, exactly — CPU fine, heap fine, request count nominal — but "green dashboard, angry customer" kept happening often enough that I started keeping a private list of incidents where every chart we had swore everything was fine. It was always the same story: we were watching the machine, not the request. A service can sit comfortably under every resource limit it has and still be failing the one thing anyone actually cares about, which is whether this specific customer's call came back correctly, and on time.

That gap between "the infrastructure looks healthy" and "the system is doing its job" is where most observability budget quietly goes to waste. And closing it doesn't take a new tool. It takes logging the right three events per request, and pointing a dashboard at them the right way.

What actually pays off

Structured logging with a correlation ID that survives the whole request. I don't mean "we use JSON logs" — plenty of teams do that and still can't answer "show me every log line for this one request" without grepping across five services and guessing at a timestamp window. What actually works is assigning a correlation ID right at the edge, dropping it into the logging context (MDC, if you're on the JVM) before anything else runs, and carrying it through every outbound call — HTTP header, message header, whatever the transport happens to be. Three services downstream, it's still there.

@Component
public class CorrelationIdFilter extends OncePerRequestFilter {
    private static final String HEADER = "X-Correlation-Id";

    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
                                     FilterChain chain) throws IOException, ServletException {
        String correlationId = Optional.ofNullable(req.getHeader(HEADER))
            .orElseGet(() -> UUID.randomUUID().toString());
        MDC.put("correlationId", correlationId);
        res.setHeader(HEADER, correlationId);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.clear(); // thread pools reuse threads; an uncleared MDC leaks into the next request
        }
    }
}

That finally block is the part people miss, and it's the part that actually matters. Application servers reuse threads across requests, and MDC is thread-local. Forget the clear and, sooner or later, you'll get log lines tagged with the wrong request's correlation ID under load. It's invisible in dev, shows up only under concurrency, and quietly lies to whoever's trying to debug with it.

Distributed tracing with spans that mean something to a human, not whatever auto-instrumentation handed you. Default OpenTelemetry auto-instrumentation gives you a span per HTTP call and per JDBC statement, which is a fine start and also not remotely what you need at 2am. What pays off is adding your own spans around the things a person would actually say out loud — "validate the batch," "apply pricing rules," "notify downstream" — tagged with whatever an on-call engineer is going to ask next (account ID, batch size, which rule set fired), not just a duration. Five well-chosen custom spans beat forty auto-generated ones nobody has time to read mid-incident.

SLOs defined on what the caller experiences, not on what the machine is doing. Success rate and latency for the actual customer-facing operation are what should page someone. CPU, heap, and thread-pool size are useful once you're already digging in, but they're a bad primary signal, because a system can blow through every user-facing SLO it has while its resource metrics stay comfortably in range — a downstream dependency degrading, a lock contention pattern, a query plan that quietly flipped after an ANALYZE. I've seen the opposite just as often, too: heap sawtoothing right up against its limit for weeks, zero customer impact, because the GC was simply doing its job.

What's cargo-culted

The flip side of all three shows up constantly. Dashboards with thirty-plus panels because a tool made it easy to add another one, not because anyone actually looks at it during an incident — you can usually spot these by asking someone what a given panel is for and watching them struggle. Alert thresholds set on infrastructure metrics because they're the easiest numbers to alert on, not because they mean anything to a customer — "CPU over 80%" paging someone at 3am for a service that's still hitting every latency target it has. And tracing bolted on because "we should probably have tracing," with nobody deciding up front what question it's supposed to answer, which is how you end up with gorgeous flame graphs that nobody opens because they don't answer whatever's actually on fire.

The cost of cargo-culted observability isn't just wasted setup time, either. It's arguably worse than having nothing, because it hands people false confidence — a green dashboard everyone's learned to trust, watching the wrong thing the whole time.

From logs to an SLO dashboard

The correlation ID buys you more than debuggability, though. It's also the join key that turns raw logs into an SLO dashboard — and doing that well really comes down to logging three events per request instead of one.

Most services log a single line per request, at the end, with an outcome and a duration. That's enough to reconstruct what happened. It's not enough to reconstruct what's currently in flight, which matters the moment you want to know whether a spike in flight requests is about to become a spike in failures. So log three distinct events instead, all carrying the same correlation ID, the API or endpoint name, and a timestamp: initiated when the request lands, then exactly one of succeeded or failed when it wraps up, with the response time attached to whichever one fires.

log.info("event=initiated api={} correlation_id={}", apiName, correlationId);

// ... on completion ...
log.info("event=succeeded api={} correlation_id={} response_time_ms={}",
    apiName, correlationId, responseTimeMs);

log.info("event=failed api={} correlation_id={} response_time_ms={} reason={}",
    apiName, correlationId, responseTimeMs, failureReason);

Three events instead of one sounds like extra logging for no good reason, right up until you write the query that consumes it. In Splunk, that event shape is exactly what stats and eval want to chew on, per API, over whatever window you pick:

index=app_logs sourcetype=api_events
| stats
    count(eval(event="initiated")) as initiated
    count(eval(event="succeeded")) as succeeded
    count(eval(event="failed")) as failed
    perc95(response_time_ms) as p95_response_ms
    perc99(response_time_ms) as p99_response_ms
    by api
| eval success_rate = round((succeeded / initiated) * 100, 2)
| eval slo_met = if(success_rate >= 99.5 AND p95_response_ms <= 300, "Met", "Breached")

One query, and you've got everything an SLO conversation actually needs, per API: a success rate pulled from real traffic instead of a sample, a latency percentile instead of an average that hides the tail, and a straight pass/fail against whatever number you've committed to. Wire a single-value panel to it with a color threshold — green on "Met," red on "Breached" — and now the dashboard just answers "are we meeting our SLO," instead of handing someone a wall of infra charts and making them do the math in their head. Trend the same query over a rolling window (daily, weekly, whatever your compliance period is) rather than a live snapshot, because a five-minute blip and a day-long breach need very different responses, and a snapshot can't tell you which one you're staring at.

Splunk actually supports this at two levels, and it's worth knowing both exist. Everything above you can build by hand in SPL against ordinary indexed logs — no special license, no separate product, just events shaped the right way. But Splunk also ships a native SLO layer built for exactly this: Splunk Observability Cloud lets you define an SLO directly as a Service Level Indicator — "proportion of requests that succeeded," say, or "proportion of requests under a latency threshold" — plus a target percentage and a compliance period, and it tracks an error budget and burn rate against that automatically, with burn-rate alerts that fire based on how fast you're eating the budget rather than one static threshold crossing once. Splunk ITSI does something similar on the core platform, rolling per-API KPIs like these into a broader service health score. Which one you reach for is a real decision — hand-rolled SPL is faster to stand up and easier to change on a whim, the native SLO objects earn their keep once you've got enough services that doing error-budget math by hand stops scaling — but either way, the capability itself is real: yes, you can define an SLO per API and see exactly how you're doing against it, not just stare at a pile of unrelated metrics and guess.

One catch worth being upfront about: this only works if api is granular enough to actually mean something. A dashboard built on one lumped success rate and one lumped p95 across every endpoint in a service can pass its SLO while its single worst, most-hammered endpoint is quietly breaching its own the entire time. That by api in the query above isn't there for style — drop it, and you've built an SLO dashboard that's technically correct and practically useless.

Verifying this actually works

The check that actually catches gaps here isn't "did the dashboard render." Kill a downstream call mid-flight on purpose, then confirm the failed event actually lands with a real response_time_ms and a populated reason — not a silently dropped event, and not one that only ever fires on the happy path because that's the only path anyone tested. Then run a load test that forces thread reuse under concurrency and grep the logs afterward for correlation IDs attached to the wrong request. Neither bug shows up in a clean-path test, and both are exactly the kind of thing that makes an SLO dashboard confidently report the wrong number during the one incident where it actually matters.

Takeaways

A green infrastructure dashboard and a failing customer request can coexist just fine. Alert on what the caller experiences — success rate and latency for the actual operation — and treat CPU, heap, and thread-pool metrics as diagnostic, not primary.
Structured logging only pays off with a correlation ID that survives the whole request across every service, and thread reuse means an uncleared MDC will eventually mislabel someone else's request.
Custom spans around units of work a human would actually name beat forty auto-generated ones nobody has time to read mid-incident.
Logging initiated, succeeded, and failed as three separate events, each carrying the correlation ID and response time, gives you exactly the fields an SLO query needs — success rate and latency percentile, per API, over any window you want.
Splunk supports SLO tracking at two levels: hand-built SPL over ordinary logs when you want something fast and cheap, and native SLO objects with error budgets and burn-rate alerts (Observability Cloud) or KPI-based service health scoring (ITSI) once you've got enough services that doing the math by hand stops making sense.
An SLO measured at too coarse a grain — one number across every endpoint — can pass while your worst endpoint is breaching on its own. Break it out by API, or you're really just measuring an average and calling it an SLO.
Test the failure path, not the clean one. Kill a call mid-flight and check that the failure event and correlation ID attribution both survive it — that's the scenario where a broken observability setup actually lies to you.

This one stayed scoped to plain request/response services on purpose. What changes once part of the system is an LLM making its own sequence of tool calls — token usage as an SLO input, latency for a call whose duration you don't control, tracing a call graph the agent invents on the fly — is enough of a shift that it deserves its own piece rather than a rushed section bolted onto this one.

Your Agent's Retry Logic Is an Event-Driven Systems Problem

Sameer Khare — Fri, 18 Sep 2026 09:18:08 +0000

Most of the event-driven architecture I've built started the same way: a synchronous call chain that worked fine in a demo and fell over under real load. On a multi-tenant B2B SaaS backend I worked on, an ingestion service took in batches of records from a customer's connected system and processed them the obvious way: validate the batch, write it, call a processing service synchronously to derive whatever depended on it, then call a notification service to tell whoever cared. One request in, three services deep before anything went back to the caller.

It worked fine until ingestion volume grew past what the slowest service in that chain could handle. Everyone already knows the fix is "put a queue in the middle." What's less talked about is what you actually inherit once you do that: ordering questions, duplicate delivery, backpressure, poison messages. And those same problems show up again, almost unchanged, in how an AI agent runs a sequence of tool calls. That second part is the one people get wrong, because it doesn't look like a queueing problem at first glance.

Why the chain breaks, and it's never the average case

The naive pipeline calls processing and notifications inline, in the request path. It survives every load test built on average-case numbers, because averages are exactly what hide this failure. What actually kills it is tail latency compounding across hops: processing might run 200ms at p50 and 4 seconds at p99 (a slow query, a lock contention spike, a downstream dependency having a bad afternoon), and ingestion inherits that p99 whether it wants to or not. Every additional hop adds its own tail, and the whole chain ends up as slow as its worst-behaved member, not its average one.

A few things break specifically as volume climbs:

A slow downstream service backpressures everything ahead of it. If notifications is degraded, ingestion requests start timing out too, even though ingestion did nothing wrong. There's no shared database or shared deploy causing that; it's just one service holding a call open while it waits on another.

A failure partway through the chain is ambiguous. If the call to processing times out, did it finish the work and respond slowly, or never get the request at all? The caller has no way to know, so it can't safely decide whether to retry: retry a request that already landed and you risk double-processing, skip the retry on one that never landed and you silently lose work.

And scaling one stage tends to force you to provision for all of them together, because they're sized as one synchronous unit even when their real load patterns look nothing alike — ingestion spikes at particular hours, processing load doesn't necessarily follow it.

None of that is a bug in any single service. It's what synchronous coupling costs once the services on either end stop sharing the same performance and failure profile, which past a certain point they always do.

The fix, and what it actually costs you

The standard move: ingestion writes the batch, publishes an event ("record batch ingested for account X, batch Y") to a queue or topic, and returns immediately. Processing and notifications each pick that event up on their own schedule, and neither being slow or down blocks ingestion from taking the next batch. That decoupling is the part everyone talks about. The part that actually matters more is what you sign up for the moment a queue enters the picture: at-least-once delivery, not exactly-once.

Nearly every message queue in production — SQS, Kafka running typical at-least-once consumer semantics, RabbitMQ under normal config — will redeliver a message under some failure condition: a consumer crashing after processing but before acknowledging, a network blip, a redelivery timeout firing early. That's not a misconfigured queue. It's the honest tradeoff a distributed queue makes, because exactly-once delivery over a network is either impossible or expensive enough that it stops resembling a queue. Which means: if your consumer isn't idempotent, adding a queue didn't fix your duplicate-processing bug, it just moved it somewhere less visible.

This is where a pattern I've written about before — deriving a natural key straight from the request data instead of minting a client-side idempotency token — pays off again in a different context. A consumer reacting to "record batch ingested" needs the same protection any synchronous handler needs against a retried request: a deterministic key built from the event (account_id + batch_id), a uniqueness constraint the database enforces, and a path that catches the violation and returns the existing result instead of erroring out. The mechanism doesn't care whether the duplicate arrived as an HTTP retry or a queue redelivery — the underlying problem is identical either way.

@KafkaListener(topics = "record-batch-ingested")
public void onRecordBatchIngested(RecordBatchEvent event) {
    String idempotencyKey = event.accountId() + ":" + event.batchId();

    try {
        processedBatchRepository.insertProcessedMarker(idempotencyKey, event);
        recordProcessingService.process(event);
    } catch (DuplicateKeyException alreadyProcessed) {
        log.info("Duplicate delivery for {}, skipping processing", idempotencyKey);
        // Ack and move on. Expected, not an error.
    }
}

Ordering is the tradeoff without a tidy answer. A single Kafka partition (or an SQS FIFO queue) gives ordering within that partition, at the cost of throughput — one consumer, one message at a time, per partition. Partition on something arbitrary and you get parallelism with no ordering guarantee across related events. Partition by account_id and you get ordering for everything that matters to that account specifically — usually the real requirement is "did this account's events land in the order they happened," not global ordering across the whole system — while still parallelizing across accounts. Get this wrong and it rarely shows up as an obvious bug. It shows up three weeks later as a config update applied before the data event it was supposed to modify, and nobody's quite sure why one account's numbers looked wrong for exactly one day.

Agent orchestration is the same problem, minus the broker

Here's the connection that made this worth writing up. An AI agent working through a multi-step task — "find the records that failed validation last month and retry them" — is structurally an event-driven system whether or not anyone building it thought of it that way. The agent decides to call a tool, the call goes out (often asynchronously: a queued job, a webhook callback, a long-running request), a result comes back, and that result feeds the next decision. Decide, dispatch, wait, decide again. That's a consumer loop. Every tradeoff from above shows up again here, usually with no broker enforcing any discipline around it.

Retries are redelivery, and need the same treatment. If an agent's call to a retryRecord API times out, the agent has no better way of knowing whether the retry landed than the synchronous caller did earlier. A sensibly built agent framework retries the call anyway. If retryRecord isn't idempotent, that retry is a duplicate-processing attempt for the exact same reason a queue redelivery causes one: the caller can't tell "no response" apart from "processed, response lost." The fix doesn't change either. The API needs the natural-key-plus-constraint treatment; telling the agent to "be careful" doesn't do anything, because an agent has no way to be careful about a race condition it can't see. Only the receiving system can close that gap.

Backpressure matters more here, not less, because nothing throttles the caller. A person clicking retry three times has friction built in — attention, patience, the time it takes to click something. An agent deciding to retry a failed step, or fanning a task out across a dozen tool calls because that looked like a reasonable plan, has none of that. It can throw call volume at a downstream API that no human workflow would ever generate, and if that API has no rate limiting or backpressure signal of its own (a 429 with a Retry-After, not just a bare timeout), the loop and the service can drive each other into a failure worse than anything a person would trigger by hand: the agent retries faster than the service recovers, the retries themselves prevent recovery, and it compounds from there. Same lesson as sizing the slowest stage of a synchronous chain — except now the "chain" gets decided at runtime by the agent instead of being fixed in code.

And the authorization scope needs to travel with the tool call the way a partition key travels with an event. An agent's tool-calling loop is, underneath, a sequence of delegated actions, which is the same actor/scope pattern from the identity piece I wrote earlier. Each call in the loop should carry its own narrow, task-scoped credential rather than a blanket token good for anything the loop might eventually decide to do. That's not a separate concern from the queueing discussion above, it's the same "don't let one component's misbehavior have unbounded blast radius" idea, just applied to permissions instead of retries.

What's actually unsolved here

Ordering across a multi-step agent plan doesn't have a clean partition-key answer, because the "correct order" isn't fixed ahead of time. It's whatever the agent decides while it's running, and two runs of the same task can legitimately take different paths through it. You can constrain a specific workflow (step C can't fire until step B returns), but there's no general mechanism yet that gives you Kafka-style partition ordering for a sequence an autonomous agent invented on the fly.

Compensating actions are the other rough edge. A failed event-driven pipeline usually has a defined recovery path: a dead-letter queue, a saga with explicit compensating transactions, a human reviewing a failed-message backlog. An agent that finishes 3 of 5 planned tool calls before something breaks doesn't get any of that automatically. Rolling back those three steps requires each one to expose a defined, idempotent undo, and almost nothing does today. It's the same partial-completion problem the identity piece raised about mid-task revocation, seen from a different angle: stopping the agent is easy, figuring out what it left behind isn't.

Testing this

The test that actually matters for an idempotent tool-call handler is the same shape as the database-constraint test from the idempotent-APIs piece, just adapted to a queue: publish the same event twice, actually twice, not "call the handler function twice in a unit test," and check that exactly one side effect happened and both deliveries got a consistent result back. For the agent-loop side, write a forced-timeout test: let a tool call actually complete server-side, then force a timeout on that first attempt so the agent's retry logic fires, and check the downstream state shows one record retried, not two. That test only catches real bugs if the timeout gets injected after real completion, not before — a mock that fails outright never exercises the ambiguous case that's actually dangerous.

Takeaways

A synchronous call chain fails on tail latency, not average latency. It inherits its worst-performing hop's p99, and that compounds with every hop after it.
Adding a queue buys decoupling but charges you at-least-once delivery. An idempotent consumer is what makes that trade worth it; without one, you've just relocated the duplicate-processing bug somewhere harder to see.
The natural-key-plus-uniqueness-constraint approach to idempotency works the same for async consumers as for synchronous APIs — the underlying problem doesn't care about the transport.
Partition by whatever needs ordering relative to itself, not by something arbitrary. Global ordering and full parallelism pull against each other, and most systems only need ordering scoped to something narrower than "everything."
An agent's tool-calling loop is an event-driven consumer loop with no broker enforcing discipline on it. Its retries are redelivery, its fan-out is a backpressure risk with none of the friction a human adds, and its tool calls need the same idempotency and scoped-credential handling as any other retryable, delegated action.
Ordering and compensation for autonomous multi-step plans are still open. There's no general equivalent yet of partition-key ordering or saga-style rollback for a sequence the agent invents at runtime instead of one that's specified in advance.
Test the ambiguous-timeout case, not just the clean failure. Inject the timeout after real completion, not before, or the test won't touch the scenario that actually produces duplicate side effects.

AI Agents Need Their Own Authorization, Not Yours

Sameer Khare — Thu, 17 Sep 2026 13:54:28 +0000

Every multi-tenant B2B SaaS platform I've worked on eventually converges on the same authorization shape: every request carries a tenant, and within that tenant, a user with some set of permissions. Two axes — which account, which person — and for years that was a complete enough model to build an entire authorization layer on top of.

It stopped being complete the day an AI agent became a caller. Not a human sitting at a browser making requests that happen to be backed by a session token, but a process that decides on its own which API calls to make, sometimes across a dozen steps, with no human confirming each one — acting for a user without being that user's browser session. The two-axis model has no slot for that. It has tenant, and it has user. It doesn't have "something else, calling on the user's behalf, that isn't the user."

This is the auth model I've watched evolve over several iterations of a multi-tenant platform, and it's also, I think, the most interesting authorization problem in software right now — because it's not hypothetical anymore. It's what Auth0, Okta, and every other identity vendor spent a chunk of their 2026 roadmaps on. Worth walking through why the old model breaks and what actually replaces it.

The two-axis model, and where it breaks two years in

The standard shape is familiar to anyone who's built one: a JWT (or an opaque session token resolved server-side) carrying a tenant_id and a user_id, with roles or permissions attached to the user within that tenant.

{
  "tenant_id": "acct_9f21",
  "user_id": "usr_4471",
  "roles": ["billing_admin", "reports_viewer"]
}

A gateway or middleware layer validates the token, resolves the claims, and every downstream service trusts tenant_id and user_id as the scoping boundary for every query it runs. It's a good model, and it holds up for a surprisingly long time. The places it actually breaks, in my experience, aren't exotic:

tenant_id drifts from "trusted claim" to "trusted-ish parameter." Somewhere around year two, a service that was only ever called internally — a reporting job, an admin tool, a migration script — starts accepting tenant_id as a request parameter instead of deriving it from the token, because it was faster to build that way and nobody was going to call it cross-tenant on purpose. Then something does, accidentally, and you have a cross-tenant data leak that traces back to one endpoint nobody thought of as "the auth boundary."
Roles get coarser, not finer, over time. "Admin" starts out meaning something specific and slowly absorbs every permission nobody wanted to model separately, because adding a new fine-grained role is a migration and a support ticket, and checking role == "admin" is one line. Two years in, half your user base has a role that grants far more than what they actually use.
The permission check gets reimplemented per service. Each team building a new microservice writes its own version of "does this user have permission X in this tenant," slightly differently, against a slightly different snapshot of what the roles mean. Nobody notices until an audit asks you to prove that permission logic is consistent across twelve services, and it isn't.

None of these are AI-related. They're just what happens when a two-axis model meets real organizational entropy. But they matter here because the AI-agent problem makes all three worse if you don't fix them first — an agent calling through a loosely-guarded tenant_id parameter, or inheriting a coarse "admin" role, is the same bug with a much larger blast radius, because the agent can trigger it a hundred times a minute without a human noticing the pattern.

What changes when the caller isn't a person

Picture a support-automation agent: a user asks it, in natural language, to "find the three invoices that bounced last month and retry them." The agent doesn't get a pre-approved plan — it decides, at runtime, to call a search API, then a retry-payment API, possibly a few times if the first attempt fails. Nobody clicked "retry" three separate times. One instruction turned into an unknown number of API calls, decided by the agent, not the user.

If you hand that agent the user's own session token, you've quietly made a decision you probably didn't mean to make: the agent now has everything the user has, for as long as that token is valid, with no way to tell, after the fact, "the user did this" from "the agent decided to do this." Three consequences follow directly from that, and none of them are edge cases:

Blast radius stops matching task scope. The task was "retry three invoices." The credential is "everything a billing admin can do in this tenant." If the agent is compromised, or manipulated by something in the data it's reading (a classic prompt-injection path — a malicious string in an invoice description, say, instructing the agent to do something else), the ceiling on what it can do is the user's entire permission set, not the three-API-call task it was asked to do.

Audit trails collapse into one identity. When something goes wrong six weeks later and someone asks "did the user do this, or did the agent," a token that only ever says user_id: usr_4471 can't answer that question. That distinction isn't a nice-to-have in a billing system — it's the difference between a user dispute you can resolve in five minutes and one you can't resolve at all.

Lifecycle and revocation stop lining up. A human session token is scoped to a login session — log out, and it's gone. An agent's task might run for seconds or, in an async workflow, for minutes across a queue. You want a way to kill the agent's ability to act without logging the user out of everything else, and a shared token gives you no lever to pull.

Worth being explicit about scope here: none of these three consequences are actually caused by multi-tenancy. A single-tenant application — no tenant_id anywhere, just user_id and roles — has the identical failure mode the moment an agent starts calling its APIs on a user's behalf. Same blast-radius mismatch, same collapsed audit trail, same revocation problem. Multi-tenancy doesn't create this; it stacks one more, independent risk on top of it — an over-scoped or compromised agent's overreach can potentially cross tenant boundaries too, not just exceed the task within one account. Everything below applies to both; multi-tenant SaaS is just the setting I've actually had to build it in.

A third identity: the actor, not just the subject

The fix isn't a new invention — it's an old OAuth pattern (RFC 8693, Token Exchange) that suddenly got a lot more relevant: the act claim, which represents delegation rather than impersonation. The distinction the RFC draws is exactly the one that matters here. Impersonation means principal A becomes indistinguishable from B — A gets B's rights and B's identity. Delegation means A keeps its own identity while acting for B — the token says, explicitly, "this is the agent, acting on behalf of this user, with this narrower scope."

Extending the two-axis claims to carry that looks like this:

{
  "tenant_id": "acct_9f21",
  "user_id": "usr_4471",
  "roles": ["billing_admin", "reports_viewer"],
  "act": {
    "agent_id": "agt_invoice_retry_v3",
    "on_behalf_of": "usr_4471",
    "scope": ["invoices:read", "payments:retry"],
    "task_id": "task_88c1",
    "exp": 1758470400
  }
}

The critical property isn't that the claim exists — it's what the authorization check does with it. A naive implementation checks "does user_id have this permission" and ignores act entirely, which defeats the entire point. The check that actually contains the blast radius is an intersection: the caller is authorized only if the action is within the user's permissions and within the actor's granted scope.

boolean isAuthorized(Token token, String requiredPermission) {
    boolean userHasPermission = permissionService
        .hasPermission(token.tenantId(), token.userId(), requiredPermission);

    if (token.actor() == null) {
        return userHasPermission; // ordinary human session, unchanged
    }

    boolean actorInScope = token.actor().scope().contains(requiredPermission);
    boolean taskNotExpired = Instant.now().isBefore(token.actor().exp());

    // Delegated calls need BOTH: the user could do this, AND
    // the agent was explicitly scoped to do this specific thing.
    return userHasPermission && actorInScope && taskNotExpired;
}

An agent scoped to ["invoices:read", "payments:retry"] can retry three bounced invoices all day long. It cannot call payments:refund or users:delete, even though the human it's acting for technically could — because the actor's scope, not the user's full role, is now the ceiling for anything the agent does. That's the actual fix: not "trust the agent less," but "give the agent its own, narrower credential instead of quietly reusing the human's."

This isn't a hypothetical extension — it's where the vendors are already building

This pattern isn't me speculating about where auth architecture might go. It's what Auth0 shipped in 2026 under exactly this framing: treating an AI agent as a first-class principal, distinct from the user, with its own identity and audit trail rather than being folded into the user's. Their on-behalf-of token exchange does the delegation handshake directly — a server trades a user's access token for a correctly-scoped downstream token instead of forwarding the original credential everywhere. Their Token Vault isolates the third-party credentials an agent needs (a calendar API key, a payments provider token) per tenant, so a bug in one organization's agent configuration can't reach across tenant boundaries. Okta's parallel work — Agent SSO, and fine-grained authorization built for permission checks at agent-call volume rather than human-click volume — is solving the same problem from the identity-provider side.

The fact that two competing identity vendors converged on "agent needs its own scoped, delegated, revocable credential, not the user's" in the same year is a good signal this is the actual shape of the problem, not one implementation's opinion about it.

What's still genuinely unsolved

The honest gap, and it's a real one: least-privilege scoping assumes you know the task in advance. "Retry three invoices" scopes cleanly to invoices:read and payments:retry. But a lot of what makes agents useful is that they don't follow a pre-approved plan — they decide their next step based on what the last one returned. An agent that starts with "summarize this account's activity" might reasonably decide, three steps in, that it needs a permission nobody scoped it for, and there's no clean answer yet for "should the agent get to ask for broader scope mid-task, and if so, who approves that without breaking the whole point of an unattended agent." Static, task-scoped tokens are a solid answer for well-defined tasks; they don't fully solve open-ended agent loops, and I haven't seen an identity vendor claim otherwise yet.

Revocation mid-task has a similar rough edge. Killing an agent's token after it's made 3 of an intended 10 calls is easy to do — revoke the token, next call fails — but reasoning about what state that leaves behind (three payments retried, seven not, and no transaction wrapping the whole task) is an application-level problem the auth layer doesn't solve for you. It just makes sure the agent can't keep going once you've decided to stop it.

Testing this

The check worth writing isn't "does the middleware read the act claim correctly" — that's a unit test and it's necessary but not sufficient. The one that actually catches the bug that matters is adversarial: mint a token where the user's role would permit an action, but the actor's scope doesn't, and assert the request is rejected. Then do the reverse — an actor scope that's broader than what makes sense, paired with a user who's been demoted or had a role removed — and confirm the user-side check still wins. The bug this pattern is actually guarding against is someone shipping a permission check that reads one claim and forgets the other; a test suite that only exercises the happy path where both claims agree won't catch it.

Takeaways

An AI agent acting on a user's behalf is a third identity — the actor — and needs to be represented as one, not folded into the user's token. This isn't a multi-tenant-specific problem; it shows up identically in a single-tenant app with just a user_id. Multi-tenancy just adds a second, independent risk on top (cross-tenant overreach).
Reusing the user's session token for an agent silently maximizes blast radius to "everything the user can do," for the lifetime of that token, with no way to distinguish the agent's actions from the user's in an audit trail.
Delegation isn't impersonation. RFC 8693's act/may_act claims formalize the distinction: the agent keeps its own identity and a narrower, explicit scope, instead of becoming indistinguishable from the user it's acting for.
The authorization check has to be an intersection, not a swap. A delegated call needs to pass both the user's permission check and the actor's scope check — checking only one defeats the purpose.
This is where identity vendors are actually building right now, not a speculative extension — Auth0's on-behalf-of token exchange and Token Vault, Okta's Agent SSO, all converge on the same "agent as first-class, scoped, revocable principal" shape.
Static task scoping doesn't fully solve open-ended agent loops, and mid-task revocation stops the agent but doesn't clean up partial state on its own — both are open problems worth being honest about, not glossed over.
Test the claim intersection adversarially, not just the happy path — a user-permits/actor-denies case and an actor-permits/user-denies case, not just the case where both claims happen to agree.

Idempotent APIs Without Idempotency Tokens

Sameer Khare — Thu, 17 Sep 2026 07:00:28 +0000

A few years into working on a utility billing SaaS platform, I ended up owning a feature that sounds simple on a slide and is genuinely nasty underneath: "what would my bill look like on a different rate plan?"

Utilities offer customers a handful of rate plans — flat rate, time-of-use, tiered, demand-based, seasonal variants — and one of the most-requested features from account holders (and from the utilities themselves, who want to nudge customers toward plans that reduce peak load) is a side-by-side comparison: take my actual usage, keep my current plan as the baseline, and show me what I'd have paid on plans B, C, and D instead.

That's not a lookup. It's re-running the full bill calculation engine, once per candidate rate plan, against a customer's interval usage data for the billing period — a genuinely expensive, synchronous REST call. And because it's expensive and synchronous, it's also a good case study in a design decision that doesn't get talked about much: when your idempotency key is already sitting in your domain model, you don't need a client-generated token to get idempotency right.

Why this endpoint had to be idempotent

The rate-comparison endpoint was a plain REST API: client calls in, server computes N bill calculations — one per rate plan being compared, with the customer's current plan as the baseline — server returns a comparison result.

Because each calculation walks a customer's interval-usage records for the full billing period and applies a rate plan's pricing rules to them, a single request could take long enough that clients — mobile apps, web dashboards, other backend services — would occasionally time out and do exactly what well-behaved clients are supposed to do: retry.

Without protection, a naive implementation would kick off a second full multi-plan comparison run for a request that was probably still in flight server-side, potentially write two comparison result records for the same logical request, and in the worst case show a customer two different numbers for "what you'd pay on Plan B" if the retry landed slightly differently (say, a rate table got updated between the two runs). For a feature whose entire job is to be trusted enough that a customer switches rate plans based on it, that's not a cosmetic bug — it's the kind of thing that erodes trust in the number on the screen.

The usual answer, and why we skipped it

The textbook approach here is an idempotency-key header: the client generates a UUID before making the request, sends it on every attempt including retries, and the server keeps a store mapping keys to results — first request executes and caches its result under that key, every subsequent request with the same key gets the cached result back without re-executing anything.

It's a solid, general-purpose pattern, and it's the right call when there's no natural way to identify "this is logically the same request" from the request's own content — think payment submissions, where two purchases of the same product on the same day by the same user are legitimately different events, not duplicates.

Our situation was different. A rate comparison request is fully described by three things: which account, which billing period, which set of rate plans are being compared. Two requests with the same three values aren't just similar — they're asking the exact same question. That's a natural key sitting right there in the domain model, which meant we didn't need to invent a token, hand it to the client, and build a store to track it. We could let the database enforce uniqueness on data that already had to exist.

The design: a unique constraint on a derived natural key

We derived a deterministic key from the request itself:

// Simplified illustration of the pattern
String comparisonKey = accountId + ":" + billingPeriodId + ":"
    + sortedRatePlanIds.stream()
        .map(String::valueOf)
        .collect(Collectors.joining(","));

Sorting the rate plan IDs before joining them matters more than it looks — without it, comparing [planA, planB] and [planB, planA] would produce two different keys for what's semantically the same comparison, and you'd have silently reintroduced the duplicate problem you were trying to solve.

The request handler runs the multi-plan calculation, then attempts to insert the result row against a uniqueness guarantee on that key. On success, that's a new comparison, and it's returned. On a violation, the handler catches the persistence exception, fetches the existing row by the same natural key, and returns that instead.

That last part is the detail that makes this actually safe under concurrency, not just under sequential retries. Two identical requests arriving milliseconds apart — a genuine race, not a retry-after-timeout — will both attempt the insert; the database guarantees exactly one of them succeeds and the other gets the violation. That's a correctness guarantee an application-level "check if it exists, then insert if not" pattern can't give you on its own, because there's a window between the check and the insert where a second request can slip through. Letting the database's own uniqueness enforcement be the source of truth removes that window entirely, instead of trying to close it with an extra layer of locking.

The edge case a natural key can't distinguish on its own

Here's the trade-off worth being honest about: this approach only works because the natural key happens to be meaningful and stable. It breaks down the moment there's a legitimate reason to recompute the same (account, billing_period, rate_plan_set) tuple — for example, a customer's usage data gets corrected after a meter re-read, and the existing comparison is now built on stale numbers.

A pure natural-key constraint can't tell "this is an unwanted duplicate" apart from "this is the same identifiers, but the underlying data actually changed and a new answer is needed." Both look identical from the key's perspective, so the schema needs to make room for a deliberate recompute as a distinct, first-class event rather than trying to make the key itself smart enough to infer intent it can't see.

The approach we used: add a superseded_at timestamp to the row, and instead of a plain unique constraint on the natural key, scope uniqueness to only the current (non-superseded) row for that key:

CREATE UNIQUE INDEX uq_rate_comparison_current
  ON rate_comparison_run (account_id, billing_period_id, rate_plan_set_key)
  WHERE superseded_at IS NULL;

A recompute becomes a transaction: mark the existing current row superseded_at = now() — which drops it out of the index — then insert the new row as the current one. Old rows stay in the table as history rather than being overwritten, which is genuinely useful in a billing context: "what number did we actually show this customer, and when" is a question support and compliance will ask eventually.

A first pass at this might reach for putting a version number directly into a plain multi-column unique constraint instead of using a partial index. That doesn't work cleanly: a flat UNIQUE (account_id, billing_period_id, rate_plan_set_key, version) means every plain request — not just recomputes — now needs an extra read to determine "what's the current version" before it can even attempt the insert, since a normal retry has no version of its own to supply. Scoping the index to current rows avoids that: ordinary requests attempt the insert directly and fall into the same catch-violation-and-fetch path already in place, with no extra lookup on the common path.

One portability caveat worth knowing before reaching for this: the WHERE clause on CREATE INDEX is Postgres/SQLite/SQL Server syntax (SQL Server calls the equivalent a "filtered index"). MySQL and Oracle don't support a WHERE clause on index creation at all. Both have workarounds — MySQL via a generated column that evaluates to NULL for superseded rows and a real value for current ones (leaning on the same "NULLs don't collide in a unique index" behavior discussed above, deliberately this time), Oracle via a function-based index doing roughly the same thing — but the mechanism differs enough by engine that it's worth checking rather than assuming the syntax above just ports.

What actually triggers a recompute is a separate, open question

Notice what this section hasn't answered: what decides that a comparison is stale and a recompute is needed? In our case the trigger was something upstream — a correction to previously-submitted interval data — but the honest answer is that "what marks something as needing recompute, and how that signal reaches this endpoint" is its own design problem, and not a small one. It could be an event from whatever system corrects the underlying data, a scheduled reconciliation job, a manual trigger from support tooling, or something else entirely, and the right choice depends heavily on how often the underlying data actually changes, how expensive a recompute is, and who needs to know the result is stale and when. That's genuinely a per-product decision rather than something this pattern hands you for free — the natural-key-plus-supersede design gives you a clean place to record a recompute once you've decided one is needed; it doesn't tell you when to decide that.

Testing it

The way we actually validated the core idempotency guarantee wasn't a unit test against the constraint definition — it was firing concurrent duplicate requests at a real (test) instance and asserting exactly one row landed in the table and both callers got the same result back. A unit test can confirm the SQL is syntactically what you meant; it can't tell you whether your application code handles the violation path correctly under real concurrency, which is the part that's actually easy to get wrong (catching the wrong exception type, catching it in the wrong place, or forgetting the "then fetch and return the existing row" half entirely and just surfacing a 500).

Takeaways

Look for a natural key before reaching for an idempotency-key header. If a request is fully identified by data that already has to be in the request — account, period, resource set — a database-enforced uniqueness guarantee on that data is simpler than a client-generated token and a separate store to track it.
Canonicalize before deriving the key. Any set-like or order-independent part of the request (a list of IDs, in our case) needs to be normalized — sorted, deduplicated — before it goes into the key, or logically identical requests will produce different keys.
Let the database close the race, don't try to close it yourself. An app-level check-then-insert has a window a concurrent duplicate can slip through. Database-enforced uniqueness, combined with catching the violation and fetching the existing row, doesn't.
A natural key can't distinguish "duplicate" from "same identifiers, genuinely new data." If that case is possible in your domain, give recomputation an explicit, distinct path (a supersede-and-reinsert pattern, in our case) instead of trying to make the key smart enough to infer intent it can't see.
Scoping uniqueness to "current" rows isn't universal SQL — Postgres, SQLite, and SQL Server support it natively; MySQL and Oracle need a different mechanism to get the same effect. Check before assuming a snippet ports across engines.
Deciding when something needs recomputing is a separate problem from how to record a recomputation. This pattern solves the second; the first depends on your system's specific data-freshness and cost tradeoffs and deserves its own design pass rather than an assumed answer.
Test the concurrent path, not just the constraint. The SQL is the easy part to get right; the application code's handling of the violation is where idempotency bugs actually hide.