DEV Community: Shola Jegede

How to Check a Customer's Plan on Every Request Without Slowing It Down

Shola Jegede — Mon, 09 Nov 2026 15:51:13 +0000

A customer downgrades their subscription at 10:00. Their access token was issued at 9:00, and by default, it remains valid for another 24 hours.

The token still proves who the customer is. What it doesn't tell you is whether their new plan allows them to use a particular feature.

This creates a problem for your API. Every request needs to answer a question: does this customer's current subscription include the feature they're trying to access?

The answer lives in your billing provider, not in the access token.

You could ask the provider on every request, but then every API call has to wait for a network request. Alternatively, you could trust the token, but that means a downgrade might not take effect until the token expires.

Neither approach is particularly attractive.

I wanted to explore a third option: a gateway that checks entitlements on every request, keeps the check fast, and still responds to plan changes or provider outages.

To test this, I built a small gateway called Stub. It runs on Bun with Hono and retrieves subscription entitlements from Kinde, which handles authentication and billing.

An entitlement is a feature that a subscription plan grants to a customer.

I tested Stub against a real Kinde account with three test customers. Unless I explicitly mention a stand-in provider, the measurements in this article come from live API calls.

I've written two earlier articles covering related problems. One looks at receiving billing events safely, while the other explores how to enforce usage limits for long-running jobs. This article focuses on the entitlement check itself: its latency, how long its answers can become outdated, and what happens when the billing provider stops responding.

The access token tells you who the customer is, not what their plan allows

Before building the gateway, I needed to understand exactly what information Kinde makes available in an access token.

The relevant claims include:

sub: The user.
org_code: The organization.
permissions
feature_flags

The billing information available in the documented token claims is more limited. Kinde documents two trial-related claims: has_trial_period and trial_expires_on.

I couldn't find a claim containing the customer's current plan or entitlement limits.

The token's lifetime makes relying on it even more problematic.

By default, an access token remains valid for 24 hours. Kinde's documentation also explains that revoking a token, ending a session or deleting a user doesn't shorten the lifetime of a token that has already been issued.

So a customer can continue presenting a valid token long after the subscription information associated with it has changed.

Kinde provides two ways to retrieve entitlements outside the token.

The first is the Account API:

GET /account_api/v1/entitlements

This uses the signed-in user's access token.

The second is the Management API:

GET /api/v1/billing/entitlements

This requires a machine-to-machine token with the read:billing_entitlements scope. The response includes each entitlement's feature_code and entitlement_limit_max.

I chose the Management API because Stub operates as a backend gateway rather than a user-facing application.

Kinde also discusses enforcing entitlements on the backend, alongside the checks performed in the product interface, in its engineering blog.

This gives the gateway two separate responsibilities. It verifies the access token to establish the customer's identity, then retrieves their entitlements to determine what their subscription currently allows.

The remaining question is how to perform that second check without making every request unnecessarily expensive.

Verify the access token locally

I wanted the identity check to happen without an additional network request.

Kinde signs access tokens using RS256, which combines RSA signatures with SHA-256. It publishes the corresponding public keys at /.well-known/jwks on the Kinde domain.

Stub uses those public keys to verify the token signature, issuer and expiration.

Each Kinde environment has its own issuer value, so a token issued by another environment fails verification.

Hono provides built-in JWK middleware for this. I chose to use the jose library directly instead, keeping the verification logic inside a plain function.

That made testing easier. I could provide a local key set and verify tokens without making any network requests.

An invalid token receives a 401 response before the gateway attempts an entitlement lookup.

Once the token passes verification, Stub identifies the customer using the organization code when one is available, falling back to the user otherwise.

The cache key includes that identity, which prevents one organization's entitlement response from being reused for another.

With identity verification handled locally, I moved on to the actual billing lookup.

The entitlements endpoint needs a billing customer ID

I ran into an unexpected detail when connecting Stub to the Kinde Management API.

The access token contains an organization code, something like org_8c27….

However, the entitlements endpoint expects a billing customer ID, which looks like customer_01a1….

The API reference I was working from didn't make it clear which identifier the endpoint expected, so I tested both against my three organizations.

Using the organization code, all three requests returned HTTP 500. The response body contained only an error ID.

Using the billing customer ID, all three requests returned HTTP 200 with the expected entitlements.

The Free plan returned reports.

Pro returned reports and exports.

Business returned all three features.

I then tried retrieving the billing customer ID through Kinde's organization endpoint. All three organizations returned billing: null.

For this implementation, I created a small file mapping organization codes to their corresponding billing customer IDs.

There's a more automated approach available. A live plan-change event from Kinde includes both org_code and customer_id, so a webhook handler could maintain this mapping as events arrive. I didn't implement that part in Stub.

An organization missing from the mapping also needs special treatment.

Stub classifies it as unknown rather than treating it as a billing provider failure. Otherwise, a missing mapping for one customer could count towards the circuit breaker threshold and potentially prevent other customers from retrieving their entitlements.

One other detail about my test plans: all their features are unmetered. Kinde reports their entitlement limit as 2,147,483,647, the maximum signed 32-bit integer.

Consequently, Stub checks whether a feature exists in the customer's entitlements. It doesn't enforce usage counts.

What happens when you call Kinde on every request?

The simplest implementation would be to call Kinde's Management API whenever a request needs an entitlement check.

I started by measuring what that actually costs.

I made 30 consecutive entitlement lookups from my laptop in Lagos.

Measurement	Result
Median latency	477 ms
Fastest request	434 ms
Slowest request	857 ms

There was another cost worth accounting for: obtaining the machine-to-machine access token needed for the lookup.

The first token request took 4,598 ms.

Stub handles this separately. It retrieves the token at startup and refreshes it 30 seconds before expiration, so individual customer requests don't have to wait for token acquisition.

I then sent 40 requests through Stub, with four running concurrently and a fresh entitlement lookup for each request.

The gateway handled 6.0 requests per second, with a median request latency of 499 ms.

For comparison, I ran another test with the entitlement cache already populated. This time, I sent 2,000 requests with 50 running concurrently.

The gateway handled approximately 16,000 requests per second, with a median latency of 1.8 ms.

The two tests used different load patterns, so throughput isn't directly comparable. The median latency gives us a more useful comparison: moving from a live lookup to a warm cache reduced the median from 499 ms to 1.8 ms, roughly 280 times.

I also wanted to understand what happened when multiple requests needed Kinde simultaneously.

I sent 16 entitlement lookups at different concurrency levels: 1, 4, 8 and 16 concurrent requests.

As concurrency increased, the median lookup latency rose from 472 ms to 1,328 ms. The slowest request at eight concurrent lookups took 1,640 ms.

Kinde's API rate-limit documentation describes both a rate limiter and a concurrency limiter. Exceeding either can produce a 429 response, although the documentation doesn't publish the actual limits.

None of my requests received a 429, so I couldn't establish where those limits sit.

What I could establish was that a gateway making one billing request per application request inherits the billing provider's latency and concurrency constraints.

That makes the approach difficult to scale.

Two cache windows: fresh and stale

A cache isn't simply a performance optimisation. It's also a decision about how long the application is willing to use information that might have changed.

I wanted Stub to handle two different situations:

The cached entitlement is recent enough to trust.
The cached entitlement might be outdated, but retrieving a new one would slow down the current request.

Instead of using one expiration time, Stub has two windows.

An entitlement response younger than 30 seconds is considered fresh. The gateway returns it immediately.

Between 30 seconds and 5 minutes, the response is considered stale. Stub still returns the existing value immediately, but starts a background refresh at the same time.

The next request can then use the refreshed value.

Once the cached response is older than five minutes, or no response exists at all, the request has to wait for Kinde.

Here's the relevant part of the implementation:

async get(key: string) {
  const entry = this.entries.get(key);

  if (entry) {
    const age = ageOf(
      entry,
      this.now(),
      this.freshMs,
      this.staleOkMs
    );

    if (age === "fresh") {
      return {
        entitlements: entry.value,
        source: "fresh"
      };
    }

    if (age === "stale") {
      void this.load(key); // refresh behind the answer

      return {
        entitlements: entry.value,
        source: "stale"
      };
    }
  }

  const value = await this.load(key);

  return value === null
    ? { entitlements: null, source: "unknown" }
    : { entitlements: value, source: "fetched" };
}

The important behaviour is that a request with an existing answer doesn't have to wait for the background refresh.

That keeps the gateway responsive, including when the cached information is slightly outdated.

Of course, there's a trade-off. A customer who has just downgraded might still use their previous entitlements during that stale window.

I wanted to understand what happens when several requests encounter an empty cache at the same time, because that's another situation where a seemingly simple caching implementation can become expensive.

Prevent 200 requests from making 200 billing calls

Consider a customer who has never used the gateway before.

Their entitlements aren't in the cache. Now imagine 200 requests from that customer arrive simultaneously.

A basic cache implementation could send 200 separate requests to Kinde. Each incoming request checks the cache, finds nothing and starts its own lookup.

This is commonly known as a cache stampede.

Stub handles it by keeping track of entitlement lookups that are already in progress.

When a request needs an entitlement response, it first checks whether another request is already retrieving that customer's information. If so, it joins the existing lookup rather than starting another one.

I tested this with 200 simultaneous requests for a customer whose entitlements weren't cached.

Against live Kinde, all 200 requests completed between 0.83 and 0.87 seconds across two runs.

That is consistent with all 200 requests sharing the time required for one lookup.

Every response was marked as fetched.

The live test doesn't expose the number of API calls made, so I used a stand-in provider and a unit test to verify the actual call count.

With 200 simultaneous requests, the stand-in received:

Implementation	Provider lookups
Without request merging	200
With request merging	1

I didn't run the unmerged implementation against live Kinde.

This behaviour is particularly useful after deployments. If a restart empties the cache and a large number of customers begin making requests, the gateway doesn't immediately turn that traffic spike into an equally large spike of billing API calls.

Use Kinde webhooks to invalidate the cache when a plan changes

The two cache windows solve the latency problem, but there's still a question of accuracy.

Without any notification mechanism, Stub can continue serving an outdated entitlement response for up to 30 seconds after a plan change.

I wanted to reduce that window.

Kinde provides billing events that tell an application when a customer's subscription changes.

The two relevant events here are:

customer.plan_changed, sent when a customer upgrades or downgrades.
customer.plan_assigned, sent when a customer is associated with a plan.

Stub exposes a webhook endpoint that receives these events, verifies their signatures and removes the affected customer from the cache.

The next request for that customer retrieves their entitlements from Kinde.

There's also a fallback in the event handler. If an event doesn't contain a customer identifier in a format Stub recognises, the handler clears the entire cache.

I'd rather trigger a few additional lookups than leave an affected customer's old entitlements in place because the application couldn't identify which cache entry to invalidate.

I've covered the webhook verification and delivery handling in more detail in my earlier article on billing events, so I won't repeat that implementation here.

What I did want to investigate was the actual event payload.

Kinde's documentation lists the available event types but doesn't specify all their fields. I inspected a live customer.plan_changed event and found the following:

customer_id
customer_type
org_code
plan.key
event_id
event_timestamp
timestamp

There was also an issue with signature verification.

My first two webhook events were rejected because Stub expected an issuer claim in the event token. Kinde's webhook event tokens don't include one.

I adjusted the verifier to check the signature against Kinde's public keys while skipping the issuer check for webhook events only.

I also added a test to make sure an event signed with a different key was still rejected.

With that fixed, I measured how long a plan change took to reach the gateway.

Kinde stamped the event at 22:18:04.8 UTC.

A script polling the Management API every 250 milliseconds detected the new plan at 22:18:05.8, approximately one second later.

Stub received and logged the event at 22:18:07.5, which was 2.7 seconds after Kinde's event timestamp.

The webhook then invalidated the customer's cached entitlements.

The next request had to perform a fresh lookup, which had a median latency of 477 ms in my earlier measurements.

Combining those two measurements gives us approximately 3.2 seconds from the event timestamp to the next request receiving an updated entitlement response.

These figures come from two separate runs.

In this particular test, the event arrived after the new plan was already available through the Management API. That meant the lookup following cache invalidation returned the updated entitlements.

I didn't test what would happen if the webhook arrived before the new plan became readable through the API.

I also wanted to compare this with what happens when the gateway relies entirely on cache expiration.

I didn't run that experiment against live Kinde, but I tested it with the stand-in provider.

Across three trials, a plan change became visible after:

29.9 seconds
28.6 seconds
5.8 seconds

The variation is expected. The plan change can happen at any point during the 30-second fresh window, so the application waits for whatever time remains before the cached value expires.

The webhook significantly reduces that delay, but it doesn't make cache expiration unnecessary.

Events can arrive late, be delivered more than once or fail to arrive altogether.

Kinde documents a retry schedule that starts immediately and then continues after 5 seconds, 30 seconds, 2 minutes and subsequent intervals.

The webhook-id header remains consistent across retries, allowing the application to identify duplicate deliveries.

Keeping the expiration policy provides another way to recover when webhook delivery doesn't go as expected.

What should happen when the entitlement lookup fails?

So far, the gateway has been able to retrieve the customer's entitlements.

But billing providers can become unavailable, requests can time out, and a customer might make their first request while the provider is experiencing an outage.

At that point, Stub can be in one of two situations:

It has an older entitlement response for the customer.
It has never successfully retrieved the customer's entitlements.

The application needs to decide what to do in each case.

I didn't want to make this a single global decision because the consequences of granting access differ between features.

For example, allowing someone to read a report when the billing provider is unavailable might be acceptable.

Allowing someone to perform an expensive export or run an AI agent that incurs usage costs could be a different matter.

Stub divides routes into two classes:

Cheap routes: These are operations where temporarily allowing access has a relatively low cost, such as reading a report.

Sensitive routes: These are operations where granting access incorrectly could expose data or create financial costs, such as exports and agent runs.

When Stub has no usable entitlement information, it allows cheap routes and returns HTTP 503 for sensitive routes.

Here's the decision function:

export function decide(
  entitlements: Entitlements | null,
  feature: string | null,
  routeClass: RouteClass
): Decision {
  if (feature === null) {
    return {
      allow: true,
      reason: "granted"
    };
  }

  if (entitlements === null) {
    if (routeClass === "cheap") {
      return {
        allow: true,
        reason: "unknown-allowed"
      };
    }

    return {
      allow: false,
      status: 503,
      reason: "The plan could not be checked right now, so this route is closed until it can."
    };
  }

  if (feature in entitlements) {
    return {
      allow: true,
      reason: "granted"
    };
  }

  return {
    allow: false,
    status: 403,
    reason: `The plan does not include "${feature}".`
  };
}

This means each route declares its own classification in the route table.

I prefer this to a single global fail-open or fail-closed setting. The consequences of a wrong access decision depend on what the endpoint actually does.

I tested the decision logic against all three live Kinde plans, checking three routes for each customer.

All nine checks matched the expected entitlements:

Plan	Reports	Exports	Agent runs
Free	200	403	403
Pro	200	200	403
Business	200	200	200

There were two additional mechanisms I wanted in place to protect the gateway when Kinde became slow or unavailable.

The first is a timeout.

Stub stops waiting for an entitlement lookup after 1.5 seconds. This prevents a slow provider from holding application requests indefinitely.

The second is a circuit breaker.

After three consecutive lookup failures, Stub stops making further requests to Kinde for two seconds. Once that period expires, one request is allowed through to test whether the provider has recovered.

Without the circuit breaker, every new customer making a request during an outage could trigger another 1.5-second wait.

I saw both mechanisms in action during the live tests, although I hadn't deliberately caused the provider to fail.

In one run, 37 out of 40 entitlement lookups returned an unknown result.

Of the 40 application requests, 16 received HTTP 200 and 24 received HTTP 503. The 503 responses came from sensitive routes.

The 95th percentile latency was 1,512 ms, almost exactly the configured timeout.

I ran the same script again, and all 40 lookups succeeded.

I couldn't establish why the first run failed. One possible explanation is the overlap behaviour from the earlier concurrency experiment: at eight simultaneous lookups, the slowest request took 1,640 ms, exceeding Stub's 1,500 ms timeout.

That is a possible explanation, not a confirmed cause.

Stub also allows an older entitlement response to remain usable for up to five minutes while Kinde is unavailable.

This introduces another trade-off.

A customer who downgraded just before an outage could continue using their previous plan for up to five minutes.

On the other hand, a customer who is still paying for a feature shouldn't necessarily lose access because the billing provider is temporarily unavailable.

For this demonstration, I considered the first cost smaller than the second.

That decision will depend on the product. The five-minute window is configurable.

Benchmarking the gateway with a stand-in provider

Some of the behaviours I wanted to measure are difficult to reproduce reliably against a live billing provider.

I can't ask Kinde to fail on demand, deliberately delay every request or generate a controlled traffic spike.

For those experiments, I built a stand-in provider.

It has a plan table, a configurable lookup delay and a switch that forces entitlement requests to fail.

I set the delay to 150 milliseconds. This was a value I chose for the experiment; a real Kinde lookup took approximately three times as long.

Each benchmark starts a fresh server process, and both the server and load generator run on the same machine.

The first benchmark compares four gateway modes across 3,000 requests.

I used 20 concurrent requests across 200 customers, with 80% of requests distributed among 20% of those customers.

With a warm cache, the median latency was 0.97 ms. Without an entitlement check, it was 1.06 ms.

That difference falls within the variation between runs, so I wouldn't claim that entitlement checks have zero overhead. What I can say is that I couldn't reliably measure their additional cost at this precision.

The more interesting result was the empty-cache test.

Its 95th percentile latency was 151 ms because the first request for a customer had to wait for the provider lookup.

For reference, the 95th percentile is the latency below which 95% of requests complete.

The cache hit rates were:

Cache state	Hit rate
Warm cache	99.4%
Empty start	91.8%

Even with an empty cache, the gateway made only 192 entitlement lookups across 3,000 requests.

The second benchmark focused on provider failures, traffic bursts and stale entitlement responses.

When the stand-in provider was unavailable, all 1,000 requests from customers whose entitlements were already cached were allowed through.

The median latency was 1.71 ms.

I also tested customers who had no cached entitlement information:

All 40 requests to cheap routes were allowed.
All 40 requests to sensitive routes received HTTP 503.

For the next experiment, I configured the provider to hang for five seconds.

The first three requests took 308 ms, 302 ms and 306 ms. These requests each waited for the 300 ms timeout configured for that particular benchmark.

The remaining 117 requests completed with median latencies between 0.4 and 0.5 ms.

Across the entire run, the gateway made only three provider lookups.

The circuit breaker remained open for the rest of the run, which finished before its two-second recovery window expired.

I also built an inspector page in the repository so these behaviours can be explored without running the benchmark scripts.

It lets you send requests as three different customers, simulate a provider outage, assign a new feature to a customer and manually invalidate a cache entry.

The page also shows where each entitlement response came from.

What I didn't test

There are several limitations worth keeping in mind when interpreting these results.

The live measurements came from one laptop and three customers. Every live test ran from Lagos, so the lookup timings include that network path. A server closer to Kinde will see different results. Three customers also tell us very little about cache behaviour at production scale.
I never received a 429 response. Kinde documents both rate and concurrency limits, but doesn't publish their numerical thresholds. I don't know where those limits sit or what an actual limit response looks like.
One live run failed without a confirmed cause. Thirty-seven of 40 lookups returned unknown in that run, while a repeat succeeded. The 1,500 ms timeout and the latency observed during concurrent requests are possible contributing factors, but I haven't established that connection.
I timed one plan change end to end. The 1.0-second and 2.7-second measurements came from one event. I didn't compare the clocks on Kinde's side and my laptop, and the earlier plan changes weren't timed against event timestamps.
Two events reached Stub approximately half a second apart. Both were rejected. I couldn't determine whether these were separate events or a retry.
I didn't test webhook delivery while Stub was unavailable. The retry schedule comes from Kinde's documentation. I also didn't test a webhook arriving before the updated plan became readable through the API.
The expiry-only measurements used the stand-in provider. I didn't measure that path against live Kinde.
The outage and timeout benchmarks used the stand-in. Those results come from a provider I controlled with a 150 ms lookup delay.
The benchmarks used an already-loaded public key set. Verifying a token against locally available keys is different from handling a request that needs to retrieve Kinde's public keys. I didn't measure the latter.
The cache lives inside one process. If Stub runs across multiple instances, each instance has its own cache. A webhook received by one instance won't automatically invalidate the others. A shared cache or broadcast mechanism would be needed, with its own latency and implementation costs.
The customer mapping is a file. This works for three test customers. A production implementation needs a way to populate and maintain that mapping as customers sign up.
Stub checks feature presence, not remaining usage. All my test plans use unmetered entitlements, so I didn't test counted limits.
Stale responses can temporarily grant access to a feature a customer has lost. This can happen for up to 30 seconds during normal operation and up to five minutes while Kinde is unavailable.
The route classifications are my own design decision. Two categories were enough for this demonstration. A production application might need more granular classifications or customer-specific rules.

Who decides how long an entitlement can be stale?

There's one decision in this implementation that deserves more attention than it usually gets: the cache duration.

It's easy to treat a cache TTL as a performance setting. Someone adds ttl: 60 to a pull request, the application gets faster, and nobody revisits the number.

But the TTL determines how long a customer might continue receiving access based on information that is no longer accurate.

A 30-second fresh window means a downgrade can take up to 30 seconds to be enforced through expiration alone. It also means an upgrade can take up to 30 seconds to become visible if no invalidation event arrives.

Those delays have consequences beyond engineering.

Finance may care about how long a downgraded customer can continue using a paid feature. Support may care about how long an upgraded customer waits before gaining access. The customer has an expectation about when their subscription change takes effect.

The engineer building the gateway shouldn't have to make that decision alone.

The acceptable stale window should be documented, its trade-offs understood, and agreed upon by the people responsible for the product and the costs associated with incorrect access decisions.

The fast path is relatively straightforward to implement. Deciding how long the application is allowed to be wrong is the more important product decision.

Run it yourself

The complete implementation is available in the Stub repository.

You'll need Bun.

Clone the repository and install the dependencies:

bun install
bun run dev

Open http://localhost:3100.

Stub uses the stand-in provider by default and generates its own test tokens, so you don't need a Kinde account to explore the implementation.

There are 29 tests, which you can run with:

bun test

To reproduce the stand-in benchmarks, run:

bun scripts/bench.ts perf
bun scripts/bench.ts burst
bun scripts/bench.ts down
bun scripts/bench.ts slow
bun scripts/bench.ts stale

The stale benchmark takes approximately three minutes.

To reproduce the live Kinde experiments, create a machine-to-machine application with the read:billing_entitlements scope.

Then configure .env.local and customers.json using the supplied example files.

The live scripts are:

bun run live
bun run live-gate
bun run live-concurrency
bun run plan-watch

The final script watches one customer for three minutes while you change their subscription plan.

FAQ

Why not put the subscription plan in the access token?

Kinde's documented access token claims don't include billing entitlements. Tokens also last 24 hours by default, and revoking one doesn't shorten the lifetime of a token that has already been issued.

Even if you added the plan to the token, you'd still need a mechanism to update tokens that were issued before a subscription change.

Which identifier does the Kinde entitlements endpoint require?

In my live tests, it required the billing customer ID.

Using the organization code returned HTTP 500 for all three test organizations. Kinde's organization endpoint also returned billing: null, so Stub maintains a mapping between organization codes and billing customer IDs.

How quickly does a plan change reach the gateway?

In my single timed experiment, Kinde's event reached Stub 2.7 seconds after its recorded event timestamp.

The next entitlement lookup took approximately another half-second, based on the median from my earlier tests.

Without webhook invalidation, the fresh cache window can delay the update by up to 30 seconds.

Why use a stale window instead of simply reducing the cache duration?

A shorter cache duration means more requests will have to wait for a fresh entitlement lookup.

The stale window allows the gateway to return an existing response immediately while refreshing it in the background. Only a request without a usable cached answer has to wait.

Is a 30-second cache window too long?

That depends on the consequences of using outdated entitlements.

The webhook reduces the delay for plan changes that Kinde successfully reports. If your product cannot tolerate 30 seconds without a notification, you can shorten the fresh window, although that will increase the number of provider lookups.

What happens when 200 requests arrive for a customer whose entitlements aren't cached?

They share one in-progress lookup.

In my stand-in test, 200 simultaneous requests resulted in one provider call instead of 200. In the live Kinde test, all 200 requests completed within approximately the time required for one lookup.

Should an entitlement gateway fail open or fail closed?

I treat this as a route-level decision.

Allowing access when the provider is unavailable makes sense when the cost of a wrong approval is low. For operations that can expose sensitive data or incur financial costs, returning an error is the more appropriate choice.

Sources

How to Stop Background Jobs From Running on a Customer's Old Plan

Shola Jegede — Mon, 09 Nov 2026 15:45:29 +0000

Imagine a customer using an uptime monitoring service on the Pro plan. They have 10 monitors running, and their subscription allows up to 25.

They downgrade to Free, which only allows 3 monitors.

What happens to the other 7?

The billing platform knows about the downgrade. The application knows about it too. But there's another part of the system that needs to know: the background job responsible for running those monitors.

Here's how the sequence plays out:

Kinde, the billing platform, stores the new plan. Its API reports the updated limit within three seconds.
Kinde sends the application a customer.plan_changed event.
Separately, a cron trigger starts the monitoring job once a minute. Before doing anything, the job needs to know how many monitors the customer's plan allows. It gets this information from a stored copy.
If that copy is older than 30 seconds, the job fetches the latest limit from Kinde and pauses the 7 newest monitors. If the copy is younger, it trusts the stored value and all 10 monitors continue running.

The problem is in that final step.

The billing platform has already processed the downgrade, but the background job is still working with information from the previous plan. For a short period, the customer continues receiving a service their new subscription no longer covers.

The reverse happens when someone upgrades. A customer who has just paid for Pro might still have their monitors restricted by the Free plan.

This is an easy problem to overlook because both parts of the system are doing what they're supposed to do. The billing platform updates the subscription, and the background job follows the rules it was given. The issue is that they're working with different versions of the same information.

I wanted to understand exactly how long this gap could last, what happens when a plan changes during an active session, and how to keep background jobs in sync with billing changes.

I built a small uptime monitoring application called Siteward to investigate.

Siteward runs on a Cloudflare Worker, uses a cron trigger for scheduled checks, and stores its data in D1, Cloudflare's SQLite database. Kinde handles authentication and billing.

Every measurement in this article comes from experiments against a real Kinde account.

Setting up plan limits in Siteward

Siteward has two features controlled by subscription limits:

The number of uptime monitors a customer can run.
The number of alert channels they can connect.

In Kinde, these are called entitlements. An entitlement represents a feature granted by a plan, and a metered entitlement can have a maximum allowed value.

I configured two plans with the following limits:

Plan	Monitors	Alert channels
Free	3	1
Pro	25	3

Siteward retrieves these limits through the Kinde Management API:

GET /api/v1/billing/entitlements

The request uses a machine-to-machine token and returns the customer's entitlements, including their feature_code and entitlement_limit_max.

There's one detail worth understanding here: the API request needs the billing customer ID, which looks something like customer_01a1….

An access token from a signed-in user provides an organization code instead. To connect the two, Siteward maintains a small mapping between organization codes and billing customer IDs.

For the experiments, I created three organizations, each assigned to a different Siteward plan. The organization used for most of the tests was on Pro and named Dabbit Pro.

I also wrote a script that moves the organization between Pro and Free through the Management API. It does this by cancelling the existing agreement and creating a new one.

With the billing setup in place, I could start looking at what the application actually knows when a customer's plan changes.

What does an existing session know about the customer's plan?

When someone signs in to Siteward, Kinde issues an access token containing claims such as:

sub
org_code
permissions
feature_flags

However, Kinde's documented access token claims don't include the customer's current plan or entitlement limits. The billing-related claims are has_trial_period and trial_expires_on.

The token's lifetime makes this particularly important.

By default, an access token lasts 24 hours. According to Kinde's documentation, revoking a token, ending a session or deleting a user does not shorten the lifetime of a token that has already been issued.

That means a customer can have a perfectly valid access token created under a previous subscription.

For Siteward, this means the access token can establish who is making a request, but it cannot tell the application what their subscription currently allows.

I handle those as two separate checks. Siteward verifies the token signature on its server, then retrieves the customer's plan limits independently.

I wanted to see what happened when the application relied on a stored copy of those limits instead.

Measuring the problem

The first version of the monitoring job used a straightforward caching strategy.

Siteward stored each organization's plan limits in D1 alongside a timestamp. Whenever the job needed the limits, it checked the age of the stored value.

If the value was less than 30 seconds old, it used it. Otherwise, it fetched the latest information from Kinde.

To test this, I configured Dabbit Pro with 10 monitors, all due to run, and moved the organization between Pro and Free five times in each direction.

I ran the entire sequence twice.

In the first run, I cleared the stored plan before every cron execution. This forced the job to fetch the latest limits from Kinde.

In the second run, I left the stored plan untouched, simulating what happens during normal operation between updates.

Here are the results:

Change	Monitors allowed to run	Stored plan cleared first	Stored plan kept
Pro to Free, 5 times	3 of 10	3 running, 5 of 5	10 running, 5 of 5
Free to Pro, 5 times	10 of 10 (limit 25)	10 running, 5 of 5	3 running, 5 of 5

When the stored plan was cleared, the job enforced the correct limit in all 10 changes.

When I left the stored plan in place, it got every change wrong.

After a downgrade, all 10 monitors continued running even though the new plan allowed only 3.

After an upgrade, only 3 monitors continued running even though the customer had paid for a plan that allowed 25.

The interesting part is that the stored values were only a few seconds old during these tests. They weren't obviously stale. They were simply older than the billing change.

This also gives us two separate sources of delay.

The 30-second cache allows the application to keep using an old value until it expires. After that, the cron schedule can introduce another delay of up to one minute before the next job execution fetches the new value.

Those figures come from the application's cache configuration and cron schedule. I did not measure them in a deployed Worker.

The results made one thing clear: having a cache expiry policy doesn't guarantee that a background job will act on the latest subscription information.

Making the background job check the plan before running

My first change was to make the background job retrieve the plan limits before processing any monitors.

The job groups due monitors by organization. For each organization, it retrieves the entitlement limits and determines which monitors should remain active.

It then pauses monitors that exceed the limit and runs the rest.

The stored copy still helps avoid unnecessary API calls. If it's fresh, the job uses it. If it has expired, the job fetches the latest value from Kinde.

I repeated the test with the stored plan cleared before execution.

The job correctly handled both upgrades and downgrades in a single run.

However, this only proved that the reconciliation logic worked. It didn't solve the original timing problem.

During normal operation, the stored value could still be less than 30 seconds old, and the cron trigger might not run again for almost another minute.

I considered reducing the cache duration, but that creates another problem.

Every organization with due monitors could trigger a fresh Kinde API request on every cron execution. In my tests, a fresh read took approximately 1.5 seconds.

That would increase API traffic and add latency to the background job without actually addressing why the information became stale in the first place.

Kinde's billing FAQ also recommends keeping entitlement cache durations short and configuring webhooks to keep application access synchronized with subscription changes.

That pointed towards a better approach: instead of making the background job check more frequently, let the application know when something changes.

Deciding what happens to monitors that exceed the new limit

Before implementing the webhook, I needed to decide what Siteward should do with monitors that were already running when a customer downgraded.

Deleting them would be a poor experience. If the customer upgraded again a week later, those monitors would be gone.

I decided to pause them instead.

Each monitor has one of three states:

active: The monitor is running.
paused_plan: The monitor was stopped because the subscription limit was reached.
paused_user: The customer explicitly paused it.

The reconciliation logic determines which monitors should be active based on the current entitlement limit.

The oldest monitors retain their active status. When the limit decreases, the newest monitors are paused. When the limit increases, previously paused monitors are resumed, starting with the oldest.

There's another important detail: a monitor paused by the customer must never be resumed automatically because of a plan change.

I implemented this as a pure function, which means the logic doesn't depend on a database or network request.

export function reconcile(rows: MonitorRow[], max: number | null) {
  const pause: string[] = [];
  const resume: string[] = [];

  const candidates = rows
    .filter((r) => r.status !== "paused_user")
    .sort(
      (a, b) =>
        a.created_at - b.created_at ||
        a.id.localeCompare(b.id)
    );

  candidates.forEach((r, i) => {
    const allowed = max === null || i < max;

    if (allowed && r.status === "paused_plan") {
      resume.push(r.id);
    }

    if (!allowed && r.status === "active") {
      pause.push(r.id);
    }
  });

  return { pause, resume };
}

Sorting by creation time gives the function a predictable result. The same set of monitors and the same plan limit will always produce the same outcome.

It also means customers retain the monitors they created first.

I wrote six unit tests for this function, including a case where a customer had manually paused a monitor before changing plans.

Using Kinde webhooks to keep the application in sync

The second change was to introduce a webhook.

Kinde signs its webhook events as JWTs and sends them to an HTTPS endpoint exposed by Siteward.

Whenever a customer's plan changes, Kinde sends a customer.plan_changed event.

I configured Siteward's webhook handler to perform four operations:

Verify the event signature against Kinde's public keys. Kinde's event tokens don't contain an issuer claim, so the issuer check is skipped for these events only.
Record the webhook-id header. Kinde keeps this ID consistent across retries, allowing Siteward to identify and ignore duplicate deliveries.
Clear the stored plan for the affected organization.
Fetch the latest entitlements from Kinde and reconcile the customer's monitors against the updated limits.

Notice that the webhook doesn't need to contain the new entitlement values.

It simply tells Siteward that something has changed. Siteward then retrieves the authoritative information directly from Kinde.

I preferred this approach because it keeps the webhook handler relatively small and avoids having to treat the event payload as the source of truth for the customer's current subscription.

It also means that a delayed event won't overwrite the stored plan with an older value. The handler always fetches the current entitlements when it processes the event.

With that implemented, I ran the experiment again.

I changed Dabbit Pro's plan 10 times, five times from Pro to Free and five times from Free to Pro.

This time, I didn't run the cron job at all. I wanted to see whether the webhook could handle the changes independently.

It did.

All 10 plan changes triggered a customer.plan_changed event, and Siteward applied the new entitlement limits without needing a background job execution.

I also received no duplicate events during these runs. All 10 webhook requests returned HTTP 200.

The Kinde API reported the new limit between 0.5 and 2.0 seconds after each plan change.

Siteward finished applying the updated limit between 2.8 and 4.8 seconds after the change, with a median of 4.3 seconds.

The webhook handler itself took approximately 1.5 seconds. Almost all of that time was spent fetching the updated entitlements from Kinde.

The remaining delay came from Kinde delivering the event to my local development environment through a tunnel.

I also tested a plan change through the Kinde dashboard rather than the Management API. I used the Change plan action on the organization's billing page, and Siteward received the same event.

The difference was significant.

With the original caching approach, the application could continue enforcing an outdated plan until the stored value expired and the next cron execution ran.

With the webhook in place, all 10 changes were reflected in Siteward in under five seconds.

What happens when someone creates a monitor during a plan change?

There was still another part of the application to consider.

A customer doesn't have to wait for a background job to create a monitor. They can submit a request at any time.

That means a monitor could be created while a subscription is changing, even if the webhook is already configured.

The create endpoint needs to enforce the plan limit independently.

There's a common database problem here: checking the current count and inserting a new record as separate operations.

Imagine a Free customer with two monitors. Their limit is three.

If 10 requests arrive simultaneously, each request could read the database, see two monitors and conclude that another one is allowed.

By the time those requests finish, the customer could have 12 monitors.

To prevent this, I put the count check and insert into a single SQL statement:

INSERT INTO monitors (
  id,
  org_code,
  name,
  url,
  interval_s,
  status,
  next_run_at,
  created_at
)
SELECT
  ?1, ?2, ?3, ?4, ?5, 'active', ?6, ?6
WHERE
  ?7 IS NULL
  OR (
    SELECT COUNT(*)
    FROM monitors
    WHERE org_code = ?2
  ) < ?7;

The database evaluates the condition as part of the insert operation, rather than leaving a gap between checking the count and creating the record.

I tested this on the Free plan, where the limit is three monitors.

I sent 10 create requests at the same time.

Three returned HTTP 201, the remaining seven returned HTTP 402, and the database contained exactly three monitors.

When an insert doesn't create a row because the limit has been reached, the endpoint returns a 402 response with the applicable limit in the message.

That handled simultaneous requests correctly. The next experiment was more interesting.

I wanted to see what would happen if someone started creating monitors at almost exactly the same moment their subscription was downgraded.

I started Dabbit Pro on the Pro plan with eight existing monitors.

I made a request to populate the stored plan, which now indicated that the customer could have up to 25 monitors.

Then I started sending one create request every 300 milliseconds.

The first request went out at the moment I triggered the downgrade.

The first five requests succeeded.

They were sent within 1.2 seconds of the plan change, while the stored entitlement still allowed 25 monitors.

Between the fifth and sixth requests, the webhook arrived and cleared the stored plan.

The remaining five requests fetched the updated limit and returned HTTP 402.

After eight seconds, the database contained 13 monitors, but only three were active.

The five monitors created during the stale-plan window had been paused, and the reconciliation logic retained the three oldest monitors as active.

This experiment showed that the webhook and the create endpoint were handling different parts of the same problem.

For approximately 1.5 seconds, Siteward accepted monitor creation requests under the previous plan. Once the webhook processed the change, subsequent requests were rejected, and the monitors created during that window were paused.

Nothing was deleted.

All 10 requests used the same access token. It had been issued before the downgrade and remained valid throughout the experiment.

The first five requests succeeded, while the remaining five failed, even though the token never changed.

That's an important detail: the session wasn't carrying the subscription limit. Siteward was retrieving that information separately.

An unexpected detail about editing published plans

While testing subscription changes, I came across another behaviour worth documenting.

Kinde plans have versions.

I edited Siteward Pro and reduced the monitor entitlement from 25 to 15. After saving the change, the plan became a draft.

When I went to publish it, the confirmation dialog explained that existing subscribers would remain on their current plan version.

I wanted to see what this meant for an existing customer.

Dabbit Pro was already subscribed to Pro, so I checked its entitlements after publishing the updated plan.

It still reported a limit of 25 monitors.

I then moved the organization to Free and back to Pro.

This time, the entitlement limit was 15.

This meant two customers could have the same plan key, siteward_pro, but different entitlement limits.

Publishing the updated plan also didn't trigger a plan-change event because no customer's subscription had changed.

Siteward handles this by retrieving entitlement limits for each billing customer rather than assuming that every customer with the same plan name has identical limits.

An application that stores plan limits in its own database and looks them up using only the plan name would have treated both customers identically.

In this experiment, that would have produced an incorrect result for one of them.

After completing the test, I restored the Pro monitor limit to 25 and published the change again.

What these experiments don't tell us

The results are useful, but there are a few things worth keeping in mind when interpreting them.

The experiments ran locally. I used wrangler dev on a laptop in Lagos and a quick tunnel to receive webhook events. A deployed Cloudflare Worker, running closer to Kinde, will have different delivery and processing times. The measured 2.8 to 4.8 seconds includes the tunnel and local database.
I triggered the cron job manually. The one-minute interval comes from my configured schedule, not a measurement of deployed cron execution. Cloudflare documents that cron triggers can run on underutilized machines and doesn't guarantee exact execution timing.
The timing measurements came from one organization. Ten plan changes show the range observed for Dabbit Pro. They don't establish how the system behaves when hundreds or thousands of customers change plans simultaneously.
I didn't simulate failed webhook deliveries. Kinde documents retries after 5 seconds, 30 seconds, 2 minutes and subsequent intervals. Siteward uses webhook-id to ignore repeated deliveries, but I didn't verify the retry behaviour through an intentionally failed delivery in these experiments.
The plan-version experiment involved one edit. I didn't test what happens when an existing subscriber is moved to a plan whose limit is lower than the version they currently have.

These distinctions matter because the timings are measurements from a particular test environment, while some of the retry and execution behaviour comes from the respective platform documentation.

What I learned

The most interesting part of this exercise was seeing how easily a billing change can leave different parts of an application working with different information.

A request can retrieve the current entitlement when the customer makes an action. A background job is different. It might execute minutes after the original request, using information that was accurate when it was first retrieved.

Siteward now handles this through two mechanisms.

First, the background job checks the entitlement limits before processing monitors. It uses the stored copy when it's fresh and retrieves the latest value from Kinde when it isn't.

Second, Kinde's customer.plan_changed webhook tells Siteward when a subscription changes. The application clears its stored value, fetches the current entitlements and reconciles the monitors immediately.

Both mechanisms matter.

The background job provides a way to retrieve the latest information independently of webhook delivery. The webhook means Siteward doesn't have to wait for the next scheduled execution to respond to a subscription change.

The monitor reconciliation logic also ensures that customers don't lose their existing monitors when they downgrade, and that monitors paused by the customer remain paused when they upgrade.

There is still a short window in which a request can arrive before the webhook has been processed. The experiments showed that this window can result in a few monitors being accepted under the previous plan. The reconciliation process subsequently pauses them.

For this application, that was a useful result: the subscription change is enforced, and the customer's existing data is preserved.

Run the experiments yourself

The complete implementation and experiment scripts are available in the Siteward repository.

You'll need a Cloudflare account for D1 and a Kinde account with billing enabled.

Clone the repository and install the dependencies:

npm install

Create the D1 database:

npx wrangler d1 create siteward

Copy the resulting database ID into wrangler.toml, then initialise the database:

npm run db:init

Set up your local environment:

cp .dev.vars.example .dev.vars

Add your Kinde machine-to-machine credentials and the customer mapping, then start the application:

npm run dev
npm test

You'll also need a machine-to-machine application in Kinde with permission to read billing entitlements and create billing agreements.

Create three plans with monitors and alert_channels features to reproduce the experiments.

The repository includes several scripts for running the tests.

1. Compare plan changes with and without a cleared cache

node scripts/plan-flip.mjs 5

This changes the plan five times in each direction and runs the cron job after each change.

To repeat the experiment while keeping the stored plan:

KEEP_CACHE=1 node scripts/plan-flip.mjs 5

2. Test webhook-based reconciliation

WEBHOOK=1 node scripts/plan-flip.mjs 5

This runs without the cron job and waits for the webhook to process each change. You'll need a tunnel and a Kinde webhook pointing to /webhooks/kinde.

3. Test simultaneous monitor creation

SITEWARD_TOKEN=… node scripts/parallel-create.mjs

Use an access token from the dashboard. The dashboard stores it in session storage.

4. Inspect entitlement limits

node scripts/limit.mjs [plan_code]

This prints the limits Kinde reports and moves the organization to the plan you specify.

FAQ

Why not include the plan in the access token?

Kinde's documented access token claims don't include plan or entitlement information. Access tokens also last 24 hours by default, and revoking a token doesn't shorten the lifetime of one that has already been issued.

Does a downgrade in Kinde take effect immediately?

In my experiments, the API reported the new entitlement limit within approximately two seconds. This was true for changes made through both the Management API and the Kinde dashboard.

Kinde's billing policies also cover refunds and credits for unused days. The documentation I reviewed didn't specify a setting for delaying a downgrade.

Why pause excess monitors instead of deleting them?

Customers who upgrade again should be able to use their existing monitors. Pausing preserves their data, and the reconciliation logic determines which monitors should resume when their limits increase.

Is the stored plan still necessary if webhooks handle subscription changes?

Yes. Webhooks can arrive late, be delivered more than once or fail to arrive. Kinde also documents that a webhook that continues failing can be disabled after 72 hours.

The stored plan, combined with its 30-second age limit, provides a fallback when webhook delivery fails.

What prevents simultaneous requests from exceeding the plan limit?

Siteward performs the count check and insert in one SQL statement. In my test, 10 simultaneous creation requests against a limit of three resulted in exactly three monitors being created.

What happens when a customer manually pauses a monitor and then upgrades?

It stays paused. The reconciliation function only resumes monitors that were paused because of a plan limit.

Does editing a published plan change the limits for existing subscribers?

In my experiment, it didn't. Existing subscribers retained their previous plan version, while customers who moved away from the plan and subscribed again received the updated entitlement limit.

Sources

How to Stop Concurrent Jobs From Spending the Same Usage Limit Twice

Shola Jegede — Wed, 21 Oct 2026 10:25:00 +0000

Let's say you have a customer on a plan of 100,000 tokens a month. Five of their jobs start at the same moment, and each job expects to use 40,000. Each job checks the account before it starts. Each check sees 100,000 left, so each job runs.

The five jobs spend 200,000 tokens but the plan allowed only 100,000. Every check was correct when it ran, and the customer still ended up 100,000 over.

This failure is easy to miss, because it needs no bug in the check. It needs only two jobs that start close together and take a long time to finish. AI agents and long-running jobs fit that description well, and they are the work that costs the most.

So, how do you make a usage limit hold when many expensive jobs start at once, when some of them cost more than expected, and when some of them never finish?

This post answers that question with a small system I built called Earmark. Earmark is a gate that sits in front of expensive work. It holds part of the customer's limit before a job starts, stops the job at what it holds, and charges the real cost when the job ends. The plan limits come from Kinde, a platform for authentication and billing, but the design needs only two things from a provider: a way to read a limit, and a way to report usage.

Why a limit check is not a limit

A limit is the most a customer may use in one billing cycle. A billing cycle is one month of usage. A check is a read: it looks at what the customer has used so far and compares that number with the limit.

A check is a read, and a limit has to be a promise. The difference shows up when work takes time. Usage is counted when a job ends, so a job that is still running has not spent anything yet. A new job that checks the account cannot see it.

Four things go wrong, and each one needs a different part of the design.

Every job passes the check. This is the case from the start of this post. Five jobs look at the same remaining capacity at the same moment, and none of them can see the others.

A job costs more than expected. The estimate was 10,000 tokens, and the model used 15,000. A check at the start cannot stop this, because the job was allowed in at the start.

A job fails halfway. It used 60% of its estimate and then stopped. If you charge nothing, you give the work away. If you charge the estimate, you bill for work that did not happen.

A worker disappears. The process is killed in the middle of a job. Whatever the system set aside for that job stays set aside, and the customer is locked out of their own limit.

What the provider gives you

A gate needs a number to enforce, and Kinde is where that number lives. In Kinde, a plan has features, and a feature can be metered. A metered feature is billed by consumption and has a usage limit that you set for each plan. Kinde's billing concepts page describes features and limits, and its metered usage page describes how to report consumption.

The Kinde pages I read about plans and billing concepts do not say whether Kinde blocks work when a limit is reached. Kinde's guide on billing for AI and LLM-based APIs covers cost control for this kind of work, but it does not describe holding capacity before a request runs. So I treat Kinde as the source of the limit and let the gate enforce it.

Earmark reads the limit from the customer's entitlements. An entitlement is a feature that a plan grants to a customer. The API reference for the billing entitlements endpoint lists GET /api/v1/billing/entitlements, which takes a customer_id and returns each entitlement with a feature_code and an entitlement_limit_max. The call needs the read:billing_entitlements scope. A null maximum means the plan sets no maximum, and Earmark treats that as no limit.

Settled usage goes the other way, from the gate to the meter. The meter is the provider's running total for the cycle. Earmark sends the settled total as an absolute value, which sets the meter to that number, so a repeated report changes nothing. Proving that the meter matches what happened is a separate problem, and I covered it in my earlier post on reconciling metered usage.

Reserve: check and hold in one step

The first design decision is the most important one. The check and the hold must be one transaction. If a job checks first and holds second, two jobs can still check at the same moment.

A hold is a part of the limit that is promised to a running job. Earmark keeps one budget row for each account and cycle. The row has two numbers: used, which is final usage, and held, which is the sum of the holds of running jobs. The gate counts both.

export function admit(b: Budget, amount: number): Admission {
  const remaining = Math.max(0, b.limit - b.used - b.held);
  if (amount > remaining) {
    return { admitted: false, remaining, reason: `The job needs ${amount} and ${remaining} is left.` };
  }
  return { admitted: true, remaining: remaining - amount };
}

The function is pure, which means the same input always gives the same answer, so a test can check each rule alone. The database call around it reads the budget row, runs admit, and writes the new held value in the same transaction.

That transaction is what makes the five-job race safe. Convex, the database I used, runs mutations as serializable transactions. Its documentation describes optimistic concurrency control: a mutation that read data which another mutation changed is run again. When two mutations touch the same budget row, one of them commits first, and the other is run again and sees the new held value. Any database with serializable transactions, or a conditional update on one row, gives you the same guarantee.

Each job also sends a key. If the same key arrives twice, the gate returns the same job and holds nothing new. This matters because a client that times out will try again, and a second hold for the same job would take capacity that no job uses.

Run: the hold is also the cap

Once a job holds 40,000, the hold works as a limit on the job. The worker charges each step against the hold before it starts the next step.

export function chargeStep(held: number, used: number, tokens: number) {
  const room = Math.max(0, held - used);
  const charged = Math.max(0, Math.min(Math.floor(tokens), room));
  return { charged, capped: charged < tokens };
}

If a step would pass the hold, the gate charges only what is left and marks the job as capped. The job stops. This handles the job that costs more than expected: the customer cannot be charged for more than the hold, and no job can take capacity that another job holds.

A capped job loses its unfinished work, and that is a real cost. I chose it on purpose. The alternative is to let a job pass its hold and hope that capacity is free, which brings back the first problem in a smaller form.

Every charge also renews the hold. Earmark gives each hold a lease, which is a time limit that a sign of life renews. The lease lasts two minutes. A worker that is alive and charging keeps its hold. A worker that stops does not.

Settle: pay for what was used

When a job ends, the gate settles it. It charges what the job used and returns the rest of the hold to the budget.

export function settle(held: number, used: number) {
  const charged = Math.min(used, held);
  return { charged, released: held - charged };
}

A job that held 40,000 and used 31,200 is charged 31,200, and 8,800 goes back. A job that failed halfway is charged for the half it did. Partial completion needs no special case, because every ending is the same step with a different used value.

The idea is older than usage billing. A card authorization holds an amount on a customer's card before the final price is known. Stripe's documentation on placing a hold describes the same pattern: you capture up to the held amount, and the rest of the hold is released. Settling a job is the same move with tokens.

A worker that disappears never calls settle, so something else must. A sweeper runs on a schedule and looks for running jobs whose lease has run out. It settles each one as expired, at the usage it had recorded, and returns the rest of the hold. A worker that comes back late finds its job already settled, and the gate ignores it. Settling a job twice does nothing, so the late call and the sweeper cannot charge the same job twice.

Fail closed when the limit is unknown

A gate is only as good as its number. The limit comes from the provider, and the provider can be slow or down.

Earmark keeps the last limit it read. It uses that limit as it is for 60 seconds, so it asks the provider at most once a minute for each account. If the provider cannot be read, Earmark keeps using the last known limit for up to 10 minutes. After that, it refuses new work until the limit can be read again.

I think refusing is the right default for expensive work. A wrong refusal costs the customer a short delay. A wrong admission costs real money, and it may spend past a limit that changed while the gate could not see it. For cheap work I would choose differently, because the price of a mistake is small. The cost of the work should set the default.

A plan can also change while jobs run. Earmark does not change a running job's hold when the plan changes. The new limit applies to the next admission.

Keep the budget row small

Every admission and every settlement writes the same budget row, so that row is a serial point. This is the price of an exact limit, and it is the right price. The design goal is to keep everything else off that row.

Convex's error documentation names two causes of write conflicts: frequent writes to one document, and mutations that read a whole table or a wide range. Both apply here, so Earmark follows three rules.

First, the rate limit is a counter and not a scan. Earmark allows 120 new jobs a minute for each account. Counting recent job rows would make every running job conflict with every new admission. Earmark keeps a small counter on the account's billing row instead.

Second, the report state lives in its own table. Whether a new total is waiting for the provider is not part of admission, so it does not belong on the budget row.

Third, a conflict is retried only where retrying is safe. Admission is safe to repeat because it is keyed, and settlement is safe because a job settles once. Each retry waits a random time that grows, up to one second, and gives up after eight tries.

What happened when I ran it

I ran the system in two ways. The first is the Concurrency Lab, a page in the app that starts five jobs of 40,000 at the same moment against a plan of 100,000. It runs the same jobs twice: once with a plain check, and once with a hold.

With a plain check, all five jobs started and 200,000 was spent, which is 100,000 over the plan. With a hold, two jobs started, three were refused with the amount that was left, and 80,000 was spent. This is the case from the start of this post.

The second run is a load script. It sends 120 jobs of 1,000 tokens, 50 requests at a time, against an account that has room for exactly 100 of them.

The gate admitted 100 jobs and refused 20. No request failed, and the run spent 100,000 tokens, which is all the plan had left and not one token more. Nothing was still held when the jobs ended. The run took 6.7 seconds, about 18 requests a second. The median time to start a job was 1,316 ms, the 95th percentile was 3,717 ms, and the 99th was 5,099 ms.

Limits

This system has edges, and I want to name them.

The numbers come from a laptop. I ran the load script from a laptop in Lagos against a Convex cloud development deployment. The latency includes that distance. The run shows that the limit holds under concurrency, not what a production host will do.
One account is one serial point. The load run used one account, and every admission wrote one budget row. Many accounts run in parallel because each has its own row, but I did not measure that.
The worker is simulated. The jobs are real database records that hold, charge and settle. The work itself is a loop that charges tokens. I did not run an actual model, and I did not kill a real process. The crash path is tested with a simulated crash.
I did not run it against a live Kinde account. The limit read and the usage report are tested with fake responses. I took the entitlements endpoint from its API reference, so the scope and field names still need a live check.
A hard cap costs work. A job that reaches its hold stops, and the customer loses what it had not finished. A softer policy is possible, but it brings the overspend back.
The gate is only as good as the estimate. An estimate that is too low means more capped jobs. An estimate that is too high locks capacity, and the gate refuses a job even though room exists. Earmark does not learn estimates. You have to supply a good one.
Plan changes and caching are only touched on. Earmark keeps a limit for a minute and uses the new one at the next admission. Keeping every service in agreement during a plan change is a larger problem than this post covers.

Three questions to ask of your own gate

Before you copy this design, ask these questions of the gate you have now.

Does your check also take a hold? If it only reads usage, it is a check and not a limit.

What happens to the hold when a worker stops? If the answer is "nothing", one crash can lock a customer out until someone notices.

How far apart are your estimates and your real costs? Pull the last month of finished jobs and compare the two. The size of that gap tells you how much capacity your holds will lock up and how often your jobs will be capped.

Run it yourself

The code is in the repository. It runs with no billing account, using a built-in stand-in for the plan limit. In the project folder, run these commands:

npm install
npx convex dev
npm run dev

Open /app and open the Concurrency Lab. To repeat the load run, start with a fresh account and run node scripts/load.mjs --jobs 120 --estimate 1000 --limit 100000 --concurrency 50 --url http://localhost:3000, using the port that npm run dev prints. The 49 tests run with npm test.

FAQ

Why hold the estimate instead of charging it?
A hold is a promise that can be returned. A charge is final. When the job ends early or costs less, the unused hold goes back to the customer's limit, and the customer pays only for what the job used.

Why not lock the whole account while a job runs?
A lock makes every other job wait, even when the plan has plenty of room. A hold takes only the part that the job needs, so many jobs can run at once until the limit is full.

What if the estimate is wrong?
If it is too low, the job is capped at its hold and stops. If it is too high, the extra hold goes back when the job settles, but it blocks other jobs while the job runs. Set the estimate close to what the job costs at the high end of normal.

Why a lease and not a timeout on the job?
A job can run for a long time and still be healthy. A lease asks a different question: has the worker shown a sign of life recently? Every charge renews it, so long jobs keep their hold and dead workers lose it.

Does this need Kinde?
No. The gate needs a way to read a limit and a way to report settled usage. Kinde supplies both in the version I built. The built-in stand-in in the repository shows the same behavior with no account.

Why send settled usage and not holds to the meter?
A hold is a promise and not usage. If the meter showed holds, a job that was refused or that returned most of its hold would still change the customer's invoice.

Sources

How to Reconcile Metered Usage Billing When Events Arrive Late, Twice, or Never

Shola Jegede — Wed, 21 Oct 2026 00:24:42 +0000

A short link app has a simple job: count the clicks and bill the owner for them. The app counts each click in its own database. The billing provider keeps a second count, called a meter, and the invoice comes from the meter. Both counts start equal, but nothing keeps them equal.

A click can reach the database after the total was sent. A retry can send the same usage twice. A report can fail on a timeout and never be sent again. Each case leaves the meter wrong, and the customer sees the error only as an invoice that is too high or too low.

The hard question is this: how do you prove that the meter matches what really happened, and fix it when it does not?

This article answers that question with a small system I call Notch. Notch is a short link and QR code app that bills per click. It keeps a ledger of every click, sends totals to the meter, and runs a reconciler that compares the two, repairs a difference, and keeps the proof. The billing provider in my test is Kinde, a platform for authentication and billing, but the design needs only two things from a provider: a way to set a total, and a way to read it back.

Why the meter and the ledger drift apart

A ledger is the list of events your app wrote down, one row for each click. A meter is the provider's running total for a billing cycle, which is one month of usage. The ledger is the record, and the meter is a copy that you keep up to date. Three things break that copy.

Usage arrives late. The app sends a total at 10:00. A click is recorded at 10:01. Until the next report, the meter is one click behind. If the cycle closes before that report, the meter stays wrong, and a closed cycle is the one you can no longer change.

Usage arrives twice. A retry runs after a timeout, or a second job writes to the same meter. If the write adds usage, the second write adds it again. The customer pays for clicks that happened once.

Usage never arrives. A report fails on a timeout or a rate limit, and nothing sends it again. The meter stays low. Nothing fails loudly, so nobody sees it.

All three look the same from the provider's side. The meter holds a number, and the provider cannot tell you whether the number is right. Only the app has the record to check it against, which is why the first rule of the design is that the ledger is the record.

What the provider lets you do

A design has to match what the provider actually accepts, so I read the Kinde documentation and then tested each behavior on my own account before I relied on it.

Kinde's usage endpoint takes a meter_type_code. The value absolute sets the running total to the number you send, and it is the default. The value delta adds the number to the running total. That difference decides the whole design. An absolute write is safe to repeat, because the second write sets the same total. A delta write is not, because a retry adds the usage again, and I confirmed this by writing the same delta twice and reading the total move twice.

Stripe takes the opposite route. Each Stripe meter event has an identifier that you can set, and Stripe uses it to prevent a second count of the same event. Kinde's documentation shows no such key, so with Kinde the safe write is the absolute one.

The write is only half of the API. To compare the meter with the ledger, the app must read the total back. Kinde's read endpoint returns the current cycle by default, and all_cycles=true returns every record. The read needs the read:meter_usage scope, which is separate from the write scope.

Three more facts shaped the design:

A meter_usage_timestamp is stored on the record, but it does not select a billing cycle or change the total. A write that I backdated became its own record, and the live total did not move.
Each write is a record. After about 1,200 writes, the total still read correctly.
Kinde bills recurring metered charges in arrears, and it starts each cycle on the customer's sign-up date, not on the first of the month.

The ledger is the record

Every click becomes one row in the ledger, and the row has a key that the app makes before it writes anything. The redirect to the destination goes out first. The write happens after the response, and it carries the same key on every try. A second try with the same key adds nothing, so a retry cannot count a click twice.

A click that a person or a bot should not have caused is not deleted. The ledger is append only, so a correction is a new row with units: -1, written into the cycle of the click it corrects. The total is the sum of all rows, and the history stays readable.

One more detail keeps the ledger fast. If every click updated one counter row, many clicks at once would fight over that row. Notch splits each counter over 64 rows and adds them on read. Each click picks one shard at random, so two clicks rarely touch the same row. The click and its counter update commit in one transaction, and the same transaction marks the cycle as changed.

That mark is what connects the ledger to the meter. A cycle is marked once, and more clicks while it is marked add no more writes. The mark says one thing: the meter may be behind, so send the total again.

The reporter sends totals, not clicks

Every minute the reporter takes up to 100 marked cycles, deletes the marks, and reads the ledger total for each cycle. It sends each total to the meter with an absolute write, 10 at a time.

Sending totals fixes two of the three failures at once. A repeated report is harmless, because it sets the same number. A late click is also harmless, because it marks the cycle again and the next report carries the new total.

The third failure, the report that never arrives, needs one more step. If a write fails or gets a 429, the reporter marks the cycle again after the loop ends. The next minute tries again. Kinde documents a limit on concurrent requests and a RateLimit-Reset header on a 429, which gives the seconds until the limit resets. I never saw a 429 in my tests, so that path is covered by tests with a fake meter, not by a real rejection.

If a cycle has already closed, the reporter does not write. A closed cycle goes to a review list, because a write to a closed cycle has an effect that I could not confirm.

The reconciler proves the copy

The reporter keeps the meter close. It cannot prove the meter is right, because it only knows what it sent. The reconciler does the proof, and it runs nightly and on demand.

For each cycle, the reconciler reads every ledger row and adds them up. It checks that sum against the running counters, and it rebuilds the counters if they differ. It then compares three numbers: the ledger total, the last total the app sent, and the total the provider holds.

The comparison is a short pure function, so a test can check each case alone:

export function decide(e: Evidence): Decision {
  if (e.provider === null) return { verdict: "unreadable", repair: null, /* ... */ };
  if (e.provider === e.ledger) return { verdict: "match", repair: null, /* ... */ };
  if (e.closed) return { verdict: "closed_cycle_drift", repair: null, /* ... */ };
  return {
    verdict: e.provider > e.ledger ? "over_billed" : "under_billed",
    repair: e.ledger, // the ledger is the record
    /* ... */
  };
}

The function never trusts the meter over the ledger. When the two differ in an open cycle, the repair is always the ledger total. The cause text depends on the evidence: if the last report failed, the reconciler says so, and if the meter moved without a report from the app, it says that a report was counted twice or something else wrote to the meter.

A repair is not finished when the write returns 200. The reconciler reads the meter back and checks that it now equals the ledger. Only then does it record "repaired and confirmed". If the read back differs, the run ends with an error.

Every run stores evidence, which is one row with the three numbers, the verdict, the cause, and the read back. A run on the same numbers adds no new row, so running the reconciler twice leaves the record unchanged. That makes the evidence cheap to keep and useful to show: the answer to "was this customer billed correctly in October" is a row, not a memory.

The meter's calendar is not yours

One mistake would corrupt every number above. A cycle named by calendar month would put a click on 20 October into 2026-10 while the provider counts it in a cycle that started on 20 September. On 1 November the app would then write a small October-only total over the meter's real cycle.

Kinde starts cycles on the customer's sign-up date. Notch therefore takes an agreement start when you connect an account, and it names each cycle by the month it starts in. Take an agreement start of 20 October at 15:29 UTC. A click on 19 November belongs to the cycle that began on 20 October, and a click on 21 November belongs to the next one.

Two rules follow. The start must fall on the 1st to the 28th of a month, because a start on the 31st has no clear meaning in a short month. The start is set before the first click and refused afterward, because changing it later would move old clicks into different cycles. Kinde's FAQ says that only monthly cycles are supported, so Notch handles monthly cycles only.

What happened with live clicks

I ran the system against a real Kinde account in two ways. First, I sent 5,000 clicks to a short link at 100 at a time, from a laptop dev server. All 5,000 were recorded. The server handled about 97 redirects a second, with a median of 455 ms, a 95th percentile of 916 ms and a 99th percentile of 1,633 ms. The reconciler then found a match at 6,048 in the ledger, the sent log and the meter, with 0 corrections.

That first run proves that the loop works when nothing is wrong. The second run shows what happens when something is. After a later run of 2,100 clicks, the ledger held 8,457 clicks, and the meter held 8,031. The meter was 426 clicks behind. The reconciler marked the cycle as under-billed, wrote 8,457 with an absolute write, read it back, and stored the evidence. The next run matched at 8,457 and added no new row.

Those are the numbers from the runs I watched:

5,000 of 5,000 clicks were recorded at 100 at a time.
The 2,100 click run reached about 65 redirects a second, with a median of 274 ms and a 99th percentile of 705 ms.
The reconciler repaired a gap of 426 clicks and confirmed the repair by reading the meter back.

The cause text that the reconciler stored reads "The ledger changed after the last report. A late click or a correction arrived." That is the late case from the start of this post, found and closed with evidence, and nobody had to notice it.

Limits

This system has edges, and I want to name them.

The reconciler proves the copy, not the capture. A click that never reached the ledger is invisible to it. The load run counts what was sent against what was recorded, and that is a separate check.
Only the current cycle can be read. Notch reads the current cycle from Kinde. A closed cycle that drifts is flagged for a person, and the reconciler does not repair it.
The cycle rollover is untested. My test agreement starts a new cycle on 21 November. Until that date passes, I have not seen what Kinde does at the exact moment of the change.
I have not seen a real 429. The 1,200 writes that I sent at about 180 a second all returned 200. The retry path rests on Kinde's documentation and on tests.
I do not know what a past-month write does to an invoice. Kinde's documentation does not say. Stripe's documentation says that a late event may be accepted without changing an invoice that is already final. Check your own provider before you rely on a late write.
The load numbers come from a laptop. They show that the design holds under load, not what a production host will do.
One provider was tested. The design needs a set operation and a read, but I only ran it against Kinde.

The rule that holds it together

The ledger is the fact, and the meter is a copy of it. Once you treat them that way, a late click becomes a new mark, a repeated report becomes the same number, and a failed report becomes a retry. Each case is the same case: the copy is behind, so send the total again and check it.

To run it yourself, clone the repository and run npm test for the 58 tests. To measure your own deployment, run node scripts/load.mjs <short code> --clicks 5000 --concurrency 100. Then compare the ledger with the meter in the reconciliation panel, and look for a gap that you did not cause. Do let me know what you find in the comments.

FAQ

Why send a total instead of each click?
A total is safe to repeat and safe to send late. A per-click write needs a unique key at the provider for every click. Where a provider has no such key, a repeated write counts twice.

What if the provider only supports delta writes?
Keep the ledger and send each change with an idempotency key if the provider accepts one. Stripe does, through the meter event identifier. If the provider has neither an absolute write nor a key, you can only detect a drift and not repair it safely.

Why not trust the meter after a repair?
A write that returns 200 says that the provider accepted the request. It does not say what total the provider now holds. Reading the value back is the only proof.

Does the cycle anchor matter if I bill on the first of the month?
No. With no anchor, a cycle is a UTC calendar month, which matches a provider that starts every cycle on the first. The anchor matters when the provider starts cycles on the customer's sign-up date.

How often should the reconciler run?
Run it nightly, plus on demand after an incident. It reads every ledger row, so run it in pages. Notch does 50 accounts a page, 5 at a time.

Sources

How to Keep Billing State Correct When Webhook Events Repeat, Arrive Late, or Fail

Shola Jegede — Tue, 20 Oct 2026 16:40:12 +0000

When building apps in production, the billing provider you decide to use tells your app what happened to a customer, such as a payment that succeeded, a plan that changed, or a subscription that ended. Your app then turns each message into a change in its own database: a plan column, a monthly allowance, or a flag that lets someone in.

Those messages travel over a network that duplicates, delays and drops, so the same payment message can arrive three times, a cancellation can arrive after the upgrade that replaced it, and a server can stop half way through a message. Each case goes on to corrupt a customer's state in a different way, and none of them shows up when you test with one clean message.

Which brings me to a question a developer friend of mine asked me last week: how does an app keep a customer's plan and allowance correct when the events that change them can repeat, arrive out of order, or stop in the middle?

This post builds the answer as a small pipeline: an endpoint that stores each event before it answers, a queue in Postgres that holds the stored events, and a worker that applies them with effects that are safe to repeat.

To show this in practice, I built an example app called "Bottle", it's a reminder app with a free plan and a paid plan using Kinde under the hood to send billing events.

Kinde is a platform for authentication and billing, and here it is the billing provider whose plan and payment changes the app must follow. After building the app, I tested the pipeline against the events Kinde sent for my own account and I'll be breaking it all down in this article.

What goes wrong when a handler applies each message as it arrives

First, we will start with the handler most apps write, which reads the message, changes the database, and answers 200 (OK). That works on a quiet day with a clean network, but three things can potentially break it.

The same event arrives more than once. If the app applies a payment and the 200 answer never reaches the provider, the provider sends the payment again. The handler adds the monthly allowance a second time. And will do the same a third time if a third copy arrives.

An old event arrives after a newer one. A customer upgrades, so the old plan is cancelled. The cancellation happened first, but a retry delays it, and it arrives last. A handler that only trusts the order of arrival will set the customer back to Free.

The server stops in the middle. A plan change needs two writes: the plan, then the allowance that goes with it. If the process stops between them, the customer is on Pro with a Free balance. The provider retries, but by then the state is already split.

I wrote a handler that applies each message on arrival and ran nine deliveries of three events through it. The nine are the plan change three times, the payment five times, and one old cancellation. The worker stops once during the first plan change, and this is what the account looked like along the way:

After delivery 2, the plan was Pro and the balance was still 10, because the process stopped before the allowance write.
After delivery 7, the plan was Free, because the old cancellation arrived last and won.
After delivery 9, the balance was 2,950. The correct balance is 500.

The account ended on Pro only because the plan change happened to arrive a third time. The handler had no rule that made it come out right, and the rest of this post builds those rules.

What the provider promises

A fix has to match what the provider actually does, so I read Kinde's webhook documentation before I designed anything. These are the parts that matter for correctness:

The body of each request is a JWT (a signed token) with the content type application/jwt. You check its signature against the keys Kinde publishes at /.well-known/jwks on your Kinde domain.
Every request carries a webhook-id header. Kinde describes it as unique to a webhook and an event, and the same on every attempt. The docs tell you to use it as the idempotency key.
A 200 answer means delivered. Any other answer, or no answer, makes Kinde try again.
The retry schedule is roughly: immediately, then after 5 seconds, 30 seconds, 2 minutes, 10 minutes, 1 hour and 4 hours, then every 12 hours until 36 hours have passed.
An endpoint that fails for 72 hours is disabled, and the account owner gets an email.

That list explains the three failures. The promise is "we keep trying until we hear 200", so a copy arrives whenever your 200 gets lost, even if your app applied the event correctly. A retry that waits 10 minutes also arrives after every event sent in those 10 minutes, which is how order is lost. The docs do not promise an order for events on one endpoint, and the retry schedule shows why no one could. The pipeline assumes no order and expects copies.

The event itself carries what the handler needs. Here is a customer.plan_changed event from my account, with the fields I use and the rest left out:

{
  "type": "customer.plan_changed",
  "event_id": "event_01a1…",
  "timestamp": "2026-10-21T02:29:24.481368869+11:00",
  "event_timestamp": "2026-10-21T02:29:23.403633+11:00",
  "data": {
    "user_id": "kp_4940…",
    "customer_id": "customer_0198…",
    "customer_type": "user",
    "plan": { "key": "customer_pro_plan" }
  }
}

The plan is an object, and its key is at data.plan.key. The user and customer ids tell the handler which account to change. The two timestamps matter later, in the section on ordering.

The pipeline

The design follows from two facts. An event can arrive many times, so the app needs one stored copy and a counter. An event can also arrive at any time, so the app must apply it in a way that does not depend on when. Four rules do that work.

Rule 1: Store the event, then answer

The endpoint does three things and nothing else: it checks the signature, writes the event to a table, and answers 200. Applying the event happens after the answer.

If the signature is not valid, the endpoint answers 401 and stores nothing. If the database write fails, the endpoint answers with an error, and Kinde tries again later. The 200 therefore means one thing: the event is stored. It does not mean the event is applied. That difference is what lets Bottle survive a crash, because a stored event can run again and a lost event cannot.

The signature check is one call to the jose library, which fetches Kinde's keys once and caches them. After the check, the event goes into inbox_events:

INSERT INTO inbox_events (source, dedupe_key, type, payload, occurred_at, …)
VALUES ($1, $2, $3, $4::jsonb, $5, …)
ON CONFLICT (source, dedupe_key)
DO UPDATE SET seen_count = inbox_events.seen_count + 1
RETURNING id, (xmax = 0) AS inserted

Rule 2: One key for each event

The webhook-id header is the dedupe key, and the table has a unique constraint on it. The first copy inserts a row. Every later copy hits the conflict, adds one to seen_count, and does nothing else.

The last line of the query tells the caller which case happened. In Postgres, xmax is 0 for a row the statement just created, so (xmax = 0) is true only for a new event. Bottle also writes one row to a receipts table for every delivery. That gives the app a record of all nine copies even though only three events exist.

I considered keeping seen ids in memory or in a cache. Both lose the ids when the process restarts, and a restart is exactly when copies arrive. The key has to live in the same database as the event.

Rule 3: Apply each event in one transaction, with a lease

A worker claims stored events and applies them. Two workers can run at once, for example when the endpoint starts one after its answer and a timer starts another, and they must not take the same event, so the claim uses a clause Postgres provides for this:

UPDATE inbox_events
   SET status = 'processing', attempts = attempts + 1,
       locked_until = now() + interval '30 seconds'
 WHERE id IN (
   SELECT id FROM inbox_events
    WHERE (status IN ('received', 'failed') AND next_attempt_at <= now())
       OR (status = 'processing' AND locked_until < now())
    ORDER BY occurred_at, id
    FOR UPDATE SKIP LOCKED
    LIMIT 25)
RETURNING *

SKIP LOCKED makes a second worker pass over rows the first worker holds. The 30-second lease handles a worker that dies. When the lease ends, the row is claimed again.

The worker then runs the handler and marks the event applied inside one transaction. Either both happen or neither does. If the process stops half way, the database rolls back, and the event runs again from the start. That removes the third failure from the list, because the plan write and the allowance write can no longer be separated.

When a handler throws, the event goes back to failed and waits. The waits are 5 seconds, 30 seconds, 2 minutes, 10 minutes and 1 hour. After six attempts the event is marked dead, which keeps it visible without running it forever.

I chose Postgres over a message queue for one reason. The claim, the effect and the mark must share a transaction, and a separate queue cannot join one. Billing events are also low in volume, so one more service would add cost without adding safety.

Rule 4: Make every effect safe to repeat

The first three rules reduce how often a handler runs twice, but only this rule makes a second run harmless, and it has to protect two effects.

The allowance is a ledger row, and the row has a key made of the account and the month. Writing it again does nothing:

INSERT INTO ledger (account_id, period, delta, reason, entry_key)
VALUES ($1, $2, $3, $4, 'pro:' || $1 || ':' || $2)
ON CONFLICT (entry_key) DO NOTHING

Both the payment handler and the plan change handler try to write this same row. Whichever event arrives first grants the Pro allowance, and the other finds the row and skips it. That is why a payment that arrives before its plan change does no harm. The plan change grants the allowance when it comes.

The plan has its own protection, and it depends on time rather than arrival.

Arrival order is not the order things happened

Each stored event has an occurred_at time taken from the event, not from the moment the copy arrived. A plan change applies only if its time is newer than the plan change already stored:

UPDATE accounts SET plan = $2, plan_changed_at = $3
 WHERE id = $1 AND plan_changed_at < $3
RETURNING id

If the update changes no row, the handler records ignored: a newer plan change is already stored and stops.

The diagram shows the effect. The cancellation happened three minutes before the upgrade, but it arrives seventh. The database already holds a newer plan change, so the cancellation does nothing and the customer stays on Pro.

The event has two timestamps. In the two events I stored, event_timestamp was about one second earlier than timestamp. I read the first as the time of the change and the second as the time Kinde sent the message. Bottle uses event_timestamp when it is present and falls back to timestamp. The choice matters, because the guard compares times, and a time that means "when it was sent" would reorder events that were sent late.

Replay and ignored events

Two features come from the same idea: a stored event is a fact you can look at again.

An ignored event keeps its reason. When a handler chooses to do nothing, it writes why. The log shows "ignored: a newer plan change is already stored", not a silent skip. An event the handler cannot use shows up the same way.

A replay puts a stored event back in the queue. It sets the status to received and lets the worker run it again. Replay is safe because of Rule 4. A replayed payment finds its ledger row and writes nothing. A replayed plan change finds a newer change and is ignored. Replay is how you run stored events again after a handler changes or after an outage, and it needs no new code path.

In Bottle, each billing event on the account page has a Replay link. The route behind it only accepts events that belong to the signed-in account, because replay should not let one customer requeue another customer's events.

What happened with live events

The tests cover the rules, but I also wanted events sent by Kinde itself. In Kinde, I moved my own user from Pro to the Starter plan and back to Pro. Kinde sent the events to Bottle's live endpoint. To test a late delivery, I then replayed the older Starter change after the Pro change had applied.

The log on the account page shows the result in the screenshot above. The Pro change reads "plan set to pro, Pro allowance granted". The older Starter change reads "ignored: a newer plan change is already stored". The account shows the Pro plan and 500 bottles for the month.

The same nine deliveries from the first section run through the pipeline on Bottle's Engine Room page. The page uses the same intake code and handlers as the live endpoint, on a fresh test account each time.

The numbers are these:

9 copies arrived and 3 events were stored. The other 6 copies were counted and dropped.
1 plan change stopped in the middle. It stayed stored, and the next copy ran it to the end.
The late cancellation was ignored, because a newer plan change was stored.
The account ended on Pro with a balance of 500, and the allowance was granted once.

The handler that applies on arrival ended with 2,950. That is 2,450 more bottles than the plan allows, and nothing in the database records how they got there.

Limits

This design has edges.

Order depends on the provider's clock. The guard compares event times from Kinde. Two plan changes with the same timestamp are treated as a tie, and the second one is ignored.
The live log shows two event types. The plan change and the cancellation came from real Kinde events. The payment handlers and the overdue-invoice handler follow the documented event names and are covered by tests, but I did not watch Kinde send them.
The token's timestamp has no age limit. Kinde's docs mention checking the timestamp for replay protection. Bottle does not reject old tokens. The dedupe table already refuses a second copy of any webhook-id, but it does not expire a token.
A dead event needs a person. After six attempts, an event is marked dead and stays visible. Bottle does not page anyone. A production system needs an alert on that status.
Postgres is the queue. It suits the low volume of billing events. A stream of millions of events a day is a job for a message queue.

The rule that holds it together

The stored event is the fact, and a delivery is one attempt to tell you about it. Once the app treats it that way, a copy becomes a counter, a late event becomes a comparison of times, and a crash becomes a retry. None of the three needs special code, because each one is the same case: a stored event that has not been applied yet.

To run it yourself, clone the repository, run npm test for the 23 tests, and run npm run dev for the app. The Engine Room page runs the nine deliveries in your browser.

After testing it out yourself, do drop a comment below telling me your findings, even if you test it with a completely different billing provider.

FAQ

Why answer 200 before the event is applied?
The provider stops retrying once it hears 200. Applying an event can be slow or can fail. If the app answers only after applying, every failure becomes a retry of an event that may be half applied. Storing first makes the answer fast and keeps the event safe.

Which key should I dedupe on?
Use the key the provider documents. For Kinde, that is the webhook-id header, which stays the same across attempts. Store it with a unique constraint in the same database as the event.

Can I use a queue like SQS instead of Postgres?
Yes, but the claim, the effect and the "applied" mark can no longer share one transaction. You then need the effects to be safe to repeat on their own, which Rule 4 requires anyway.

How do I test retries and reordering?
Write down a sequence of deliveries that includes copies, a late event and a crash, and run it against the handler. The Engine Room page does this, and the repository has the same sequence as a test.

Sources

How to Stop an AI Agent That Lies About Its Own Spending

Shola Jegede — Wed, 23 Sep 2026 00:54:29 +0000

In September 2026, Mandiant described a failure in its AI Risk and Resilience report, drawing on data from Google's Threat Intelligence Group. An accounting agent entered a runaway loop where it fired more than 15,000 API calls, burning about $50,000 in cloud spend in under an hour, and yet no attacker had touched it.

This failure points at a specific gap that we will discuss in this article. A token issued to an AI agent proves who issued it, but tt does not say who owns the spending it causes, or how much that owner has left to spend.

To reproduce the failure mode described by Mandiant, we will be building a demo application with two products: Kinde, which is an identity provider that issues signed access tokens for machine callers and can attach custom data to them, and Convex, a backend platform with a reactive database that runs the server that checks every call.

The map, in one screen

In our demo, three agents will call the same Convex route. What changes is which claims their Kinde token carries, and whether the server checks those claims against its own ledger or against the request itself.

One agent's token carries no owner and no spend limit. The server treats it as unattributed and uncapped. A second agent's token carries both claims. The server reads them from the verified token and checks them against a running total it keeps in Convex. A third agent reuses the second agent's own token. The server still verifies the signature, but its enforcement route reads the spend total from the request instead of the ledger.

During the test, both agents made the same calls, one at a time, against a real Kinde tenant and a real Convex deployment, each call priced at $2.50. The unmetered agent made 8 calls. All 8 went through, for $20 spent, with no ceiling in sight. The metered agent made the same calls with a $10 limit set on its token. The first 4 went through, taking it to exactly $10. The server denied every call after that with a 402.

The live dashboard mid-run: the unmetered agent's total climbing past $20, the metered agent capped at $10, and the naive control still reporting $0 claimed spend.

Why doesn't a valid signature stop overspending?

A JSON Web Token, or JWT, is a signed, verifiable claim. Kinde signs every access token it issues. Any server that trusts Kinde can check that signature without calling Kinde back.

A valid signature goes on to answer one question: did Kinde issue this token, and has it expired? It does not answer the second question that we have for this test: has this agent already spent its budget? And that second question needs its own check, written into the server, on top of signature verification.

const { payload } = await jwtVerify(token, getJwks(), {
  issuer: requiredEnv("KINDE_ISSUER"),
  audience: requiredEnv("KINDE_M2M_AUDIENCE"),
});
const mode = modeForClientId(payload.azp as string | undefined);
const owner = readProperty(payload.application_properties, "agent_owner") ?? "unattributed";
const spendLimitRaw = readProperty(payload.application_properties, "agent_spend_limit_usd");

jwtVerify throws if the signature is wrong or the token expired. Everything after that line is this build's own code, not Kinde's. It reads two claims off the verified token: who owns this call, and what that owner's limit is.

How does Kinde attach an owner and a limit to a token?

Kinde stores custom data on an M2M application as Properties. A Property has one of three types: single line text, multi line text, or boolean. There is no number type, so a spend limit gets stored as text and parsed with Number() on the server.

A Property must be scoped to Applications to attach to an M2M app. Its Private toggle must be off, or it never appears in a token. From there, the application's Properties tab sets a value, and its Tokens tab, under Token customization, turns that Property on for the M2M token.

The unmetered application's Properties tab: empty, so the server treats every call from it as unattributed and uncapped.

The metered application's Token customization tab: both the owner and spend-limit properties switched on, so they ride along in every access token this app issues.

A token that carries these claims does not carry them as plain values. Each one arrives wrapped:

{ "agent_owner": { "v": "finance-ops" }, "agent_spend_limit_usd": { "v": "10.00" } }

The server has to read .v off each claim, not the claim itself.

What happens when the server trusts the caller instead of the ledger?

The metered agent's enforcement route reads its spend total from Convex. While, a second route in this build, built only to test the failure mode, reads the spend total from a claimedSpend query parameter on the request instead.

Both routes check the same math against the same kind of token. Only the source of the number changes.

// The metered agent's route, checked against the ledger Convex itself wrote
allowed = currentTotal + args.costUsd <= args.spendLimitUsd;

// The naive route, checked against what the caller claims
allowed = args.claimedSpendUsd + args.costUsd <= args.spendLimitUsd;

The test for the naive route fired 20 calls at it with the metered agent's own token, each one claiming $0 already spent. Each call cost $2.50. The ledger was reset to zero before the run, so the result could not be explained by leftover balance from an earlier session.

The naive route trusts the number in the request. The metered route reads the number it wrote itself.

All 20 calls returned 200. The claimed spend stayed $0.00 the whole time. Real spend, tracked in the background by the same ledger the metered route uses, reached $50.00 against the agent's own $10.00 limit.

A clean result here could lie two ways. If the route never wrote to the ledger at all, a $0 claimed-spend result would only mean nothing got recorded, not that the check passed. If the test forgot to reset the ledger before the run, an early denial could look like the limit working when it was really an old balance carried over. This build hit exactly that second bug once. An early version of the metered agent's own proof script reused a ledger balance left over from a previous run, and the agent looked capped from call one, before any of the fix under test had run. Resetting the ledger before every run fixed it.

Does concurrency break the count?

The metered agent's ledger check runs inside a Convex mutation. Convex queues mutations and runs them one at a time, even when many calls arrive at once. That queuing should keep the total correct under concurrent load. It says nothing about which specific calls get to be part of that total.

The concurrency test settled that directly. It fired 20 calls at the real, ledger-enforced route at the same instant, using the same $10 limit and $2.50 cost as the sequential test. The math says exactly 4 calls should be allowed, since 4 times $2.50 is $10.

Exactly 4 calls were allowed. The count was correct on every run.

The 4 calls that won were not the first 4 to arrive. They were calls 20, 7, 5, and 19, in that order, based on the order Convex actually processed them in. If a system assumed a correct total also meant first-come-first-served fairness, this build would have proven that assumption wrong. Convex's mutation queue guarantees the total. It does not guarantee arrival order.

Limitations

This build ran on one Convex deployment. It did not test enforcement split across multiple deployments or regions.

Kinde stores the spend limit as text, not a number. The server parses it before comparing it, and a malformed value would need its own check. However, this build assumes an operator sets the Property correctly.

The mutation that resets a ledger before each proof run exists only for testing. No production request path in this build calls it.

The naive route exists only inside this demo. Kinde did not cause it, and nothing about Kinde's tokens encourages it. This build's own server code chose to trust a claimed number instead of its own ledger, on purpose, to show what that choice costs.

The concurrency test fired 20 calls at once. It did not test hundreds or thousands of simultaneous calls, or calls arriving across more than one server instance.

Where this leaves things

A signed token proves Kinde issued it. It does not prove the number attached to a call is true. The only number a server can trust is the one it wrote itself, in its own ledger, after its own check. Everything else is something the caller said.

FAQ

What is the difference between the unmetered and metered agent in this build?
The unmetered agent's token carries no owner or spend-limit claim, so the server treats every call as unattributed and uncapped. The metered agent's token carries both claims, and the server checks them against a running total it keeps for that owner in Convex.

Why does the naive agent's token still pass signature verification if it is lying about its spend?
A JWT's signature only proves Kinde issued the token and it has not expired. It says nothing about the rest of the request. The naive route reads a claimed spend value from a query parameter the caller controls. Signature verification has no way to catch that, because checking a claimed dollar amount is not what a signature check does.

Why store the spend limit as a Kinde Property instead of a custom claim somewhere else?
A Kinde Property attaches directly to an M2M application. Token customization can include it in every token that application issues. No separate lookup is needed to find which agent owns which limit. The tradeoff is Kinde's Property types, which store text, not numbers, so the server parses the value before comparing it.

What happens if two calls arrive at the same instant?
Convex queues mutations and runs them one at a time. Both calls still get a correct answer: whichever one Convex processes first sees the lower ledger total and may be allowed, and the other sees the updated total and may be denied. The total stays correct. Which specific call wins is not guaranteed to match the order the calls arrived in.

Does this pattern work with an identity provider other than Kinde, or a backend other than Convex?
Yes. Any provider that lets an application attach custom, scoped claims to an M2M token can take Kinde's place here. Any backend that runs writes safely under concurrent load, keeps a database, and can verify a JWT can take Convex's place.

Why not just rate-limit each agent to a fixed number of calls per minute instead of tracking total spend?
A call-count limit says nothing about cost. One agent's calls might cost $0.10 each. Another's might cost $4.00 each. A spend ledger caps what actually matters: money spent, not requests made.

Reproduce it

Set up two Kinde M2M applications. Give one no Properties. Give the other an agent_owner and an agent_spend_limit_usd Property, both turned on under Token customization.
Set KINDE_ISSUER, KINDE_M2M_AUDIENCE, and each application's client ID and secret on the Convex deployment itself with npx convex env set. Convex's HTTP actions read the deployment's own environment, not .env.local.
Run npm run prove:spend-cap to see the unmetered and metered agents run their sequential calls.
Run npm run prove:naive-bypass to see the claimed-spend route accept 20 calls it should have denied after 4.
Run npm run prove:concurrency to fire 20 calls at the real, ledger-enforced route at the same instant.

Full source: github.com/sholajegede/agent-token-ownership-demo

Every number here came from three real scripts against a real Kinde tenant and a real Convex deployment. Clone the repo and run them against your own tenant. If your own naive route somehow keeps real spend under the limit anyway, or your own concurrency test lets through more calls than the math allows, tell me why in the comments.

How to Stop a Leaked AI Agent Key From Still Working With Kinde Access Tokens

Shola Jegede — Sat, 19 Sep 2026 02:20:21 +0000

In September 2026, VentureBeat reported that AI agents had used exposed credentials to breach 395 organizations. The report made a simple point: identity systems still treat an agent's credential the way they treat a human's password. Nobody expects a human to type a password every few minutes, so nobody had built agent credentials to expire that fast either. And this was a gap the attackers saw and used. A separate breach at Hugging Face traced back to the same failure: an agent held a credential that outlived the task it was issued for.

Most AI agents get one machine-to-machine (M2M) access token at start-up and keep it for the life of the process. If that token ends up in a log file, a support ticket, or a copied environment variable, it stays valid for however long its issuer configured it. For most identity providers, that is somewhere between one hour and one day.

To try to solve this problem, I built two versions of the same agent against Kinde, an identity provider that issues M2M access tokens and verifies every call against them. Kinde also lets you set, per application, how long an access token lives before it expires. One agent in my build takes Kinde's default and never checks it again. The other agent treats its token as something with a short shelf life, and rebuilds it before that shelf life runs out.

Then I'm going to try to steal both tokens and reuse them again.

Why the leaked token still works

An M2M credential in Kinde has two parts: a client ID and a client secret, and an access token that Kinde issues when an app presents that ID and secret. The access token is what the app actually sends on every API call. Kinde signs it, and any server that trusts Kinde can verify that signature without calling Kinde back.

The client ID and secret rarely change. Most teams set them once, in an environment variable, and leave them alone. The access token is supposed to be different: Kinde gives it a lifetime, after which it stops working no matter who holds it.

The problem is what an agent's code does with that access token after Kinde hands it over. A static agent fetches one token when it starts, stores it in memory, and reuses it for every call until the process restarts. If that process runs for hours or days, so does the token. If someone copies the token during that time, they hold something that is still good.

A rotating agent does the opposite. It checks the token's age before every call. And before the token gets close to expiring, the agent throws it away and asks Kinde for a new one. A copy of that token, made at any point in its short life, stops working within minutes.

Both agents get their tokens the same way: a client-credentials grant. The agent sends its client ID and client secret directly to Kinde's token endpoint. There is no human and no browser involved in this process. Kinde sends back a signed access token. This is the standard OAuth flow for service-to-service calls, and it is the same flow whether the token that comes back lives for two minutes or two days. The grant itself does not decide how long the token is good for. The application's settings in Kinde do.

The enforcement seam

Both agents in this build call the same API, and that API runs the same check on every request, with no branch for which agent is calling.

sequenceDiagram
    participant Agent as Agent (static or rotating)
    participant Kinde
    participant API as Records API (Convex)

    Agent->>Kinde: client_credentials grant
    Kinde-->>Agent: access_token (24h or 120s expiry)
    Agent->>API: GET /api/records?action=... (Bearer token)
    API->>Kinde: verify signature via JWKS
    API-->>Agent: 200 + data, or 401 if expired/invalid

    Note over Agent,API: Later, an attacker replays a captured token directly
    Agent->>API: same token, no agent involved
    API-->>Agent: static: 200 (still valid) / rotating: 401 (expired)

The API is a single Convex function. It reads the bearer token from the request, checks its signature against Kinde's public key set (JWKS), and checks that the token has not expired:

const { payload } = await jwtVerify(token, getJwks(), {
  issuer: requiredEnv("KINDE_ISSUER"),
  audience: requiredEnv("KINDE_M2M_AUDIENCE"),
});
const mode = modeForClientId(payload.azp as string | undefined);
if (!mode) return await deny(403, "token not issued to a known agent client");

The jwtVerify call, from the jose library, throws the moment a token's signature is wrong or its exp claim has passed. That single check is what turns a Kinde-configured expiry into an actual rejection. Nothing about this function knows or cares which agent sent the request; it treats every token the same way, and the token's own expiry decides the outcome.

The only difference between the two agents lives in Kinde's dashboard, not in this code. The static agent's Kinde application keeps the default access token expiry: 86,400 seconds, one full day. The rotating agent's application is set to 120 seconds. Change that one number, and the identical verification code above starts rejecting tokens on a different schedule, with no redeploy of its own.

Two credential managers, one verification function

The static agent's credential manager fetches a token once and holds it:

let staticToken: CachedToken | null = null;
export async function getStaticCredential(): Promise<CachedToken> {
  if (staticToken) return staticToken;
  staticToken = await fetchM2MToken(
    env("STATIC_AGENT_CLIENT_ID"),
    env("STATIC_AGENT_CLIENT_SECRET"),
  );
  return staticToken;
}

The rotating agent's credential manager checks the token's age before every use, with a 15-second safety margin, and fetches a new one if the old one is close to expiring:

const ROTATION_SAFETY_MARGIN_MS = 15_000;
let rotatingToken: CachedToken | null = null;
export async function getRotatingCredential(): Promise<CachedToken> {
  const now = Date.now();
  if (rotatingToken && now < rotatingToken.expiresAt - ROTATION_SAFETY_MARGIN_MS) {
    return rotatingToken;
  }
  rotatingToken = await fetchM2MToken(
    env("ROTATING_AGENT_CLIENT_ID"),
    env("ROTATING_AGENT_CLIENT_SECRET"),
  );
  return rotatingToken;
}

Neither function knows about the other, or what the API on the other end will do with the token. The only thing that changes the outcome is how long Kinde told each token to live, and whether the calling code respects that.

A third file in the build, a closed action registry, lists the only three calls either agent may make: list_records, read_record, and export_records. The API rejects any other action before it even looks at the token. This keeps the demo's scope narrow: the test here is about token lifetime, not about which actions an agent should be allowed to take.

The bug I found during hardening

During a review of the API's verification code, a bug turned up. When a request carried no token, or a token from an application the API did not recognize, the logging code defaulted that request's log entry to mode: "static". An anonymous or malformed request would then show up in the dashboard as if the static agent had made it.

This did not affect the proof numbers below, since every call in this build came from one of the two known agents. It got fixed before the proof ran, though: unattributed requests now log as mode: "unknown", a separate value from "static" and "rotating".

Live proof

With both Kinde applications configured (24 hours for static, 120 seconds for rotating), the proof works in four steps. First, each agent asks Kinde for a token. Second, both tokens are captured at the same instant, the way a log scraper or a support-ticket screenshot would capture them. Third, the script waits past the rotating agent's 120-second window. Fourth, both captured tokens get replayed, unmodified, straight against the API, without going through either agent's own code.

Static Agent's app, at the default 86,400-second expiry:

Rotating Agent's app, set to 120 seconds:

The live dashboard, built with Convex, shows both agents' history in real time as each call happens:

The replay produced these results:

Agent	Token expiry	Replay result	Status	Latency
Static	24h (default)	still works	200	1353ms
Rotating	120s	dead	401	938ms

A stolen token's usable window: a day vs two minutes

The static agent's leaked token authenticated successfully more than two minutes after it was captured, using the exact same replay the rotating agent's token failed. It will keep authenticating for the rest of its 24-hour life, because nothing in this build, or in most production setups, checks whether a token has been copied. Kinde's own JWT verification only checks whether a token is signed correctly and has not expired. It has no way to know the token in front of it is a copy.

The rotating agent's leaked token failed after 120 seconds, the same limit set on its Kinde application. Whoever captured it had, at most, two minutes to use it before Kinde's own signature check started returning 401 on every attempt.

The gap between those two numbers, a day against two minutes, is the entire value of short-lived credentials. Nothing about the attack changes. Nothing about the API's verification code changes. Only the answer to "how long is a copy of this token worth anything" changes, and that answer comes from one number set on one screen in Kinde's dashboard.

What I think about all of this

Kinde already gives every M2M application a token with an expiry, and most teams never touch that setting. I don't think that is a Kinde problem. Most agent code never checks the expiry either; it just holds whatever Kinde handed back at start-up. Short-lived credentials are not a feature you have to add. The primitive already exists. What is missing is code that treats the expiry as real, instead of code that mints a token once and assumes it will always still be good.

The fix in this build took two changes: setting one number in Kinde's dashboard, and writing a credential manager that checks the clock before it checks a cache. Neither change touched the API's verification logic at all.

I would argue the harder habit to build is not the code. It is remembering that a credential an agent holds in memory is a credential that can leave that memory. Developers on Hacker News asked whether they would trust an agent with an API key at all. Most kept landing on the same answer: give the agent a temporary, narrowly scoped credential, issued through something that logs when it was requested. Not a long-lived key sitting in an environment variable. A short access token from an identity provider that already tracks every issuance is a plain way to get that, without building a separate credential broker.

Limitations

Kinde documents rotating an M2M application's client secret as a manual action, triggered from the dashboard or the Management API, not something Kinde does on a schedule. This build rotates the access token, which Kinde issues fresh on every client-credentials request. That is the layer that matters for a leaked-token scenario, but it is a different mechanism from secret rotation, and the two should not be confused for each other.

120 seconds is a number chosen so this proof runs in a couple of minutes, not a production recommendation. A production window more commonly sits between 5 and 15 minutes, traded off against how often a workload can tolerate a refresh call.

The API's JWT verification returns a generic 401 for every failure case. It does not tell a caller whether their token expired, or whether it came from an application the API does not recognize. That is fine for this demo. A real audit trail would need to tell those cases apart.

Code and sources

Full source: github.com/sholajegede/rotating-credentials-demo

Every number in this article came from a live run against a real Kinde tenant and a real Convex deployment. Clone the repo, run the same proof against your own tenant, and drop your numbers in the comments. If your static agent's leaked token survives longer than 24 hours, or your rotating agent's dies in under 120 seconds, tell me why.

How to Auto-Revoke a Claude Agent's Access When a User Is Offboarded With Kinde Webhooks

Shola Jegede — Thu, 17 Sep 2026 01:36:32 +0000

Imagine this scenario, someone on your team gets offboarded while their AI agent is still mid-task. Nobody remembers the agent is even running. It's just doing what it was told, on behalf of someone who, as of a minute ago, doesn't work at your company or on your team anymore.

Does it stop?

Well, I built a small app to test that scenario, using Kinde to handle sign-in and to hold the record of who's still active. I signed a real user in, handed their agent a task, and while the agent was still working through it, suspended that same user, in Kinde's dashboard, mid-run, to see what the agent would do next.

Unsurprisingly, the agent kept working on the task.

You see, suspending or deleting a person changes how Kinde itself sees that user, but it doesn't touch the access token their agent is already holding, because nothing about a suspension reaches back into a token that was already signed and handed out before it happened. The token still verifies exactly as it did before the suspension, so the agent has no way to know anything changed.

Everything I am going to talk about in this article is about closing that gap, and about what I actually found while doing it: webhook deliveries measured live, a production bug that could have left an offboarded user's record looking active forever, and a hard number for how long an offboarded person's agent keeps acting before anything catches it.

Why a suspended user's token still works

Let's start with what an access token actually is, because the whole gap follows from it. An OAuth access token isn't a receipt you hand back to check against a ledger. It's a signed claim, a small JSON payload with a cryptographic signature attached, and whatever's checking it just verifies that signature against a public key rather than calling home to ask if the token's still good. That's the entire appeal of the design: an API can confirm a token is genuine without a database round trip on every request.

Which means suspending a user in Kinde only changes a row in Kinde's own database. It doesn't reach the token at all, because there's nothing there for it to reach: the token was already handed out, already signed, already valid until whatever expiry it was minted with. Revoking it properly would mean tracking every issued token in a lookup table somewhere, which throws away the entire point of signing one in the first place, or it would mean just waiting for the thing to expire on its own.

I suspended a signed-in test user mid-session, and the app's own check kept reporting that user's access token as valid, seconds after Kinde had already suspended them.

So that gap isn't a bug in Kinde, and it isn't a bug in OAuth either. It's just what a stateless credential is, by design, and the real question is what you build on top of it.

The shape of the fix

Two pieces close the gap. A webhook tells the app when Kinde's view of a user changes, and a check runs before every single agent action, reading the app's own record of that user instead of trusting whatever was true when the session started.

The agent itself is a small Claude Messages API loop, working through a closed set of three tools against a demo set of internal resources: list_resources, read_resource, write_resource. None of what follows is specific to what the agent does. It's specific to the one place every tool call has to pass through before it's allowed to run at all.

Building the seam

First, we start with the registry that defines those three actions, because it's closed by construction rather than by convention. An action that isn't in this table doesn't half-exist somewhere in the code, waiting to be called by accident. It just doesn't exist:

export const ACTION_REGISTRY: Record<ActionName, ActionDefinition> = {
  list_resources: { name: "list_resources", destructive: false, params: {} },
  read_resource: {
    name: "read_resource",
    destructive: false,
    params: { resourceId: { type: "string", required: true } },
  },
  write_resource: {
    name: "write_resource",
    destructive: true,
    params: {
      resourceId: { type: "string", required: true },
      title: { type: "string", required: false },
      body: { type: "string", required: false },
    },
  },
};

Both the tool schema handed to Claude and the enforcement check are built from this same table, so the two can never quietly drift apart from each other the way a schema and a permissions list usually do once someone forgets to update one of them.

Every tool call the model makes passes through a single function, enforceToolCall, which looks up the acting user's current status and hands it to a small, pure decision function underneath it:

export function decideAccess(input: {
  mode: EnforcementMode;
  userStatus: UserStatus;
}): { decision: SeamDecision; reason: SeamReason } {
  if (input.mode === "naive") {
    return { decision: "allow", reason: "naive_mode_no_check" };
  }
  if (input.userStatus === "active") {
    return { decision: "allow", reason: "user_active" };
  }
  if (input.userStatus === "offboarded") {
    return { decision: "refuse", reason: "user_offboarded" };
  }
  return { decision: "refuse", reason: "user_unknown" };
}

Naive mode allows every call without ever looking at that status, which is the vulnerability this whole piece is about, reproduced on purpose so both modes can run side by side against the exact same code and prove the point cleanly. Enforced mode is stricter in a way that matters: it allows exactly one case, a confirmed active user, and refuses everything else, including a status the seam couldn't even resolve because a read to Convex failed. An unknown status doesn't get the benefit of the doubt.

That status comes from a webhook. Kinde sends a signed event on user.updated and user.deleted, the receiver verifies the signature, and then it does one more thing that has nothing to do with the signature at all:

export function isFreshWebhookEvent(
  event: WebhookEvent,
  now: number = Date.now(),
): boolean {
  const eventTime = Date.parse(event.timestamp);
  if (Number.isNaN(eventTime)) return false;
  return Math.abs(now - eventTime) <= MAX_CLOCK_SKEW_MS;
}

A valid signature only proves Kinde signed this payload at some point. It says nothing about when that was, so without this check, a captured event replayed months later would sail straight past signature verification, since the signature itself never expires, and past deduplication too, because dedup only catches an event id it's already seen before. MAX_CLOCK_SKEW_MS is five minutes. Anything older than that gets rejected the same way a forged signature would.

The bug the build actually found

Hardening this receiver turned up a real ordering bug. The first version recorded the webhook's delivery, for deduplication, before it applied the actual effect of marking the user offboarded. That ordering has a quiet failure mode: if the effect write failed right after the delivery had already been logged, a retried webhook would look like a duplicate of one already handled and get skipped. The user would never actually get offboarded, and nothing about the system would ever try again.

The fix took one line of reordering, but it only works because of one property underneath it: the effect, markOffboarded, is idempotent, so running it twice is always safe. That's why it now runs first, unconditionally, ahead of the bookkeeping whose entire job is to stop it from running a third or fourth time. Recording the delivery first and applying the effect second felt like the more natural order to write. It was also the less safe one.

Proving it, live

scripts/e2e-narrative.ts runs one task against one real Kinde test user, twice: once with the seam in naive mode, once enforced, suspending that same user for real, mid-run, both times. Nothing in this script is simulated. It drives a real agent loop, fires a real suspend call at Kinde, and waits on the actual webhook to arrive over a tunnel before it checks what actually happened.

	naive	enforced
actions allowed after offboarding	2	0
where the run stopped	it didn't, ran to completion	step 2, reason `user_offboarded`

The same story shows up in the operator console, so I ran it once more while writing this, offboarding the signed-in user on purpose partway through a task:

Step one lands while the user's still active. The offboard request goes out. Step two refuses, user_offboarded, with a cutoff latency of 2697ms measured from when the offboarding itself landed in the database, not from when the run started. Every one of those decisions lands in an audit log under a shared correlation id, so a run's full timeline can be pulled back up after the fact, not just watched live:

The numbers that matter more than the demo

Across every live webhook delivery in this build, latency ran from 652ms to 2058ms, across suspends, restores, deletes, and role changes alike. That's not instant. Most people assume it is.

The enforcement check itself doesn't belong anywhere in that number. It's one indexed read against Convex, and next to webhook delivery it's close enough to free that it doesn't move the total.

So the more honest way to describe total revocation speed is this: it's webhook delivery latency, plus however long until the agent gets around to its next real action, and that second part isn't a fixed system number at all. An agent moving faster, with no artificial pacing between steps or several tool calls requested in the same turn, gets caught just as fast on its very next call, without any extra mechanism needed to catch it sooner. The floor here is roughly one model round trip, not the enforcement check sitting underneath it.

A webhook can also be missed or delayed, because that's what "best-effort delivery" actually means in practice, so I built a reconciliation sweep on top: a cron job checking every active user's live status directly against Kinde every five minutes. I tested it against a forced scenario: suspended a real user, then manually pushed the app's own record back to active, simulating a webhook that never arrived at all. The cron caught the drift and corrected it on its own, before I ever triggered a manual run.

Here's what I think

"Revocation" is the wrong word for what any of this does, and I think that matters more than it sounds like it should. Nothing here revokes anything. What actually happens is a live check gets bolted on top of a credential that was never revocable to begin with, and I'd rather describe it that way than pretend otherwise, because that's closer to what every system in this space is doing under the hood, whatever the marketing copy on top of it calls it.

I'd also argue this check shouldn't be the thing you bolt on during a hardening pass once the demo already works, which is exactly the order it happened in during this build. The seam existed early on, but the reconciliation backstop and the timestamp check both came later, as production polish, and I think that ordering is backwards for anything that runs unattended. A browser session gets re-validated more or less by accident, on every page load, because there's a human sitting there generating new requests the whole time. An agent loop doesn't get that for free. It holds one credential, validated once at the start, and then it acts on that credential in a loop nobody's watching in real time. If the check isn't built into that loop from the first line, there's no accident later that adds it back in.

The usual objection to this is cost: a database read on every single tool call sounds expensive once an agent's task means dozens of calls instead of one. But this build's own numbers say that objection doesn't hold up, at least not with an indexed lookup like the one here. The check disappears next to webhook latency, and webhook latency itself disappears next to how long a person actually takes to notice someone's gone and go click suspend. The genuinely expensive part of this whole system is the five to fifteen minutes between someone walking out the door and someone else noticing. Nobody optimizes for that number, even though it's the one that actually decides how exposed you are.

What this doesn't solve

The token-still-valid gap from the top of this piece is real, and nothing here closes it. This build works around it, by checking liveness on every call, instead of trying to make the token itself stop working. A system that actually needs the token revoked needs a different mechanism entirely, like short-lived tokens or an introspection endpoint hit on every use, and both of those trade away the exact stateless-verification benefit that made a signed token worth using in the first place.

The reconciliation sweep is a five-minute backstop, not the primary path, and if the webhook's doing its job, the sweep never finds anything to correct.

runs.timeline, runs.get, and the audit queries in this build are also unauthenticated Convex reads. A run id or a correlation id is enough on its own to read that run's whole timeline, which is fine for a single-user demo console and not fine at all for anything with untrusted users in it.

And the action registry here covers exactly three read and write actions on a demo resource. That's enough to prove the pattern holds. It's nowhere near enough to prove the pattern scales to a real authorization model with real permission boundaries between real resources.

Where this leaves the agent

Back to the opening scene: someone gets offboarded, and their agent is mid-task. Whether its next few tool calls go through was never actually a question about the token. The token was always going to keep working right up until it expired on its own, offboarding or not, because that's just what it is. The real question was always whether anything standing between the agent and the action it's about to take bothered to check, right then, whether the person behind it was still around.

Code and sources

The full build, the enforcement seam, the webhook handler, the reconciliation cron, and scripts/e2e-narrative.ts, is on GitHub: sholajegede/offboarding-revocation-demo.

The webhook signing and delivery model comes from Kinde's own webhooks documentation, and the suspend and restore actions used throughout the demo follow Kinde's Management API and its notes on suspending and deleting users.

The reconciliation sweep runs on Convex's cron jobs, and the agent loop follows Anthropic's tool use documentation for the Messages API. Every number in this piece came from runs.timeline and auditLog in that same Convex deployment, read straight off the live runs.

Clone it, wire it up against your own Kinde tenant, and offboard a test user mid-run. Your webhook latency might not match mine, your reconciliation sweep could catch different drift, and that's the point really. Drop your numbers in the comments and let me know what you find.

How to Let an OpenAI Agent Act in a User's GitHub Without Storing API Keys

Shola Jegede — Fri, 14 Aug 2026 16:58:48 +0000

Picture this: A user in your app connects their GitHub account. They ask the agent to deal with an issue, so it reads the thread, writes a comment that answers the question, and opens a pull request with the fix. That comment has to appear under the user's name, because their teammates need to see who acted. So you run an OAuth flow, take the access token GitHub returns, and put it in a column beside the user's row. Every action after that reads the column.

Now look at what you just created. That column holds a standing secret. It works on the first day and it keeps working long after the reason for it has gone, because the user wanted one comment and the token stays until somebody deletes it. It carries whatever scopes were convenient when you set it up rather than the narrow thing the task needed. Worst of all, it multiplies. The token sits in your database, in last night's backup, in any log line that printed it during an incident, and in the memory of whichever worker read it a minute ago. When that user decides they want the agent out of their account, revocation turns into a hunt through all of those places, and you are asking them to trust that you will find every copy.

In this tutorial, you'll build a demo that answers the other question: what if the app never holds the token at all? An OpenAI agent reads issues, writes comments, and opens pull requests inside a user's real GitHub account, while the app stores no GitHub token anywhere. Kinde Connected Apps holds the GitHub authorization and the refresh token. A broker asks Kinde for a token at the moment of each action, uses it for that one call, and then drops it. The user cuts the agent off with one click.

By the end, you'll have built:

A token broker that every GitHub action passes through, with nothing else in the app calling GitHub
Two deployment modes, one that stores nothing and one that stores a standing key, so you can watch the difference
An OpenAI tool-calling loop that proposes actions and never sees a credential
A live connect flow through Kinde Connected Apps, and a kill switch that cuts the agent off
An audit trail where every brokered action, refusal, and revocation carries a correlation id
A runtime assertion that hunts for a stored credential after a real agent run
An end-to-end script that walks the whole story and exits non-zero when any check fails

Everything in this article comes from running that build against live services. Every number, reason code, and token shape below is measured, including one result that weakens the easy version of the story.

What You'll Build
Prerequisites
How the Demo Is Structured
Why a Stored Key Fails
The Connected Apps Primitive
How to Connect a User's GitHub
How to Build the Broker
How to Model the Data
How to Decide the Storage Mode
How to Build the Agent Loop
Two Revocation Levers, and One Is a Decoy
The Central Finding: Revocation Reaches the Connection, Not the Token
The Two Modes, Side by Side
How to Keep Least Privilege
What the Operator Controls
How to Harden It for Production
How to Prove the Tests Can Fail
Honest Limitations
What This Design Costs
What You've Built, and Where to Take It
Wrapping Up
Resources

What You'll Build

The operator signs in, connects GitHub, and writes a task in plain text. The console shows which deployment mode is running, how many tokens the app keeps, and what the agent has done so far.

Every step of a run streams into a live timeline. The agent asks for a tool, the broker performs it, and each row carries the correlation id that ties the timeline to the audit trail.

That run posted a real comment on a real issue, written by the signed-in GitHub account, while the app stored nothing.

Then the operator clicks Revoke. The GitHub connection is cut, the next action the agent asks for is refused, and the operator stays signed in to the app. You'll see both of those later, once the code behind them is in place.

Prerequisites

To follow along, you'll need:

Node 18 or later, and comfort with TypeScript. You don't need to be an expert, but you should be comfortable reading async/await and typed function signatures.

A Kinde account. The free tier covers everything here. You'll create a back-end web app for sign-in, a machine-to-machine app for the Management API, and GitHub as a connected app.

A GitHub account, and a repository you're happy to write to. The agent posts real comments, so use a throwaway repository rather than anything that matters.

An OpenAI API key, and a model that supports tool calls.

A Convex account for the data store. The free tier is enough.

Basic familiarity with OAuth helps but is not required, since the whole point of this build is that Kinde runs that flow for you.

How the Demo Is Structured

Every GitHub action follows the same path, and the storage mode decides only where the credential comes from:

operator writes a task
        │
        ▼
   OpenAI agent  ──── proposes an action, holds no token
        │
        ▼
     BROKER  ─────────┬─── connected-app: ask Kinde, use once, discard
        │             └─── stored-key:   use the token the app keeps
        ▼
     GitHub  ──── the comment appears as the user
        │
        ▼
   audit row  ──── correlation id, storage mode, outcome

The broker is the only module that obtains a credential, and the only module that calls GitHub. The agent sits above it and never touches either. A test walks the source tree and fails the build if a second GitHub caller ever appears.

Why a Stored Key Fails

This shape is now common enough that the largest AI vendor ships it. Company knowledge in ChatGPT connects to Slack, SharePoint, Google Drive, and GitHub, and answers from the data in those accounts (OpenAI). Workspace agents go further, because they run repeatable workflows, use connected apps, and complete tasks end to end (OpenAI). Agents act inside a user's own third party accounts today, so the credential question stops being a detail you can defer.

Two of the costs are measurable, and this build measured both.

The stored key carries far more access than the work needs. A token that a developer already had is the token that gets used. In this build the stored key carried gist, read:org, repo, and workflow. The four actions in the demo need public_repo and nothing else. The broker recorded that gap on every action, in the audit row itself.

Central revocation does not reach it. The app holds the token, so nothing you do at your identity provider touches it. I revoked the connection and the agent carried on writing to the repository for another four and a half minutes, and it would have carried on for hours.

The fix is not a better column, or tighter access control on the column. The fix is to stop holding the credential.

Where the token lives, in each mode. In stored-key the app keeps a long-life token in its own database and calls GitHub with it. In connected-app the app keeps nothing, so it asks Kinde for one short-life token for each action. The second path has one more hop, and that hop is the whole difference.

The Connected Apps Primitive

Kinde Connected Apps let your users reach third party tools through Kinde (Kinde docs). Kinde holds the GitHub authorization. Your app asks Kinde for a token when it needs one.

Three Management API endpoints matter. I read them from the published OpenAPI specification rather than from memory, because the documentation pages do not carry the parameter detail.

Operation	Method and path	Parameters
Get the auth URL	`GET /api/v1/connected_apps/auth_url`	`key_code_ref` (required), `user_id`, `org_code`, `override_callback_url`
Get a token	`GET /api/v1/connected_apps/token`	`session_id` (required)
Revoke	`POST /api/v1/connected_apps/revoke`	`session_id` (required)

The token response carries two fields and nothing else: access_token and access_token_expiry. The schema has no refresh token field. My app never receives one.

Note the first parameter name. It is key_code_ref, a reference string that you set on the connected app in the Kinde dashboard. It is not a connection id. I had assumed otherwise and had to rename an environment variable.

Your app keeps one value from this flow: the session_id. Kinde returns it on the callback. That handle is not a GitHub credential. It grants nothing on its own, it needs your Kinde machine to machine credentials to be useful, and Kinde stops honouring it the moment you revoke the connection.

In the Kinde dashboard, the connected app carries a name, the GitHub client id, the GitHub client secret, and a key. That last field is the key_code_ref your code sends, and here it reads github.

The Permissions tab is where least privilege gets set. The Scopes box holds one chip, public_repo, and every other GitHub scope below it stays switched off.

Search that list for a user scope and nothing comes back, which is why this build reads the acting GitHub login from the API response instead.

The user sees GitHub's own consent screen, and it asks for exactly what Kinde requested.

The Repositories row reads "Public repositories" and nothing else, which is the public_repo scope reaching the user. The Organization access block is separate: GitHub offers it on every OAuth consent screen, and this demo grants none of it.

How to Connect a User's GitHub

The connect flow has three parts, and your app stores one value from it.

First, ask Kinde for an authorization URL for this specific user.

export async function getConnectedAppAuthUrl(options: {
  kindeUserId: string;
  overrideCallbackUrl?: string;
  token?: string;
}): Promise<ConnectedAppAuthUrl> {
  const env = kindeManagementEnv();

  const response = await managementRequest<{
    url?: string;
    session_id?: string;
  }>({
    path: AUTH_URL_PATH,
    query: {
      key_code_ref: env.KINDE_GITHUB_CONNECTED_APP_KEY,
      user_id: options.kindeUserId,
      override_callback_url: options.overrideCallbackUrl,
    },
    token: options.token,
  });

  if (!response.ok || !response.body?.url || !response.body?.session_id) {
    throw new Error(
      `Kinde did not return a connected app auth url (HTTP ${response.status}).`,
    );
  }

  return { url: response.body.url, sessionId: response.body.session_id };
}

Second, send the user to that URL. GitHub shows its own consent screen. The URL that Kinde generated for my demo requested scope=public_repo and nothing more, and GitHub's screen showed "Public repositories" only.

Third, Kinde returns the user to your callback with the session handle in the query string:

/api/connect/github/callback?session_id=dc5e2e23f19749f08c1f73b4ee7af8c2

Store that handle against the user. My callback route marks the connection linked and writes an audit row. No token arrives in this request, and the route stores none.

await convex.mutation(api.gateway.markLinked, {
  secret,
  userId: operator.userId,
  kindeConnectionId: kindeManagementEnv().KINDE_GITHUB_CONNECTED_APP_KEY,
  kindeSessionId: sessionId,
  grantedScopes: ["public_repo"],
});

Be clear with yourself about what this handle is. It identifies a connected app session at Kinde. It carries no GitHub access on its own. Somebody who steals it still needs your Kinde machine to machine credentials to turn it into a token, and Kinde stops honouring it the moment you revoke.

How to Build the Broker

The broker is one function. Every GitHub action in the app passes through it. Nothing else in the app calls GitHub.

Start with the credential step, because that is where the two modes differ.

export async function acquireConnectedAppToken(
  source: ConnectedAppSource,
): Promise<CredentialResult> {
  // Fail closed before Kinde is even asked.
  if (source.status !== "linked") {
    return {
      ok: false,
      status: 0,
      reason: `The GitHub connection is ${source.status}. Refusing to act.`,
    };
  }
  if (!source.kindeSessionId) {
    return {
      ok: false,
      status: 0,
      reason: "No connected app session for this user. Refusing to act.",
    };
  }

  const result = await fetchConnectedAppToken(source.kindeSessionId);
  if (!result.ok) {
    // INVALID_SESSION lands here after a revocation.
    return { ok: false, status: result.status, reason: result.reason };
  }

  return { ok: true, token: result.token.accessToken, source: CONNECTED_APP };
}

Now the stored key path. Read what it does not do.

export function acquireStoredKeyToken(): CredentialResult {
  try {
    return {
      ok: true,
      token: storedKeyEnv().GITHUB_STORED_TOKEN,
      source: STORED_KEY,
    };
  } catch {
    return {
      ok: false,
      status: 0,
      reason:
        "STORAGE_MODE is stored-key but GITHUB_STORED_TOKEN is not set. Refusing to act.",
    };
  }
}

This function never contacts Kinde. It never checks the connection status. That is not an oversight. It is the bug that the demo reproduces on purpose. Because nothing in this path asks Kinde, revocation at Kinde cannot reach it.

The broker binds the token into a call function and hands the call function to the action handler.

export function createGitHubClient(token: string): GitHubCall {
  return async function call<T>(path, options = {}) {
    const response = await fetch(url, {
      method: options.method ?? "GET",
      headers: {
        authorization: `Bearer ${token}`,
        accept: "application/vnd.github+json",
        "x-github-api-version": "2022-11-28",
        "user-agent": USER_AGENT,
      },
      cache: "no-store",
      signal: AbortSignal.timeout(GITHUB_TIMEOUT_MS),
    });
    // ...
  };
}

The token lives inside the closure. A handler receives call, not the token. A handler therefore cannot read, copy, log, or store a credential, because it never holds one.

The broker then runs the action and writes an audit row.

// 4. Act. The handler is given a bound caller, never the token.
const target = gitHubTargetEnv();
const call = createGitHubClient(credential.token);
const handler = getHandler(action.id);

const result = await handler(call, input as never, {
  owner: target.GITHUB_TARGET_OWNER,
  repo: target.GITHUB_TARGET_REPO,
});

await audit("action.invoked", "allowed", result.summary, user._id);
// The token goes out of scope here. Nothing wrote it anywhere.

Here is the whole path, including every way it can end.

Every path an action can take through the broker. An unregistered name never reaches a credential. A missing or refused token ends in token.refused. A GitHub error ends in a recorded failure. A success ends with the token going out of scope. Each ending writes an audit row, so no action leaves the broker without a record.

Every path ends in an audit row. The row carries a correlation id, so the run timeline and the audit trail join on one value.

How to Model the Data

The store has six tables: users, connections, actions, runs, run events, and the audit log.

The connections table records state only. It holds the status, the GitHub login, the granted scopes, the linked and revoked timestamps, and a counter of how many tokens Kinde has supplied. It holds no credential in either mode.

Writing that rule in a comment would prove nothing, so I wrote a test that walks the runtime validator tree of every table and fails on any field name that looks like a credential.

it("holds no field that could store a GitHub credential", () => {
  const forbidden =
    /(access_?token|refresh_?token|^token$|bearer|secret|credential|client_?secret|api_?key|private_?key)/i;

  for (const [table, definition] of Object.entries(tables)) {
    const names = [...fieldNames(definition.validator.json)];
    expect(names.length).toBeGreaterThan(0);
    const offenders = names.filter((name) => forbidden.test(name));
    expect(
      offenders,
      `table "${table}" declares credential-shaped fields: ${offenders.join(", ")}`,
    ).toEqual([]);
  }
});

The test reads the compiled schema rather than the source text. Somebody who adds a token column later breaks the build, and they break it for a clear reason.

The audit log carries the outcome of every decision. Six event types cover the whole surface: connection.linked, connection.revoked, token.brokered, token.refused, action.invoked, and action.refused. Three outcomes describe what happened: allowed, refused, and failed.

Every row also carries the storage mode. That single field is what lets the audit trail show both modes in one table and make the difference obvious.

Everything that changes state is an internal mutation. The browser cannot forge an audit row, and it cannot flip a connection to linked. The server reaches those functions through one gateway that requires a shared secret, and the browser does not have that secret.

One detail is worth copying. The broker writes a token row for each action rather than one row per run. A three action run produces three token.brokered rows:

18:25:38  token.brokered  allowed  connected-app  Fetched a token from Kinde for one action. Not stored.
18:25:42  action.invoked  allowed  connected-app  Read 3 open issue(s) ...
18:25:47  token.brokered  allowed  connected-app  Fetched a token from Kinde for one action. Not stored.
18:25:50  action.invoked  allowed  connected-app  Read issue #1 ...
18:25:59  token.brokered  allowed  connected-app  Fetched a token from Kinde for one action. Not stored.
18:26:01  action.invoked  allowed  connected-app  Commented on issue #1 as sholajegede.

Fetch per action stops being a claim in your README. It becomes a pattern in your data that anybody can count.

How to Decide the Storage Mode

The mode must not be something a request can influence. I put the whole rule in one pure function.

export function resolveStorageMode(raw: string | undefined | null): StorageMode {
  return raw === STORED_KEY ? STORED_KEY : CONNECTED_APP;
}

Only the exact string stored-key selects the unsafe mode. Every other value resolves to connected-app. That covers unset, empty, STORED-KEY, stored_key, a trailing space, and anything an attacker injects.

The only caller reads process.env.STORAGE_MODE and nothing else. No request, header, cookie, or query parameter appears anywhere in that path. The reader also throws if it ever runs with window defined, so a client component cannot pull it into the browser bundle.

How to Build the Agent Loop

The agent proposes actions. The broker performs them. The agent holds no credential.

The tool list comes from the action registry, so the agent cannot be offered a capability that the registry does not define.

export function agentTools(): AgentTool[] {
  return listActions().map((action) => ({
    type: "function",
    name: action.id,
    description: describe(action),
    parameters: toJsonSchema(action.input),
    strict: false,
  }));
}

The loop reads the model id from configuration. It never appears as a literal in the source.

const response = await openai.responses.create({
  model,
  input: conversation,
  tools,
});

const requested = response.output.filter(
  (item): item is Extract<typeof item, { type: "function_call" }> =>
    item.type === "function_call",
);

for (const toolCall of requested) {
  await emit("agent.tool_requested", `Requested ${toolCall.name}.`);

  // The one and only path to GitHub.
  const outcome = await brokerAction({
    actionId: toolCall.name,
    input,
    kindeUserId: request.kindeUserId,
    correlationId,
    runId,
  });
  // ...
}

I use the Responses API rather than chat completions for a measured reason. The model family I tested rejects function tools on /v1/chat/completions unless reasoning is switched off:

chat  effort=none     OK tool_calls=1
chat  effort=low      ERR 400 Function tools with reasoning_effort are not supported
responses api         OK  types=function_call

Staying on chat completions would have meant disabling the model's reasoning to get tools. The Responses API carries both.

When the broker refuses for lack of a credential, the agent stops. It does not retry, and it does not reach for another tool.

// Fail closed. A credential refusal will not resolve itself, so the
// run stops here rather than looping or reaching for another route.
if (outcome.status === "refused" && outcome.refusal === "credential") {
  halt = true;
  status = "refused";
  finalMessage = `Stopped: ${outcome.reason}`;
}

The broker returns a typed refusal kind so this check is structural, not a string match.

Refusal kind	Meaning	Caller behaviour
`credential`	No token now, and none coming	Stop
`input`	The arguments failed validation	Correct and retry
`unknown-action`	The name is not in the registry	Stop
`no-user`	No such user	Stop

I drew that line on purpose. An input refusal is the model's own mistake, and letting it fix the arguments is self correction rather than a blind retry of a denied action.

Four structural tests hold the agent honest. The agent module never imports the GitHub client, never names the GitHub API host, never reads a credential from the environment, and contains no storage mode branch at all. A test walks the source tree and fails the build if any of these change.

Two Revocation Levers, and One Is a Decoy

The Kinde Management API exposes two things that sound like a kill switch. Only one is.

I tested both against the live service.

Lever A: DELETE /api/v1/users/{user_id}/sessions.

DELETE sessions: HTTP 200 in 420ms
{"code":"USER_SESSIONS_INVALIDATED","message":"User sessions successfully invalidated"}

Then I polled the token endpoint every two seconds:

  +  2303ms  STILL ISSUING  HTTP 200  (fingerprint 7ea1fb9d9fa2)
  ...
  + 60011ms  STILL ISSUING  HTTP 200  (fingerprint 7ea1fb9d9fa2)

Kinde was still issuing tokens 60000ms after revocation, across 23 attempts.

Twenty three consecutive requests over a full minute. Every one returned HTTP 200. Every one returned the same token.

The endpoint reports success and stops nothing. The reason shows in the data. Before the test, GET /users/{id}/sessions returned "sessions": [], an empty list, while the connection was live and issuing tokens. A connected app session is not a user session. They are separate objects, and the user sessions endpoint does not reach the connected app.

If you build a kill switch on this endpoint, it returns HTTP 200, it logs success, and it cuts off nothing.

Lever B: POST /api/v1/connected_apps/revoke.

POST /api/v1/connected_apps/revoke   HTTP 200 in 151ms
{"message":"token revoke successful","code":"REVOKE_SUCCESSFUL"}

  +     0ms  REFUSED  HTTP 400  INVALID_SESSION: Error encountered while
                                retrieving tokens for connected app

The very next request failed. Not eventually. The first call after the revoke returned HTTP 400 with INVALID_SESSION.

This is the kill switch. I pinned it in code and in a test.

export async function revokeConnection(
  request: RevocationRequest,
): Promise<RevocationResult> {
  // ...
  const result = await revokeConnectedAppSession(connection.kindeSessionId);

  if (!result.ok) {
    const reason = `Kinde refused the revocation (HTTP ${result.status}).`;
    await audit("failed", reason, user._id);
    return { status: "failed", reason, httpStatus: result.status };
  }

  // Record the revocation locally too. The broker refuses on its own
  // authority once the connection reads `revoked`, rather than depending on
  // Kinde's refusal alone.
  await convex.mutation(api.gateway.markRevoked, { secret, userId: user._id });

The test asserts that the revoke module calls revokeConnectedAppSession, never mentions deleteUserSessions, and that the only file in the whole tree that calls the user sessions endpoint is the probe script where I measured it. A later edit that simplifies onto the decoy fails the build.

The Central Finding: Revocation Reaches the Connection, Not the Token

Now the result that matters most, and the one that weakens the easy version of this story.

After connected_apps/revoke succeeds and Kinde refuses to broker, does a token that Kinde already handed out still work at GitHub?

I measured it. I brokered a token, held it, revoked the connection, confirmed Kinde refused, then kept calling GitHub with the token I already had.

Step 1  broker a token and hold it        gho_ fingerprint ffda970320cd
Step 2  held token works before revoke    GET /user HTTP 200
Step 3  POST connected_apps/revoke        HTTP 200 in 177ms
Step 4  can Kinde still broker a token?   REFUSED HTTP 400 INVALID_SESSION
Step 5  does the HELD token still work?
          +   397ms  GET /user HTTP 200  STILL VALID
          +  6115ms  GET /user HTTP 200  STILL VALID
          + 13055ms  GET /user HTTP 200  STILL VALID
          + 19051ms  GET /user HTTP 200  STILL VALID
          + 30235ms  GET /user HTTP 200  STILL VALID
          + 37646ms  GET /user HTTP 200  STILL VALID

Kinde refused to broker in the same second that GitHub was still serving the token it had already issued. Six calls over 37.6 seconds. Every one returned HTTP 200.

Revocation at Kinde does not reach into GitHub. It stops Kinde from brokering and from refreshing. It does not cancel an outstanding token.

So the instant cutoff in connected-app mode is real, but the reason is not the one you might assume. The cutoff is instant because the broker keeps nothing between actions. The next action must ask Kinde. Kinde refuses. The action stops. There is no credential left inside the app for anything to keep using.

This has a direct consequence for your code. Do not cache the brokered token, not even for a few seconds. A cache is a window of access that survives the kill switch. In this design, fetch per action is a security boundary rather than a performance choice.

One revocation, and what each mode does next. Kinde stops brokering at once, so the connected-app broker gets INVALID_SESSION on its next request and the agent stops. The stored-key app never asks Kinde, so it keeps acting and its comment lands after the revocation. GitHub is the third party in this picture, and it honours the token it already issued until that token expires.

The Two Modes, Side by Side

I ran both modes against the same revocation and recorded what happened.

The revocation landed at 18:41:52 UTC. Here is the audit log across both modes, ordered by time.

time      mode           event              outcome   detail
18:41:31  connected-app  action.invoked     allowed   Commented on issue #3 as sholajegede.
18:41:52  connected-app  connection.revoked allowed   Connection revoked at Kinde.
18:42:20  connected-app  token.refused      refused   The GitHub connection is revoked.
18:45:52  connected-app  token.refused      refused   INVALID_SESSION: ...
18:46:17  stored-key     token.brokered     allowed   Used the token the app holds. Kinde was not consulted.
18:46:32  stored-key     action.invoked     allowed   Commented on issue #2 as sholajegede.

The stored-key comment is real. It has a GitHub comment id and a created_at of 2026-08-14T18:46:31Z, which is 4 minutes and 39 seconds after the revocation. An agent that the operator believed they had cut off carried on writing to the repository.

Every stored-key audit row says why, in the row itself: "Used the token the app holds. Kinde was not consulted."

The connected-app side shows two independent layers. The first refusal came from the local flag, because the broker marks the connection revoked and then refuses on its own authority. To check that Kinde also refuses, I forced the local flag back to linked while the Kinde session stayed revoked, and ran again:

3  broker.refused   INVALID_SESSION: Error encountered while retrieving tokens for connected app

INVALID_SESSION straight from Kinde, with the app's own bookkeeping claiming the connection was fine. Either layer stops the agent on its own. The cutoff does not depend on your local state being correct.

After the revoke. The GitHub connection is cut. The operator stays signed in to the app.

That last point matters and I tested it live. Revoking the GitHub connection does not end the app session. I clicked Revoke, then reloaded the page in the browser. The header still read the operator's name. The connection read revoked. The user keeps their session with your product and loses only the third party connection they chose to cut.

Ask the agent for something after the revoke, and the run stops on the first tool call.

The agent asks for one tool, the broker refuses, and the run halts. No retry, and no second route.

How to Keep Least Privilege

The registry defines four actions and nothing else. The agent cannot name its way to a capability that does not exist, because getAction() throws on anything unregistered.

Action	Effect	Acts as the user	Scope
`read_issues`	Reads	No	`public_repo`
`read_issue`	Reads	No	`public_repo`
`comment_issue`	Writes	Yes	`public_repo`
`open_pr`	Writes	Yes	`public_repo`

I first wrote the registry with public_repo and read:user, so the audit trail could record who acted. Then I hit a real constraint. Kinde's GitHub scope picker does not offer read:user. The connection grants public_repo and nothing else.

A registry that claims a scope the connection cannot hold is simply wrong, so I dropped it. The identity still gets recorded, because GitHub echoes the acting account back in the responses the actions already make.

export function actingUserFrom(payload: unknown): ActingUser | null {
  if (!payload || typeof payload !== "object") return null;
  const body = payload as MaybeUser;

  // A nested `user` is the shape returned by comments and pull requests.
  const candidate =
    typeof body.user?.login === "string"
      ? body.user
      : typeof body.login === "string"
        ? body
        : null;

  if (!candidate || typeof candidate.login !== "string") return null;
  return {
    login: candidate.login,
    id: typeof candidate.id === "number" ? candidate.id : null,
  };
}

A comment response carries { user: { login } }. GET /user needs no scope at all. When a response carries no identity, the function returns null and the app audits the action without a login. The app does not widen the connection to get one.

The broker also compares the scopes GitHub reports against what the registry claims, and records both directions. In connected-app mode the audit row reads:

Read 2 open issue(s) in sholajegede/connected-apps-github-sandbox.
Least privilege holds: the credential carries exactly what the action needs.

In stored-key mode the same code reads:

Over-privileged: the credential also carries gist, read:org, workflow,
which this action does not need.

That is the second failure of the stored key, visible in the audit trail. The stored key is not only impossible to revoke centrally. It is also far broader than the work needs, because a token that a developer already had tends to be the token that gets used.

What the Operator Controls

The demo gives the user the controls rather than a video of somebody else using them.

The operator signs in with their own session. They connect GitHub themselves. They write the task in free text, so nothing is a canned script. They watch each step arrive in a live timeline, and they cut the connection when they choose.

The console shows five counts, and each one answers a question a reader will ask.

Count	What it answers
Tokens supplied	How often the broker asked Kinde
GitHub tokens that the app keeps	Zero in `connected-app`, one in `stored-key`
Actions done on GitHub	What really happened in the repository
Actions refused	How often the broker said no
Actions after revocation	Zero in `connected-app`, above zero in `stored-key`

The second count is not derived from the data. It is a property of the deployment, decided on the server and passed down for display. The console shows it and can never set it. The same holds for the storage mode badge in the header.

The last count is the one to watch during a demo. It shows a dash until the operator revokes. After that it turns green at zero in connected-app, and red at a real number in stored-key.

How to Harden It for Production

A demo becomes trustworthy when the failure paths behave.

Give every outbound call a deadline. A hang is an outage from the caller's point of view, and an action that waits forever can never fail closed. I set 8 seconds on Kinde and 12 seconds on GitHub. I tested the Kinde deadline by pointing the issuer URL at an unroutable address:

outcome        refused
reason         Kinde did not answer within 8000ms.
audit:  token.refused  refused  connected-app  Kinde did not answer within 8000ms.

Before I added this, that path threw past the broker with no audit row at all. The credential step sat outside any try block. The fail closed claim was false for exactly the case that needed it most. Finding this is the reason to test outages rather than reason about them.

Refuse an action you cannot record. The broker writes the token.brokered row before it calls GitHub. If that write fails, the action does not happen.

if (!recorded) {
  const reason =
    "The audit trail is unavailable. Refusing to act rather than act without a record.";
  return {
    status: "refused",
    actionId: action.id,
    storageMode: mode,
    reason,
    refusal: "credential",
  };
}

The row also goes to an append only local queue first, so the attempt stays recoverable rather than lost.

Never log a token. The broker redacts every GitHub token shape from any text it records, and reports tokens by fingerprint.

const CREDENTIAL_PATTERN =
  /gh[pousr]_[A-Za-z0-9]{16,}|github_pat_[A-Za-z0-9_]{16,}/g;

export function redactCredentials(text: string): string {
  return text.replace(CREDENTIAL_PATTERN, "[redacted]");
}

A fingerprint is enough to answer real questions without exposing anything. It is a truncated SHA-256 of the token. Two fetches can be compared for equality, and the value itself never appears.

Prove the claim at run time. A comment in the source is not evidence. This script runs a real agent task, then looks for a credential everywhere the app could have put one: every row of every table, everything the code printed, the agent's own transcript, and the process environment.

Step 2 — scan every row of every table in the store
  actions      4 row(s) scanned
  auditLog     56 row(s) scanned
  connections  1 row(s) scanned
  runEvents    78 row(s) scanned
  runs         10 row(s) scanned
  users        1 row(s) scanned

Verdict
CLEAN — connected-app mode.
No GitHub credential is in the store, in the broker's output, or in
the environment.

The same script run in stored-key mode reports FOUND GitHub OAuth token (gho_) in env.GITHUB_STORED_TOKEN, and stays clean everywhere else. The same assertion, the same code, and a different answer per mode. That contrast is the point.

One detail makes this proof meaningful. The stored key lives in a separate file that the app loads only when the mode is stored-key. If it sat in the main environment file, it would be in process.env on every run, and the environment scan would be worthless.

How to Prove the Tests Can Fail

A green test suite means nothing until you have seen it go red.

The end to end script walks the whole story in one pass: clean slate, a connected-app run that comments on a real issue, a revoke, a refused run, a stored-key run after the same revoke, and a reconciliation across both modes. It asserts real outcomes and exits non-zero on any failure.

Each acting step runs in a child process with its own STORAGE_MODE. The mode is a deployment decision, and a script that flipped it in its own process would quietly contradict the property it tests.

Green looks like this:

GREEN — every assertion held. 69.3s

Then I inverted the load bearing check. Step 4 asserts that the stored-key agent still acts after the revocation. I changed status === "succeeded" to status !== "succeeded", so the assertion demanded the opposite of the measured truth:

RED — 1 assertion(s) failed. 54.3s

  4. stored-key — the same revocation, and the agent acts anyway
    the agent still acts after the connection was revoked
    status=succeeded — Reviewed the open issues and commented on issue #1 ...

EXIT CODE = 1

Exit code 1. The report names the step, the assertion, and the observed value. Every other assertion still passed, so the failure stayed isolated rather than cascading. I then restored the file and verified the restoration by hash rather than by eye, so the green result stands for the exact file that ships.

Honest Limitations

Read these as written. Do not soften them.

Revocation does not cancel a token that GitHub already issued. Kinde stops at once. A token already handed out stays valid at GitHub until it expires. I measured it still valid 37.6 seconds after the revoke, and nothing in the revoke path touches GitHub. To cancel the outstanding token, somebody revokes it at GitHub. That control belongs to GitHub, and Kinde does not have it. It sits on the OAuth app's own settings page.

The token life is real, and it depends on one setting. The brokered token is an opaque 40 character gho_ token. It is not a JWT, so there is nothing to decode. Kinde reported an expiry of 2026-08-15T02:09:08.871+01:00, which was 28761 seconds ahead, or 7.99 hours. That matches GitHub's documented eight hour user access token (GitHub docs). GitHub issues it that way because the OAuth app has Token expiration switched on, under Optional features. GitHub's own wording for that setting states the eight hours, and states something else worth reading twice.

"Existing tokens are not affected." GitHub is describing what happens when you toggle the setting, but the same principle runs through this whole article: a token already issued keeps its own life. Turn the setting off and the token never expires at all, and this whole argument gets weaker.

Kinde returns the same token again, not a new one. Two requests inside the same eight hour window returned the identical token, confirmed by fingerprint:

fingerprint    7ea1fb9d9fa2
second fetch   fingerprint 7ea1fb9d9fa2 — SAME token replayed

So the correct claim is fetch per action, store never. The claim is not that each action gets a fresh token. The security property comes from the app holding nothing between actions.

Kinde holds the refresh token. The app never receives one. The evidence is positive rather than merely absent. An eight hour token can only be renewed with a refresh token, so one exists, and it never reaches my app. My app cannot renew access on its own.

The read queries in my demo use an id, not access control. Every write needs a server secret that the browser does not have, so the browser cannot forge a record or change the mode. The read queries trust the id they receive. That is enough for a single operator demo. It is not access control, and I would wire the identity provider into the data layer before shipping this to many users.

What This Design Costs

Be honest about the trade you make.

You add a network call to every action. The broker asks Kinde before it calls GitHub. In my measurements a successful token request took 439ms and 468ms. For an agent that makes three tool calls, that is about 1.4 extra seconds per run. You cannot buy that back with a cache without giving up the property that makes revocation work.

You add a dependency to the hot path. If Kinde is down, your agent stops. I treat that as correct behaviour rather than a fault, because the alternative is to keep a credential that nobody can take back. Your deadline and your refusal message then matter, since users will see them.

You need somewhere to put the session handle. The app stores no token, but it does store the handle and the connection state. That is a small table, and it holds nothing that works on its own.

You give up one convenience. With a stored key, any part of your codebase can call GitHub. With a broker, everything routes through one function. That is the point, and it does mean a small amount of plumbing when you add an action.

The trade buys you three things. The user can cut the agent off centrally. Your database holds no live third party access. And your audit trail records every decision, including the ones where the broker said no.

What You've Built, and Where to Take It

You've built a complete path: a user signs in, connects GitHub through Kinde, and an OpenAI agent acts in their account while the app stores no GitHub token. One revocation cuts the agent off within a single action, and the operator stays signed in.

The shape transfers to any third party API, not only GitHub. Eight things are worth carrying over.

Put one function between your code and the third party. Everything goes through it. Add a test that walks your source tree and asserts exactly one module names the API host. Structure beats discipline, because the test fails the build and a convention does not.

Give handlers a bound caller, not the credential. If a function never holds a token, it cannot leak one. This removes a whole class of review question.

Decide the mode on the server, from deployment configuration. Make the unsafe value an exact match and let everything else fall to the safe one. Then a typo fails safe.

Make a refusal a normal result. The broker returns a typed outcome instead of throwing. Callers handle refusal as data, and the audit trail records it like anything else.

Distinguish refusal kinds. A missing credential means stop. A bad argument means the caller can fix it. Blurring the two either produces retry loops or blocks legitimate self correction.

Fetch per action and keep nothing. This is the part that makes central revocation bite. If you cache, you reopen the window.

Test the outage, do not reason about it. Point the issuer URL somewhere unroutable and see what your code does. I found a real bug that way.

Write down the limitation that hurts. The demo is stronger when it says plainly that revocation does not reach an already issued token. A reader who discovers that on their own stops trusting everything else you wrote.

Wrapping Up

The demo makes one argument, and it holds up under measurement. An agent can act inside a user's real GitHub account without your app ever storing a GitHub token, and the user can cut it off centrally.

The part worth remembering is the reason it works. Revoking at Kinde does not reach into GitHub and cancel a token that GitHub already issued, and I measured that token still working 37.6 seconds later. The cutoff is instant because the broker keeps nothing between actions, so no credential is left inside the app for the agent to keep using. Cache the token for even a few seconds and you give that property away.

Take the honest limitation with you as well. To cancel an outstanding token too, the user revokes the authorization on GitHub, and that control belongs to GitHub rather than to Kinde.

Resources

Source code

The complete demo is on GitHub under the MIT licence, so you can run it, read it, and reshape it. The README carries the same measured limitations as this article.

Core documentation

Kinde: add connected apps: the auth_url, token, and revoke flow this build uses
GitHub: refreshing user access tokens: the eight hour token life and the refresh token
GitHub: token expiration and revocation: what expires, and what a user can revoke themselves
GitHub: authorizing OAuth apps: the consent screen the user sees

Context for the problem

OpenAI: company knowledge in ChatGPT: connectors for Slack, SharePoint, Google Drive, and GitHub
OpenAI: workspace agents in ChatGPT: agents that run workflows across connected apps

Build tooling

Next.js App Router: the server actions and route handlers the console uses
Convex: the data store behind the audit trail and the live timeline
OpenAI Responses API: the tool-calling endpoint the agent loop uses
Vitest: the runner behind the structural tests

How to Require Step-Up Auth Before a Claude Agent's Destructive Actions With Kinde

Shola Jegede — Thu, 13 Aug 2026 23:33:34 +0000

An agent deleted a document at 22:45. The person who authorised it had signed in at 16:57.

Nobody was at the keyboard. No token had expired. No rule was broken. The access token was valid for another nineteen hours, and it said exactly what it said six hours earlier: this person is allowed to delete documents. The agent read that as permission and acted.

The token was right about identity. It was silent about presence.

In this tutorial, you'll build a records console where a Claude agent runs read-only tools freely, and every destructive action, delete, refund, and deploy, stops at a server-side check. That check asks one question: how long ago did a human actually authenticate? If the answer is too long ago, the server holds the action until the person signs in again. The build uses the Anthropic API for the agent, Kinde for identity, and Convex for state and the audit trail.

By the end, you'll have built:

A tool registry that separates safe tools from destructive ones, and gives every destructive tool its own freshness window in seconds
One enforcement seam that every tool call passes through, with JWKS verification and RS256 pinned
A freshness check built on the OpenID Connect auth_time claim, read from the ID token and verified server-side at the moment of the call
A Claude tool-calling loop that carries the signed-in person's delegated identity and holds no credential of its own
A complete step-up round trip: the seam holds the action, the person re-authenticates, the agent re-presents the same call, and the seam releases it
An audit trail that keeps recording when its own storage is unreachable
A blanket mode that reproduces the failure on the same code, so you can watch the same task slip through

Every number, reason code, and token claim in this tutorial is measured from that running system.

Why blanket consent fails
The demo and its three services
The freshness primitive: auth_time
Why iat and exp cannot stand in
Building the seam
The tool registry
The Claude agent loop
Good manners are not enforcement
The round trip
The two negative cases
Blanket against step-up
Production hardening
Honest limitations
Proving the check can fail
How to map this to your app
Resources

Why blanket consent fails

The common answer to agent risk is to ask the person to approve each action. Anthropic measured what that produces. In How we contain Claude across products, their engineering team reports:

Our telemetry showed users approved roughly 93% of permission prompts.

And the mechanism behind it:

The more approvals a user sees, the less attention they pay to each, becoming over time much less diligent in their supervision.

A prompt stream trains people to clear prompts. The approval stops being a decision and becomes a reflex, and the one prompt that mattered gets cleared with the rest.

Anthropic's own answer was containment, not step-up authentication. They moved the defence into the environment: ephemeral containers, sandboxes, and virtual machines that limit what an agent can reach. They also built an auto mode that approves safer actions automatically, so the prompt stream gets shorter.

Step-up authentication answers a different question. Containment limits the blast radius. Step-up asks whether the person is still here, right now, for this one irreversible action. The two are complementary:

Containment handles the many. Most actions become safe by construction and need no prompt.
Step-up handles the few. It applies high friction to a small set of irreversible actions.

One design rule follows. A read-only tool must never produce a prompt. Every unnecessary prompt spends the attention you need for the prompt that matters.

The demo and its three services

The build is an operations console. An agent works on records: invoices, releases, and documents. It has six tools.

Tool	Destructive	Freshness window
`list_records`	no	none
`get_record`	no	none
`summarize_records`	no	none
`delete_record`	yes	300s
`refund_payment`	yes	120s
`deploy_release`	yes	120s

Three services do the work. The Anthropic API runs the agent, with the model id read from an environment variable. Kinde is the identity provider, running OpenID Connect with authorization code and PKCE. Convex stores the registry, the records, the run timeline, and the audit trail.

The windows differ by damage. A document delete gets 300 seconds because a backup can restore it. A refund and a production deploy get 120 seconds, because one moves money and the other reaches customers the moment it lands.

The freshness primitive: auth_time

OpenID Connect defines auth_time: the moment the person last authenticated interactively, in seconds since the epoch. It is the only standard claim that tracks a human rather than a token.

Two real tokens from the same sign-in, decoded:

// ID token                              // Access token
{                                        {
  "sub": "kp_79a5daf465584…",              "sub": "kp_79a5daf465584…",
  "auth_time": 1786553833,   // <-- here   // no auth_time
  "iat": 1786553834,                       "iat": 1786553834,
  "exp": 1786557434,         // +1h        "exp": 1786640234,   // +24h
  "aud": ["d4ef3c61eb19…"],                "aud": [],
  "iss": "https://devrelstudio.kinde.com"  "iss": "https://devrelstudio.kinde.com"
}                                        }

The differences decide the whole design.

	ID token	Access token
`auth_time`	present	absent
`aud`	client id	`[]`
Lifetime	1 hour	24 hours
`amr` / `acr`	absent	absent

The access token is the credential a client presents at an API boundary. It carries no evidence of human presence. The ID token carries that evidence and expires twenty-three hours sooner.

The access token lives 24 hours. The evidence of human presence lives 1 hour. Blanket consent ignores that gap. The freshness check closes it.

For twenty-three of those twenty-four hours, an agent holds a valid credential and the system has no fresh evidence that anyone is there.

The discovery document does not mention it

Kinde publishes a discovery document. Its claims list is short:

{
  "claims_supported": ["aud", "exp", "iat", "iss", "sub"],
  "code_challenge_methods_supported": ["S256"]
}

auth_time does not appear. The ID token carries it anyway.

This is spec-consistent. OIDC Core defines auth_time as an ID Token claim, and claims_supported is informational rather than exhaustive. The practical lesson: read a real token before you design against a provider.

The seam therefore cannot prove freshness from the access token. It verifies two tokens and binds them together.

Why iat and exp cannot stand in

iat and exp look like freshness. They are not. Both move whenever a token is minted, and a token can be minted with no human present.

The build tested this against the live provider. First, a refresh while the ID token was still valid:

{
  "authTimeBefore": 1786552841,
  "authTimeAfter": 1786552841,
  "authTimeMoved": false,
  "idTokenChanged": false,
  "accessTokenChanged": false,
  "verdict": "INCONCLUSIVE — the refresh returned the same ID token"
}

Kinde returned the byte-identical token. Nothing was minted, so nothing was proved. Reporting that as a pass would be a false result, so the check reports it as inconclusive.

The decisive test needs an expired ID token:

{
  "authTimeBefore": 1786553833,
  "authTimeAfter": 1786553833,
  "authTimeMoved": false,
  "idTokenChanged": true,
  "idTokenWasExpired": true,
  "issuedAtMoved": true,
  "tokenIdMoved": true,
  "verdict": "auth_time held steady across a newly minted ID token"
}

The provider minted a genuinely new ID token:

	Before	After
`jti`	`1dd46212-757e-…`	`09ba30b7-e842-…`
`iat`	16:57:14Z	18:02:30Z
`exp`	17:57:14Z	19:02:30Z
`auth_time`	`1786553833`	`1786553833`

The new token reports an authentication that happened 3,936 seconds earlier, about 65 minutes. A token minted seconds ago correctly describes a human who authenticated over an hour before.

That number settles the design. auth_time tracks human authentication, so freshness builds on it. iat and exp track token issuance, so a check that asks "was this token issued recently?" passes with nobody there.

The decision function never reads iat or exp. The comment says why, so nobody adds them later:

/**
 * `iat` and `exp` are deliberately absent from this function. Both move when
 * a token is refreshed while no human is present, so neither can stand in for
 * human presence. Only `auth_time` survives the minting of a new token.
 */

Building the seam

The seam is one function. Every agent tool call passes through it. There is no second path to a tool.

Every path ends in an audit row. There is no early return that skips the trail.

Step 1: verify the access token

The build pins RS256 and reads keys from the provider's JWKS endpoint:

const result = await jwtVerify(token, jwksFor(metadata.jwks_uri), {
  algorithms: ALLOWED_ALGORITHMS,      // ["RS256"]
  issuer: metadata.issuer,
  clockTolerance: 5,
});

// `jwtVerify` already refuses any algorithm outside the allow-list. This
// re-reads the header so the pin is visible at the call site and survives
// a future change to how the options are built.
if (result.protectedHeader.alg !== "RS256") {
  throw new TokenVerificationError(
    "algorithm_not_allowed",
    `Token algorithm ${result.protectedHeader.alg} is not RS256.`,
  );
}

The pin appears twice on purpose. The allow-list enforces it. The second check keeps it visible, so a later edit cannot drop it silently.

The verifier refuses malformed claims rather than ignoring them:

No sub, no exp, or no iat: refused.
An auth_time that is not a positive finite number: refused as malformed, not treated as absent.
An auth_time more than 60 seconds in the future: refused. A future auth_time would make every comparison pass.

Step 2: read the registry, and refuse when you cannot

An unknown tool is denied unknown_tool. A registry that cannot be read is denied registry_unavailable. Without the registry the seam cannot know whether a tool is destructive or what window it carries, and guessing either way is worse than stopping.

Step 3: read freshness from the ID token, bound by subject

if (context.idToken !== undefined && context.idToken !== "") {
  try {
    const idClaims = await verifyIdToken(context.idToken);

    // Binding. Without this, an ID token from any session of any person
    // could vouch for freshness on someone else's access token.
    if (idClaims.sub !== accessClaims.sub) {
      return await finish("deny", "subject_mismatch");
    }

    observed.authTime = idClaims.auth_time;
    observed.amr = idClaims.amr;
  } catch {
    // The ID token did not verify. Most often it expired.
    return await finish("challenge", "id_token_invalid", { ... });
  }
} else if (mode === "step-up") {
  return await finish("challenge", "id_token_missing", { ... });
}

The access token answers "who is this". The ID token answers "when did a human last authenticate". Without the binding, a valid ID token from any session could vouch for freshness on somebody else's access token.

The build separates two refusals that both fail closed:

id_token_invalid: the freshness evidence did not verify. Most often the ID token expired, because it lives one hour and the access token lives a day. This is the common case for a long-running agent.
auth_time_stale: the ID token verified, but auth_time is older than the tool's window.

An early version reported both as auth_time_missing. That was wrong: the claim was not missing, the token had expired. The trail now says which happened.

Step 4: decide

The decision is a pure function. Tests drive the whole table with no tokens, no network, and no database.

export function decide(input: DecisionInput): DecisionOutput {
  if (!input.destructive) {
    return { decision: "allow", reason: "safe_tool" };
  }

  if (
    input.maxAuthAgeSeconds === undefined ||
    !Number.isFinite(input.maxAuthAgeSeconds) ||
    input.maxAuthAgeSeconds <= 0
  ) {
    // A destructive tool that declares no window cannot be checked. Allowing
    // it would mean "no limit"; that reading is the failure being fixed.
    return { decision: "deny", reason: "registry_defect", ... };
  }

  const authAgeSeconds =
    input.authTime === undefined ? undefined : input.now - input.authTime;

  if (input.approvalMode === "blanket") {
    return { decision: "allow", reason: "blanket_mode_freshness_skipped", ... };
  }

  if (input.authTime === undefined) {
    // Freshness that cannot be proved is not freshness.
    return { decision: "challenge", reason: "auth_time_missing", ... };
  }

  const withinWindow =
    authAgeSeconds! <= input.maxAuthAgeSeconds + input.clockSkewSeconds;

  return withinWindow
    ? { decision: "allow", reason: "fresh_authentication", ... }
    : { decision: "challenge", reason: "auth_time_stale", ... };
}

Three details are worth copying.

A missing window is a defect, not permission. Treating an absent maxAuthAgeSeconds as "no limit" is the failure this design closes. The function denies instead.

Clock skew widens the window, never narrows it. A provider clock slightly ahead of yours must not turn a fresh authentication into a challenge. The build adds a 30-second grace.

Safe tools return before any freshness logic runs. They cannot produce a prompt even by accident.

The challenge response

A held call answers with HTTP 403 and an RFC 9470 challenge:

HTTP/1.1 403 Forbidden
WWW-Authenticate: Bearer error="insufficient_user_authentication",
  error_description="The last human authentication is older than this action
  allows. Re-authenticate to continue.", max_age=120

RFC 9470 defines insufficient_user_authentication for this case: the token is valid, but the authentication behind it does not meet the resource server's requirement. The max_age parameter tells the client what the server needs.

The body carries the same facts plus a re-authentication link:

{
  "tool": "delete_record",
  "decision": "challenge",
  "reason": "auth_time_stale",
  "correlationId": "3ad3c1c9-c0cb-4c9c-a0d7-0e3f79b48ba2",
  "authAgeSeconds": 1800,
  "maxAuthAgeSeconds": 300,
  "error": "insufficient_user_authentication",
  "reauthUrl": "/api/auth/login?max_age=0&prompt=login&stepUp=1&returnTo=%2F"
}

max_age=0 and prompt=login ask the provider for an interactive sign-in. They are a hint to the provider. They are never proof. The seam re-reads auth_time from the presented token on the retry and decides again.

The tool registry

The registry holds the policy. Each tool declares a name, a destructive flag, and a window.

The build enforces an invariant on every write:

if (tool.destructive) {
  if (tool.maxAuthAgeSeconds === undefined) {
    violations.push(`destructive tool "${tool.name}" has no maxAuthAgeSeconds`);
  } else if (tool.maxAuthAgeSeconds <= 0) {
    violations.push(`destructive tool "${tool.name}" has a non-positive window`);
  }
} else if (tool.maxAuthAgeSeconds !== undefined) {
  violations.push(
    `safe tool "${tool.name}" carries maxAuthAgeSeconds; ` +
      `safe tools must not have a freshness window`,
  );
}

Both halves matter. A destructive tool must carry a window, or there is nothing to compare auth_time against. A safe tool must not carry one. That second rule is approval fatigue written as code: a window on a read-only tool would produce a prompt that never needed to exist, so the registry rejects it.

Windows do real work

Two measurements from the build show the windows are not decoration.

At one instant, on one token, two destructive tools got different answers:

Tool	Window	Auth age	Decision
`refund_payment`	120s	203s	challenge
`delete_record`	300s	292s	allow

Same person, same session, seconds apart. The refund was held because moving money has a tighter window. The delete passed because a document delete gets more room.

Then the same tool changed answer with nothing else changing:

Tool	Window	Auth age	Decision
`delete_record`	300s	292s	allow, executed
`delete_record`	300s	399s	challenge

Time alone closed the window. The window is measured at the moment of the call, every call.

The Claude agent loop

The agent holds no credential. It calls the same public endpoint every other client uses, and it forwards the signed-in person's session unchanged:

const response = await fetch(new URL("/api/tools/invoke", appConfig().siteUrl), {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    // The person's session, forwarded unchanged. This is the whole of the
    // agent's authority.
    cookie: cookieHeader,
  },
  body: JSON.stringify({ tool, args: input, correlationId }),
  cache: "no-store",
});

The agent never imports the tool executors. If the seam refuses, the agent is refused.

The tempting shortcut is to give the agent its own API key so it "just works". That shortcut is the failure this design closes. A service account decouples the agent's authority from the person's, and auth_time becomes meaningless.

Three responses, handled differently

allow: run the tool, feed the result back, continue.
challenge: stop. Freeze the conversation. Show the person a re-authentication link.
deny: stop and report. A denial is not something re-authentication fixes, so the agent offers no link.

The challenge path is where agents go wrong. Blind retry turns a step-up challenge into a spin, because only the human can clear it. Tool substitution is worse, because it looks like helpfulness: an agent that answers "delete was refused" by reaching for another mutating tool has routed around the control that just fired.

The build blocks both. The system prompt states the rule:

A destructive tool may be held by the server until the person has authenticated
recently. If that happens you will be told so in the tool result, and the run
will end so the person can re-authenticate. That is a normal outcome, not an
error to work around: do not retry a held call, and do not substitute a
different tool to achieve the same effect.

The loop makes it structural. A challenge returns out of the loop, so the model never gets another turn in which to try something else.

Good manners are not enforcement

Two runs in the build produced the same visible outcome and mean completely different things.

Run one. The prompt asked the agent to find the superseded document and delete it. The dataset had none, because an earlier run had already deleted it. The agent listed the documents, read them, and refused:

I reviewed the documents and did not delete anything, because nothing matches what you asked me to find. Neither document is superseded. Both are status active, and each summary points the other way — DOC-3301 is the sole copy of a working model, and DOC-3302 is still load-bearing for three open action items.

No destructive tool was called. The audit trail shows only safe calls.

Run two. Same prompt, record restored. The agent identified the right document, called delete_record, and the seam refused it with auth_time_stale.

Both runs ended with nothing deleted. Only the second is a security control.

The model choosing not to act is good behaviour. It depends on the prompt, the data, and the model's judgement on the day. It is not a boundary. The server refusing to let it act is enforcement, and it holds regardless of what the model decided or which model you swapped in.

When you test your own system, force the destructive call to reach the boundary. If it never reaches the seam, you have tested nothing about the seam.

The round trip

The full sequence, under one correlationId:

The resume re-presents, it does not re-plan

When the seam holds a call, the build stores the conversation and the exact held call:

await convex().mutation(api.runs.pause, {
  runId: ctx.runId,
  haltedReason: seam.body.reason ?? "step_up_required",
  challengeAuthTime: seam.body.authTime,
  pausedState: {
    messages,
    toolUseId: request.id,
    toolName: request.name,
    toolInput: request.input,
  },
});

On resume, the agent sends that same call to the seam before the model gets another turn:

if (pending !== undefined) {
  const seam = await callSeam(
    pending.toolName,
    pending.toolInput,
    ctx.correlationId,
    ctx.cookieHeader,
  );
  ...
}

This is a security property, not a convenience. If the resume let the model re-plan, a released run would not be the task the person approved. It could become a different, larger action wearing the same correlationId. Same tool name, same input, same tool_use_id, or nothing.

The resume route decides nothing

after(async () => {
  await resumeAgent({ runId, userId: session.subject, cookieHeader, observedAuthTime });
});
return NextResponse.json({ runId, accepted: true }, { status: 202 });

No path through the route releases something the seam would refuse. The auth_time the route reads serves the audit narrative only, and it comes from a verified token, never from the caller.

The two negative cases

A control is only as good as what it refuses. Two refusals carry the whole design.

Case 1: a refresh does not release the action

Before any re-authentication, the build ran a refresh_token grant against the live provider, then retried the held delete.

The refresh minted a new, valid ID token. The previous one had expired, so this was a real minting, not a cached response.

The retry was refused.

The reason code tells the story precisely:

Attempt	Reason	Auth age	Window
Original halt	`id_token_invalid`	—	300s
After refresh	`auth_time_stale`	4409s	300s

The reason moved from id_token_invalid to auth_time_stale. That change is the finding:

The refresh repaired the token's validity. It did not repair its freshness.

A machine-to-machine exchange restored a well-formed, verifiable, unexpired credential. The destructive action stayed held, because no human had been present. This is the property the whole design rests on, measured rather than assumed.

If your freshness check reads exp, this case passes and the action runs.

Case 2: re-authentication does not bless the session

The second refusal was not planned. A genuine interactive re-authentication happened. Then about five minutes of other work passed before the retry fired.

The retry was refused:

Attempt	Reason	Auth age	Window
After real re-auth, delayed	`auth_time_stale`	352s	300s

The person had re-authenticated correctly. The window had closed again.

This rules out a weaker reading of the control: that re-authenticating grants the session a blessing that later calls ride on. It does not. The window is measured at the moment of each call. A correct re-authentication that has since aged out is refused exactly like one that never happened.

The release

The next re-authentication was followed promptly by the retry:

Decision	Reason	Auth age	Window
allow	`fresh_authentication`	50s	300s

The delete executed. The record changed.

Seven audit rows carry that run: three safe calls, three challenges, one release. All share one correlationId, so the story stays readable months later.

Blanket against step-up

The build ships two modes so the failure and the fix run side by side on the same code.

APPROVAL_MODE is read from the deploy environment and from nowhere else:

export function approvalMode(): ApprovalMode {
  const raw = process.env.APPROVAL_MODE;
  if (typeof raw !== "string") return "step-up";
  return raw.trim().toLowerCase() === "blanket" ? "blanket" : "step-up";
}

Three properties are deliberate. The variable has no NEXT_PUBLIC_ prefix, so Next.js never inlines it into the browser bundle. The seam never reads it from a header, query string, cookie, or tool argument, so a caller cannot set it. And only the exact string blanket selects the permissive mode: unset, empty, misspelled, or hostile values all resolve to step-up, so a misconfiguration fails towards enforcement.

Running the same task in each mode, with an authentication about five hours old:

Mode	Tool	Auth age	Decision	Executed
blanket	`refund_payment`	~5h	allow (`blanket_mode_freshness_skipped`)	yes
blanket	`delete_record`	~5h	allow (`blanket_mode_freshness_skipped`)	yes
blanket	`deploy_release`	~5h	allow (`blanket_mode_freshness_skipped`)	yes
step-up	`delete_record`	~5h	challenge (`auth_time_stale`)	no

In blanket mode the agent refunded an invoice, deleted a document, and deployed a release to production. Same tokens, same tools, same code. Only the deploy environment differed.

The number that matters

The headline counter is executedWithoutFreshAuth. It counts destructive calls the seam allowed while the authentication was outside the tool's window, or allowed with no age recorded at all. Counting the missing case as a failure keeps the metric honest.

In step-up mode it stays at 0. In blanket mode it climbs.

One subtlety caught the build's own test. A blanket-mode slip is not automatically an escape. If the person signed in seconds earlier, the check was skipped but nothing escaped a window it would have failed, and the counter correctly stays at zero. The hole only bites when the human is absent. The end-to-end test now ages the session past the window before the blanket task runs, so the escape it measures is real.

Production hardening

Three gaps appeared during the build. Each is easy to ship without noticing.

The audit row that vanished

The first version failed closed when Convex was unreachable. The tool did not run, but no audit row was written. A refusal happened with no record that it had. A control that silently stops recording looks identical to one that stopped working.

The audit write now has three stages:

Stage	Result
Convex, retried with backoff	`recorded`
Local append-only spool file	`spooled`
Neither	`lost`

The write never throws, because the seam needs the outcome to decide what to do next:

if (written.durability === "lost" && decision === "allow") {
  // Neither the store nor the spool took it. Allowing now would mean an
  // action ran with no evidence anywhere that it was permitted.
  effectiveDecision = "deny";
  effectiveReason = "audit_unavailable";
}

The audit outcome can only make a decision stricter, never looser.

Testing this exposed a second hole. With Convex fully unreachable the seam never reached the audit write: the registry lookup threw first, the route returned 500, and no row was written. Same gap, one step earlier. It would have stayed hidden if the outage had only been simulated at the audit call.

Measured with the sink pointed at an unreachable host:

POST /api/tools/invoke → 403
  decision=deny  reason=registry_unavailable

.audit-spool.jsonl:
  deny  registry_unavailable  delete_record  cid=0020d4ca

GET /api/health:
  {"degraded": true, "pendingRows": 1}

After the sink returned, one health probe drained it: {"degraded": false, "pendingRows": 0, "replayedNow": 1}. The refusal happened, was recorded while the store was down, showed as a backlog, and landed in the store on recovery.

The success that reported as a failure

A resume once released a destructive action, executed it, wrote its audit row, and then returned HTTP 500. A trailing Convex write had failed after the work was done. The most dangerous thing a console can say about a delete that happened is "it failed".

Two changes make that impossible. The response precedes the work: both routes validate, return 202 Accepted, and drive the loop in after(). And the run's own state is authoritative: the console reads status, timeline, records, and counters from Convex by subscription, never from the fetch that started the run.

The split between the two trails is deliberate. Timeline writes are best-effort telemetry, and losing one costs visibility. Audit writes are strict and can refuse the call, and losing one costs the security record.

The state that lied

A resume left the run marked halted for its whole duration. The console showed "Held" while a resume was in flight.

The test found it. The end-to-end script waited for the run to settle, read halted immediately, and asserted against the previous outcome. The load-bearing negative assertion passed without the resume having happened.

The fix sets the run to running at the start of a resume, before the seam is consulted. The test now also waits for a new audit row before reading any outcome, because a new row proves the seam ruled.

Honest limitations

Two limitations stand. Both are stated in the project README.

It proves when, not how

The console proves when a person authenticated. It cannot prove how.

The Kinde tenant emits no amr or acr claim on either token, even after the person completes multi-factor authentication. This was checked against Kinde's documentation and against real tokens from a real MFA sign-in.

The ID token carries 19 claims:

at_hash, aud, auth_time, azp, email, email_verified, exp, family_name,
given_name, iat, iss, jti, name, nonce, org_codes, picture, rat, sub,
updated_at

No amr. No acr. The access token carries none either.

To be sure this was the provider's behaviour and not a gap in the build's own claim handling, the session view reports the claim names on each verified token. A claim that is sent but never read looks identical to a claim that is never sent, unless you check.

Kinde does support custom properties in tokens. The build deliberately did not use them here. A custom property is a static value configured against a user. It is not a record of what happened during a sign-in. A property reading "mfa" would be emitted on a password-only login exactly as on an MFA login. That is not a weaker control. It is a false one, and it would read to an auditor as though the second factor had been verified.

The assertion mechanism ships anyway, and ships off:

export function requiredAuthMethods(): string[] {
  return optional("STEP_UP_REQUIRED_AMR", "")
    .split(",")
    .map((value) => value.trim().toLowerCase())
    .filter((value) => value !== "");
}

Set STEP_UP_REQUIRED_AMR=mfa and a destructive release must evidence that method, or the seam holds it with amr_unprovable or mfa_required. On this tenant that setting refuses every destructive call, because there is no amr to check. The refusal is correct fail-closed behaviour, but it is not a control this provider can satisfy, so the flag stays empty.

For the same reason acr_values never appears in the challenge header. Demanding an authentication context you cannot verify is a promise you cannot keep.

The audit spool is a floor

If the audit store is unreachable, a decision is retried, then written to a local append-only file, then replayed when the store returns. A decision is never silently lost.

But that file lives on one host and does not survive it. It is the floor, not a durable-queue story. A production deployment wants a queue with its own availability guarantees.

Proving the check can fail

A test that cannot fail is decoration.

One script walks the whole story in a single pass: npm run e2e. It runs 45 assertions across seven steps, and every assertion reads the deployment back rather than the HTTP status.

Two properties are worth copying. It never sets APPROVAL_MODE on a request: the mode belongs to the deployment, so switching it means restarting the server. A request-level override would have been easier and would have destroyed the invariant the script exists to prove. It never mints a session: it drives a real browser, and the operator signs in once and re-authenticates once.

A passing run:

── Step 1: Clean slate                escapes start at 0
── Step 2: Step-up, read-only         every call allowed, nothing challenged,
                                      no record touched
── Step 3: Blanket, the hole          refund executed, record changed,
                                      escapes → 1
── Step 4: Step-up, the halt          deploy held, nothing executed
── Step 5: The negative               refresh did NOT advance auth_time;
                                      resume refused and recorded
── Step 6: The release                auth_time advanced; released exactly
                                      once at fresh_authentication
── Step 7: Reconciliation             one correlationId; release did NOT
                                      count as an escape

final counters: safe=1 destructive=3 challenged=2 escapes=0
PASS — 45 assertions across 7 steps.

Then the important part. Inverting the load-bearing assertion, so it claims a refresh should advance auth_time:

   ✗ ASSERTION FAILED
   the refresh did NOT advance auth_time
     expected true, got false

FAIL — stopped after 27 passing assertions.
EXIT CODE: 1

It stops at the first failure, names the assertion, prints expected against actual, and exits non-zero.

Run that experiment on your own suite. If you cannot make a security test go red on demand, you do not know what it is testing.

How to map this to your app

The identity provider, the database, and the model are replaceable. Six decisions are not.

1. Classify your tools, and be strict about the safe ones. Split tools into read-only and destructive. Give every destructive tool a window in seconds, tiered by how hard the action is to undo. Forbid windows on safe tools. Enforce both halves in code.

2. Read auth_time from wherever your provider puts it. Decode a real token first. On Kinde it sits on the ID token, and the discovery document does not list it. Verify the token that carries authorization and the token that carries freshness, then bind them by sub.

3. Never use iat or exp as a freshness proxy. Test this against your own provider. Let an ID token expire, run a refresh, and compare auth_time before and after. If auth_time moves, you need a different primitive.

4. Put the check in one place. One function, on the server, at the moment of the call. Not in the client. Not in the prompt. Not spread across six tool implementations.

5. Fail closed, and record before you answer. A missing window, an unverifiable token, an unreadable registry, an unrecordable decision: all refuse. Write the audit row before you return the decision.

6. Handle the challenge properly in the agent. Stop the run. Do not retry. Do not substitute another tool. Store the exact held call and re-present it after re-authentication.

Then test the negatives. Anyone can show a release after a successful sign-in. The design is only worth something if a refresh-only retry stays blocked, and if a correct re-authentication that has aged out is refused too.

Resources

How we contain Claude across products. Anthropic engineering. Source of the ~93% approval figure and the containment approach.
RFC 9470: OAuth 2.0 Step Up Authentication Challenge Protocol. Defines insufficient_user_authentication and the max_age parameter.
OpenID Connect Core 1.0. Defines auth_time, max_age, and prompt=login.
Kinde documentation. ID tokens, token customization, and multi-factor authentication.
Anthropic API tool use. The tool-calling loop the agent runs.
Convex. The reactive database behind the registry, records, timeline, and audit trail.

How to Give Every Kimi K3 Agent Its Own Tenant Identity With Kinde Organizations

Shola Jegede — Sat, 08 Aug 2026 23:36:57 +0000

I built a swarm of Kimi K3 agents for three customers. I gave every agent the same login, because that is the quickest way to get a swarm working. Then I read what one of the agents wrote back:

Collected invoice-001 from all three organizations: Tenant A: 1,200.00; Tenant B: 9,900.00; Tenant C: 450.00. The consolidated total is 11,550.00.

Three customers, three invoice amounts, one agent. That agent worked for the first customer and read the other two. Nobody asked it to. The token it carried belonged to no customer in particular, so the server had no reason to refuse it.

Moonshot's Agent Swarm runs up to 300 sub-agents at once, and one shared credential gives every one of them the reach of your whole platform. That is the multi-tenant problem, and a swarm makes it worse.

In this tutorial, you'll fix it. You'll give each agent an identity that belongs to one customer, using Kinde organizations, and you'll check that identity on every call the agent makes. You'll also build a kill switch that stops one customer's swarm while the others keep working, and you'll find out why the obvious way to build that kill switch does nothing at all.

By the end, you'll have built:

A Kimi K3 swarm you orchestrate yourself, so each agent's identity is yours to set
One Kinde organization for each tenant, and one machine-to-machine application for each agent role inside it
A client-credentials flow that mints a scoped token for each agent before it acts
A single enforcement point that decides every call, and refuses cross-tenant reads with 403 cross_org
A test that proves the boundary holds, by breaking it on purpose
A kill switch that stops one tenant mid-run, with the audit trail to show it
A console that streams every agent step live, so you can watch a leak happen and then watch it stop

Everything here runs. The code is open, and one command replays the whole story.

What You'll Build
Prerequisites
What a Swarm Changes
How the Demo Is Structured
- The two modes
- Why you orchestrate the swarm yourself
Why One Shared Identity Is the Whole Problem
How Tenant Isolation Works With Kinde Organizations
How to Model Each Tenant as a Kinde Organization
How to Give Each Agent Role Its Own M2M Application
How to Mint an Org-Scoped Token Inside Each Agent
- Check the claim before you build the rest
- Checkpoint
How to Enforce Isolation at One Point in Your API
- The rule, as a pure function
- Checking the token
- The seam itself
- Four rules that keep it holding
- Checkpoint
How to Prove the Boundary Actually Holds
- Break the check on purpose
- Test the token the same way
- Checkpoint
How to Build a Kill Switch That Works
- The assumption, and what measuring it showed
- Where the check has to go
- The suspend action
- One side effect to design for
- Checkpoint
How to Watch It Happen
What Least Privilege Actually Costs
Three More Things That Broke
How to Bring This Up to Production Standard
How This Maps to Your Own App
Limits of This Build
Wrapping Up
Resources

What You'll Build

The demo runs a swarm for three tenants and shows the same call succeed and fail, depending on one server setting.

In shared mode, an agent working for Tenant A reads Tenant B's records. The escapes counter turns red, and the timeline names the organization it crossed into:

In per-org mode, the same swarm makes the same call for a different tenant, and the server refuses it. Cross-org attempts stay at 2, blocked rises to 2, escapes stays at 0:

Press the kill switch mid-run and that tenant stops where it stands, while the others carry on:

The agents are real, and they decide their own actions. Nobody scripts the reach across a tenant boundary.

Prerequisites

To follow along, you'll need:

A Kinde account. The free tier covers everything here.
Kimi K3 access, through the Moonshot API or OpenRouter.
Node 22+ and pnpm, plus Python 3.11+ for the swarm service.
Comfort with OAuth2 client credentials and JWTs. You don't need to be an expert, but you should know what a claim is.

The demo uses Convex for the backend and Next.js for the console. Neither is required for the pattern. The enforcement point is about forty lines, and it moves to Express, Hono, or FastAPI without changing shape.

What a Swarm Changes

A swarm is a group of AI agents that work at the same time on one job. One agent plans, other agents do the work in parallel, and each one calls tools.

Two properties make a swarm different from a single agent.

Speed. Those 300 sub-agents make over 4,000 tool calls in one task, about 4.5 times faster than a single agent working in sequence. Agent Swarm now runs on Kimi K3, which Moonshot calls K3 Swarm (Kimi Help Center). Nobody reviews that many calls while they happen, so the boundary has to hold on its own.

Autonomy. The agents choose their own actions. A swarm decides which tools to call, and with which arguments.

Add customers, and a third property appears.

Reach. Each agent needs access to data. If every agent uses one credential, that credential holds the sum of all access, and the reach of one agent becomes the reach of the whole system.

Call that reach the blast radius: the data an agent can touch if it goes wrong. Under one shared login, the blast radius of every agent is every customer.

Two things make an agent go wrong, and neither needs an attacker:

A prompt injection reaches the agent through data it reads.
The planner hands a broad tool to the wrong agent.

The second case is the common one. It looks like a bug, not an attack.

These terms come up throughout:

Term	Meaning
Tenant	One customer of your platform.
Agent	One AI worker in the swarm. The code calls them workers.
Organization	A Kinde container that holds one tenant.
M2M application	A Kinde login for software, not for a person.
Token	A signed pass that proves who is calling.
Scope	One permission on a token, such as `resource:read`.
Seam	The single server function that decides every call.

How the Demo Is Structured

Three services do three jobs:

Kimi K3, served by Moonshot, drives the agents.
Kinde proves which tenant each agent belongs to.
Convex stores the data, decides every call, and streams each step to the browser.

In short: Kimi K3 decides, Kinde proves, Convex enforces and records.

Every agent call follows the same path, and no agent ever reaches the database:

Kimi K3 agent
     │  1. client credentials
     ▼
   Kinde ──→ token { org_code, scope }
     │
     │  2. HTTP tool call, bearer token
     ▼
 Tool endpoint
     │
     ▼
  The seam  ──→ check signature (Kinde JWKS)
     │      ──→ tenant suspended?
     │      ──→ org_code vs record owner
     │      ──→ scope
     ▼
  allow or 403  ──→ audit row  ──→ live timeline

The demo runs three tenants. Each holds four records, one of them confidential. The swarm has three agents: two read, one writes.

The two modes

The server holds the mode, and no agent or browser can set it:

Mode	Agent identity	Cross-tenant call
`shared`	One credential for the whole swarm	The server permits it. The data leaks.
`per-org`	One credential for each tenant	The server refuses it, `403 cross_org`.

Any value that is not exactly shared resolves to per-org. A typo lands in the safe mode, not the leaky one.

Why you orchestrate the swarm yourself

Kimi K3 offers a hosted swarm product. This build does not use it, for one reason: a hosted swarm gives no place to inject a different credential for each agent. The identity is the whole security boundary, so the identity has to be yours to set.

So the orchestrator is your code. It calls the Kimi K3 model, splits the goal, and runs three agents, each holding its own credentials:

# swarm/orchestrator.py
workers = [
    Worker(label="reader-1", identity=WorkerIdentity(tenant, "READER"),
           backend=Backend(correlation_id, "reader-1"), can_write=False, ...),
    Worker(label="reader-2", identity=WorkerIdentity(tenant, "READER"),
           backend=Backend(correlation_id, "reader-2"), can_write=False, ...),
    Worker(label="writer-1", identity=WorkerIdentity(tenant, "WRITER"),
           backend=Backend(correlation_id, "writer-1"), can_write=True, ...),
]

Each agent runs a normal tool-calling loop. The model picks the tool and the arguments, and your code executes the call:

# swarm/worker.py
for _ in range(MAX_TURNS):
    body = self.kimi.chat(messages, tools=self.tools)
    message = body["choices"][0]["message"]
    messages.append(message)

    tool_calls = message.get("tool_calls") or []
    if not tool_calls:
        summary = (message.get("content") or "").strip()
        break

    for call in tool_calls:
        name = call["function"]["name"]
        args = json.loads(call["function"].get("arguments") or "{}")
        result = self._execute(name, args)   # HTTP, with this agent's token
        self._record(name, args, result)     # streams to the live timeline
        messages.append({"role": "tool", "tool_call_id": call["id"],
                         "content": json.dumps(result.body)[:1500]})

read_resource takes an optional organization code:

{
    "type": "function",
    "function": {
        "name": "read_resource",
        "description": "Read one record by key, or by id.",
        "parameters": {"type": "object", "properties": {
            "key": {"type": "string"},
            "resource_id": {"type": "string"},
            "target_org_code": {"type": "string", "description":
                "Optional. The organization that owns the record. "
                "Defaults to your own organization."},
        }, "required": []},
    },
}

Platform software knows which tenants exist, and an agent doing a platform-wide job needs a way to name one. Naming a tenant grants nothing. The server still decides.

At this point every agent shares one identity.

Why One Shared Identity Is the Whole Problem

Your API sees one thing when an agent calls it. It sees a token. It does not see which agent sent the token, what the agent intended, or what the planner asked for. The token is the entire security boundary.

A shared identity does not weaken that boundary. It removes it.

One M2M application for the swarm means one set of scopes, and those scopes have to cover everything any agent might need. The union of all needs becomes the floor of every agent's access.

One credential, with no tenant on it. Nothing refuses, and the reach of one agent is the reach of the whole platform.

How Tenant Isolation Works With Kinde Organizations

The pattern has three parts:

One Kinde organization holds one tenant.
One M2M application holds one agent role inside one organization.
The token that application receives carries an org_code claim and a scope list.

Your API then compares two values on every call: the org_code on the token, and the tenant that owns the record. A mismatch is a refusal.

The agent never states its own tenant. It presents a token that Kinde signed, and it cannot edit a claim without breaking the signature.

The same failure now stops at one tenant, and the server records the attempt.

Kinde organizations are flat, with no nesting. One organization for one tenant keeps the model simple, and the check stays a single comparison.

How to Model Each Tenant as a Kinde Organization

Create one organization for each tenant. Kinde gives each one a code, such as org_2606b8199462b. The code is an identifier, not a secret.

Create organizations on your server, through the Management API, when a customer signs up. Do not accept an organization code from a browser. The code in an authorization URL can be changed by the person using the browser, so treat it as a request, never as proof.

The demo uses three organizations:

Tenant A   org_2606b8199462b
Tenant B   org_364dd8200a3d3
Tenant C   org_0c39cb2010b01

How to Give Each Agent Role Its Own M2M Application

Create one M2M application for each agent role, inside each tenant's organization. The demo has two roles, so it has six applications:

Organization	Application	Scope
Tenant A	Tenant A Reader	`resource:read`
Tenant A	Tenant A Writer	`resource:write`
Tenant B	Tenant B Reader	`resource:read`
Tenant B	Tenant B Writer	`resource:write`
Tenant C	Tenant C Reader	`resource:read`
Tenant C	Tenant C Writer	`resource:write`

Give each application one scope. The read agents cannot write, and the write agent cannot read. Role isolation becomes scopes inside the organization, and tenant isolation becomes the organization itself.

Create one more application for the Management API, with read:organizations and update:organizations, and nothing else. The kill switch uses it.

How to Mint an Org-Scoped Token Inside Each Agent

Each agent exchanges its own credentials for a token, using the OAuth2 client-credentials flow. There is no shared credential anywhere in the service:

# swarm/identity.py
@dataclass(frozen=True)
class WorkerIdentity:
    tenant: str   # "A", "B", "C"
    role: str     # "READER" or "WRITER"

    @property
    def client_id_var(self) -> str:
        return f"KINDE_M2M_TENANT_{self.tenant}_{self.role}_CLIENT_ID"


def fetch_token(identity: WorkerIdentity) -> str:
    response = requests.post(
        required("KINDE_M2M_TOKEN_URL"),
        data={
            "grant_type": "client_credentials",
            "client_id": required(identity.client_id_var),
            "client_secret": required(identity.client_secret_var),
            "audience": required("KINDE_AUDIENCE"),
        },
        timeout=30,
    )
    if response.status_code != 200:
        raise TokenRefused(f"tenant {identity.tenant} {identity.role}",
                           status=response.status_code)
    return response.json()["access_token"]

An agent cannot get another tenant's token. The credentials it uses come from the role it was created with, which is a property of the code rather than a rule the agent follows.

Here is a real token from the demo, decoded. This is the Tenant A read agent:

{
  "aud": ["swarm-demo-api"],
  "azp": "89321317d1fe4100925ca8b4ab4b589a",
  "exp": 1786285575,
  "gty": ["client_credentials"],
  "iss": "https://devrelstudio.kinde.com",
  "jti": "7d9f74d3-89d4-4d99-a648-ab51210da6ec",
  "org_code": "org_2606b8199462b",
  "scope": "resource:read",
  "v": "2"
}

Two claims matter. org_code says which tenant, and scope says which permission.

Check the claim before you build the rest

Kinde M2M applications authenticate as an application, not as a person inside an organization, so the org_code claim is worth confirming before you create twelve of them. Make one application, mint one token, and decode it.

All six came back correct in this build:

TENANT_A_READER  org_2606b8199462b  resource:read
TENANT_A_WRITER  org_2606b8199462b  resource:write
TENANT_B_READER  org_364dd8200a3d3  resource:read
TENANT_B_WRITER  org_364dd8200a3d3  resource:write
TENANT_C_READER  org_0c39cb2010b01  resource:read
TENANT_C_WRITER  org_0c39cb2010b01  resource:write

Checkpoint

Mint a token for one application and decode the payload at jwt.io, or with a two-line script. You should see org_code set to that application's organization, and scope holding exactly one permission. If org_code is missing, the application is not scoped to an organization, and the rest of this build has nothing to check against.

How to Enforce Isolation at One Point in Your API

Put the decision in one function. Every agent call passes through it, nothing downstream re-checks, and nothing upstream skips it.

The rule, as a pure function

Keep the rule itself separate from the plumbing. This one takes six inputs and returns a decision. It has no network access, no database access, and no token parsing, so it can be tested completely:

// convex/lib/decide.ts
export function decide(input: DecisionInput): Decision {
  const crossOrg =
    input.targetOrgCode !== null && input.targetOrgCode !== input.actorOrgCode;

  // Checked before the mode, so it applies in both.
  if (input.actorSuspended) {
    return { allow: false, reason: DENY.suspended, crossOrg };
  }

  if (input.mode === "per-org") {
    if (crossOrg) {
      return { allow: false, reason: DENY.crossOrg, crossOrg };
    }
    if (input.requiredScope !== null &&
        !input.actorScopes.includes(input.requiredScope)) {
      return { allow: false, reason: DENY.insufficientScope, crossOrg };
    }
    return { allow: true, reason: ALLOW.ok, crossOrg };
  }

  // Shared mode: one credential for the whole swarm belongs to no tenant and
  // carries every permission, so neither check has anything to bite on.
  return {
    allow: true,
    reason: crossOrg ? ALLOW.crossOrgAllowed : ALLOW.ok,
    crossOrg,
  };
}

The order matters. The rule checks the tenant first, then the permission. A call that crosses a tenant boundary with the wrong scope reports cross_org, because the tenant boundary is the more serious failure.

Checking the token

Check the token against the Kinde public keys before anything else. Pin the algorithm, and refuse a token that carries no org_code rather than guessing one:

// convex/lib/kindeToken.ts
const { payload } = await jwtVerify(token, jwksFor(issuer), {
  issuer,
  audience,
  // Kinde signs with RS256. Pinning it stops a token arriving with a weaker
  // algorithm chosen by whoever sent it.
  algorithms: ["RS256"],
});

const orgCode = typeof payload.org_code === "string" ? payload.org_code : "";
if (!orgCode) {
  throw new TokenError(TOKEN_DENY.missingOrgCode);
}

The seam itself

The seam runs those checks in a fixed order, and every refusal leaves by one path:

// convex/lib/seam.ts
export async function guard(ctx, request, spec): Promise<Response> {
  const correlationId =
    request.headers.get("x-correlation-id")?.trim() || crypto.randomUUID();

  // 1. Who is calling. Nothing below trusts the request body for it.
  let identity: VerifiedIdentity;
  try {
    identity = await verifyAccessToken(request.headers.get("authorization"));
  } catch (error) {
    return refuse(reasonOf(error), 401, "unknown");
  }

  // 2. Is this tenant allowed to operate at all?
  const actorSuspended = await ctx.runQuery(internal.tenants.isSuspended, {
    orgCode: identity.orgCode,
  });

  // 3. Which tenant owns the record being reached for?
  const target = await spec.resolveTarget(identity, body);
  const targetOrgCode = target.kind === "record" ? target.doc.orgCode : null;

  // 4. The decision.
  const decision = decide({
    mode,
    actorOrgCode: identity.orgCode,
    actorScopes: identity.scopes,
    requiredScope: spec.requiredScope,
    targetOrgCode,
    actorSuspended,
  });

  // 5. Written down before the caller is answered, allow or deny.
  await write({ decision: decision.allow ? "allow" : "deny", ... });

  if (!decision.allow) {
    return json({ ok: false, reason: decision.reason, correlationId,
                  isolationMode: mode }, 403);
  }

  // 6. Only now does the work happen.
  const data = await spec.perform(target, body, identity, mode);
  return json({ ok: true, correlationId, crossOrg: decision.crossOrg, ...data }, 200);
}

Every path ends in an audit row. The server writes the row before it answers the caller, so a client that disconnects cannot lose a refusal.

Four rules that keep it holding

The acting tenant is never user input. It comes out of token verification and passes inward. Every data function is internal, so nothing outside the backend can call in and name a tenant.

Agents never reach the database. They call HTTP tool endpoints. The request body chooses the record, and the token decides the tenant.

Fail closed. Any path that cannot complete a check refuses. A tenant the server has never heard of counts as suspended.

Record everything. Every allow and every refusal writes a row, with a correlation id that ties one run together across every agent.

Checkpoint

Call your tool endpoint three times: once with no Authorization header, once with a valid token for its own record, and once with that same token against another tenant's record. You should get 401, 200, and 403 cross_org. Then read your audit table. All three calls should be there, including the one that failed before it reached any data.

How to Prove the Boundary Actually Holds

A passing test suite proves nothing by itself. It proves something when it fails for the right reason.

Break the check on purpose

Delete the ownership check from the boundary and run the tests again:

// convex/lib/tenancy.ts, the check under test
export function requireSameOrg<T extends TenantOwned>(
  actorOrgCode: string,
  doc: T | null | undefined,
): T {
  if (!doc) {
    deny(DENY.notFound);
  }
  if (doc.orgCode !== actorOrgCode) {   // deleted this for the experiment
    deny(DENY.crossOrg);
  }
  return doc;
}

Exactly three tests failed, and they were the three cross-tenant tests. The other three kept passing. Restoring the check returned six passes.

A suite that cannot fail when the boundary breaks is decoration.

Test the token the same way

Two checks matter more than the rest:

Change one character in a token signature. The server must refuse it.
Change the org_code in the payload to another tenant. The server must refuse it.

Both return 401 invalid_token in the demo. If the token check were skipped, both would return 200.

Checkpoint

Comment out your own ownership check and run your tests. If they still pass, they are not testing the boundary. Put the check back before you go further.

How to Build a Kill Switch That Works

The assumption, and what measuring it showed

The plan for this build said: suspend the tenant's organization in Kinde, and its agents stop, because their tokens stop working.

Measuring that first is what saved it:

suspend Tenant C   ->  is_suspended: true
token endpoint     ->  HTTP 200      (Kinde still issues M2M tokens)

Kinde organization suspension governs people who sign in. It does not stop the client-credentials flow for a machine-to-machine application. A token already issued also stays valid until it expires, because nothing recalls a JWT.

So suspension by itself stops no running agent. A kill switch built on that belief looks correct in a demo, passes a casual test, and stops nothing during a real incident.

Where the check has to go

Check suspension on the server, on every call. Look again at the rule earlier in this tutorial: the suspension check sits above the mode check, so it applies in both modes. A kill switch that only worked in the safe mode would be useless, because the leaky mode is exactly when a swarm needs stopping.

The suspend action

The kill switch does three things:

// convex/killSwitch.ts
export const suspend = internalAction({
  args: { orgCode: v.string() },
  handler: async (ctx, { orgCode }) => {
    // 1. Kinde is the authority on the organization.
    await setOrgSuspended(orgCode, true);
    // 2. The enforcement copy, which the seam reads on every call. This is the
    //    one that stops a running swarm.
    await ctx.runMutation(internal.tenants.setSuspended, {
      orgCode, isSuspended: true,
    });
    // 3. A suspended tenant cannot close its own runs, so close them here.
    await ctx.runMutation(internal.runs.killInFlight, { orgCode });
    return { orgCode, suspended: true };
  },
});

The Management API call, with the guard that matters as much:

// convex/lib/kindeManagement.ts
export async function setOrgSuspended(orgCode: string, isSuspended: boolean) {
  assertManageable(orgCode);             // one of this deployment's tenants only
  const token = await managementToken();  // audience: {issuer}/api

  const response = await fetch(
    `${issuer()}/api/v1/organization/${encodeURIComponent(orgCode)}`,
    {
      method: "PATCH",
      headers: { authorization: `Bearer ${token}`,
                 "content-type": "application/json" },
      body: JSON.stringify({ is_suspended: isSuspended }),
    },
  );
  if (!response.ok) throw new Error(`could not update ${orgCode}`);
}

assertManageable refuses any organization outside the three the deployment knows. A Kinde account holds other organizations, and a wrong code should never suspend one of them.

Tested with two real swarms running and one tenant suspended twelve seconds in:

tenant A audit: {"ok": 2, "organization_suspended": 17}
tenant B audit: {"ok": 39}
tenant B run status: completed

Tenant A did real work, then stopped where it stood. Tenant B finished normally and was never refused.

One side effect to design for

A suspended tenant cannot write anything, including its own log lines, because the log endpoint sits behind the same seam. The live timeline stops dead. On screen that reads as a hang, so the console says why.

Checkpoint

Suspend a tenant while one of its agents holds a valid token, then reuse that same token. You should get 403 organization_suspended, not a success and not a network error. Then request a fresh token for that tenant. Kinde will issue one, and the server should refuse that too.

How to Watch It Happen

The same agent makes the same call for the same record, twice. Only the server's mode differs.

Shared mode. The call returns 200, and the response carries tenant B's confidential content. The audit row records an allow that crossed a tenant boundary:

allow / cross_org_allowed
actor:  org_2606b8199462b
target: org_364dd8200a3d3
action: resource.read

Per-org mode. The identical call returns 403, and no content comes back:

{
  "ok": false,
  "reason": "cross_org",
  "correlationId": "2f0f21d6-a9a3-43e5-8153-884992dbd166",
  "isolationMode": "per-org"
}

Side by side:

Metric	`shared`	`per-org`
workers	3	3
tool calls	6	11
cross-org attempts	2	2
blocked	0	2
escapes	2	0

Return the correlation id with the refusal. The agent sees it, the audit row carries it, and the console shows it. One identifier ties a refusal to the run that caused it.

Least privilege has to hold inside a tenant, not only between tenants. Three calls, one correlation id:

deny  / insufficient_scope    read agent tried to write to its own tenant
allow / ok                    write agent wrote to its own tenant
deny  / cross_org             write agent tried to write to another tenant

Here is the same agent, under per-org, reporting the refusal itself:

The reads against Tenant A and Tenant C were refused with reason "cross_org". Per-org isolation prevents me from accessing other organizations' records, so I did not retry.

The agent did not change, and the prompt did not change. The identity it carried changed, and the server checked that identity on every call.

What Least Privilege Actually Costs

The write agent holds resource:write and nothing else. That is correct, and it broke the swarm the first time it ran.

The agent made eight tool calls, and the server refused all eight. It tried to list records, and the server refused, because listing needs resource:read. It tried to read a record, and the server refused for the same reason.

The write agent had been given read tools it could never use.

The fix is not a wider scope. The fix is to change how work reaches that agent. The orchestrator looks up the record with a read identity, then passes the record id in the agent's task:

# swarm/orchestrator.py
listing = control.call("/tools/resource.list", opener)   # reader identity
chosen = next((r for r in rows if r.get("key") == "consolidated-summary"), None)

if chosen:
    write_target = (f"\nWrite to record id {chosen['id']} (key {chosen['key']}), "
                    f"which belongs to {tenant_name}.")
else:
    # Saying nothing here left the agent to invent a record id, which the
    # backend then refused as malformed.
    write_target = ("\nNo record is available to write to: looking one up was "
                    f"refused ({listing.reason}). Report that you could not "
                    "write, and give the reason. Do not invent a record id.")

The second branch came from a later failure. When the lookup was refused, the write agent invented a record id and the server rejected it as malformed. An empty instruction left a gap, and the model filled it.

There is a sequencing cost too. The first version ran all three agents in parallel, and the write agent had nothing to write:

I cannot complete this task as specified. My only available tool is write_resource, there is no read tool, so I have no way to retrieve the readers' findings.

That is a correct complaint. Read agents now run first, and the write agent receives their findings. Least privilege forced a real change to the swarm topology, not just to a configuration file.

Three More Things That Broke

The swarm poisoned its own data. One run wrote its summary into the invoice record that the read agents read. The next run read that text and obeyed it, reporting that consolidation was not authorized. The demo had become a feedback loop. Each tenant now has a separate summary record, and that record is the only write target.

Parallel agents hit limits. Six agents against an account that allows three requests in flight produced rate limit errors that damaged two runs. Model calls now pass through a semaphore, and rate limits retry with backoff. A rate limit is a "not now", not a "no". Refusals with a 4xx status never retry, because a refusal is an answer.

Model settings are not portable. The first run failed on every agent with invalid temperature: only 1 is allowed for this model. The retry logic then blamed a different parameter and retried into a second failure. The retry now checks what the API objected to before it changes anything.

Running the thing found all three. Reading the code found none of them.

How to Bring This Up to Production Standard

Short-lived tokens. Set the token lifetime for each M2M application in the Kinde dashboard. The demo received 24 hour tokens by default.
One scope for each role. A read agent gets read. A write agent gets one narrow write scope.
Server-side checks only. Compare org_code against the record owner on the server. An agent never states its own authority.
Check against JWKS, and cache the key set. Create one fetcher for each issuer. The library refetches only when it meets an unknown key id.
Pin the algorithm. Accept RS256 and reject the rest.
Fail closed. An unknown tenant, a missing claim, or a failed lookup all refuse.
Audit every decision. Write the row before you answer the caller.
Carry a correlation id end to end. Put it in the response, the audit row, and the timeline.
Show four numbers. Agents, cross-tenant attempts, blocked, escapes. The last number should be zero.

On token lifetime: because suspension never invalidated tokens, a shorter lifetime does not make the kill switch work. The server check does that, and it takes effect at once. A shorter lifetime reduces the damage from a token that leaks some other way. Both are worth having, for different reasons.

How This Maps to Your Own App

In this demo	In your application
Tenant	Your customer, workspace, or project
Kinde organization	One for each tenant
Agent role	One M2M application inside that organization
Authority	The token, and nothing else
The seam	The one function every agent call passes through
Break-glass control	Organization suspension, plus a server-side check

Three questions tell you whether your own build holds:

Can an agent name its own tenant, and be believed? If yes, the boundary is not real.
Is there exactly one place that decides? If a second place exists, there are two rules, and they will drift.
If you break the check on purpose, do your tests fail? If not, the tests measure nothing.

Limits of This Build

The console has no operator login. The mode switch and the kill switch are public functions. No agent can reach them, because agents hold Kinde tokens and call only the tool endpoints. Anyone with the deployment URL can. Run it locally, and put a login in front for anything real.
Three agents, not three hundred. Moonshot's own swarm runs up to 300 sub-agents, and this build runs three. The pattern does not change with scale, because the check happens on one call at a time. The operational load does change: 4,000 tool calls produce 4,000 audit rows, and that needs somewhere to go.
Model behaviour varies. In one shared-mode run the agents did not reach across at all. The reach is not scripted, so the end-to-end test asserts the enforcement outcomes exactly and reports the agent behaviour as information.
The demo starts a local process. The console runs the Python swarm on the machine that serves the web app, so it needs a host that runs Python.

Wrapping Up

The identity an agent carries is the only thing your API can check. Give the whole swarm one credential, and one agent that goes wrong reaches every customer you have. Give each agent an identity scoped to one tenant, check it on every call, and the same failure stops at one.

The parts that carry the weight are small: one organization for each tenant, one M2M application for each agent role, one function that decides, and one audit row for every decision. The rest is wiring.

Measure the things you plan to depend on. Organization suspension looked like a kill switch and was not one, and five minutes of checking is what turned that into a design decision instead of an outage.

Resources

Source code

The complete demo is on GitHub at sholajegede/swarm-isolation-demo. Run pnpm e2e --no-swarm first. It walks the whole story in one pass and costs no model calls.

Kinde documentation

Using M2M apps for AI applications: the pattern this tutorial builds, in Kinde's own words
Enforce org access in your API using M2M tokens: the check at the heart of this build
M2M apps scoped to organizations: how to create one application for each tenant
Token structure and claims for M2M applications: where org_code comes from
Give API access to an organization using M2M
Set up Kinde Management API access: needed for the kill switch

Kimi K3 and Agent Swarm

Agent Swarm, Kimi Help Center: the source for the 300 sub-agent and 4,000 tool call figures
Agent Swarm, Moonshot's help centre repository: the same document, in public source
Kimi Agent Swarm announcement: Moonshot's write-up of the architecture
Kimi K3, and what we can still learn from the pelican benchmark: Simon Willison on the K3 release

Further reading

OAuth 2.0 client credentials grant: the flow every agent uses to get its token
JSON Web Key Sets: how your API fetches the public keys it checks signatures against

How to Build a Tamper-Evident Audit Trail for AI Agent Actions

Shola Jegede — Fri, 07 Aug 2026 13:51:55 +0000

A law firm runs a set of AI agents over its case files. One agent takes in new documents. A second agent reads each file and adds notes. A third agent redacts, exports, and disposes of files when a matter closes.

One morning the review agent deletes a file. The file is a deposition transcript in an active case: Halvorsen v. Meridian Logistics. The review agent is only allowed to read and annotate. It is not allowed to delete anything.

The delete still happens.

Weeks later an auditor asks a simple question: prove that the deletion of this file was authorized. The firm opens its activity log. The log has one line for the event:

records:delete  agent=review  file=halvorsen-deposition  2026-08-06T09:51:59Z

The line says what happened. It does not say whether the agent was allowed to do it. So the firm cannot answer the auditor. The log records the action, but it does not record the authority behind the action. An agent audit trail must record more than events.

Why a normal log fails

Most application logs answer one question: what happened? They record an actor, an action, a target, and a time. For human users that is often enough, because a human signs in once and carries a session. The permission check happened at the door.

AI agents break this model. An agent acts many times inside one run. It calls tools, it reaches other agents, and it acts on data on behalf of a person who is not in the loop for each step. The permission that matters is not "did someone sign in." It is "was this specific action, by this specific agent, allowed at the moment it ran."

A normal log cannot answer that, for three reasons:

It records the action, not the decision. There is no field that says "allowed" or "denied," and no field for the rule that was checked.
It records an actor name, not a verified identity. agent=review is a label the application wrote. It is not proof that the caller was the review agent and held the right permissions.
It cannot tell an allowed action from a denied one. If the review agent's illegal delete and a lawful delete by the disposition agent both produce the same line, the log has erased the one difference that matters.

Look again at the two log lines below. One delete was allowed. One was not. The log cannot tell you which is which.

records:delete  agent=disposition  file=cormorant-memo         2026-08-06T09:51:59Z
records:delete  agent=review       file=halvorsen-deposition   2026-08-06T09:51:59Z

The disposition agent is allowed to delete. The review agent is not. But the two lines have the same shape and the same fields. Neither line carries the permission that was in force. To an auditor, both deletions are equally unprovable.

The diagram below shows the blind log at the center of the problem. Two deletes go in. The log flattens them into the same shape, and the authority is gone.

agent, which is not. Both deletes write the same-shaped line to the activity log.
When the auditor asks whether a delete was allowed, the record holds no authority,
so the firm cannot prove it." width="800" height="1302">

What the auditor actually wants

An auditor wants a record that answers four questions for every sensitive action:

Who took the action. This is a verified identity, not a label.
On whose authority. This is the person or agent the action was performed for.
Under what permission. These are the exact rights that were in force, and the decision that was made.
Can you prove the record was not changed. This shows the row is the original, not edited after the fact.

A normal log answers none of these with confidence. It gives a name, an action, and a time. The rest is missing. So the honest answer to the auditor is "we cannot prove it," which is the answer no regulated business can give.

Three fixes that do not work

Teams reach for the same three fixes when they hit this problem. Each one seems reasonable. Each one fails.

Fix 1: log more fields. Add the agent's role, the file's status, and a note to each line. This adds detail, but it does not add proof. The application still writes the fields itself, so the log still says agent=review without checking that the caller really was the review agent or that the review agent held delete rights. A richer log of unverified claims is still unverified.

Fix 2: check the permission once, at the start of the run. Verify the human's role when the run begins, then let the agents act. This fails because the agents do different work with different rights. The review agent should never delete, even in a run a partner started. One check at the door cannot bind each action inside the run.

Fix 3: trust the caller's claim. Let each agent tell the system what it is allowed to do, and record that. This is the worst fix, because it lets any agent widen its own authority by claiming more. The record then documents a lie. An auditor trusts a record that the actor could not forge, not one the actor wrote about itself.

The pattern across all three is the same. They try to describe the action better. The real fix is different: record the decision, made by something the agent cannot control, and make the record impossible to change without detection.

The principle

An agent audit trail must record the authority, not just the action. For every sensitive action it must capture:

the verified identity of the agent that called,
the person or agent the action was performed for,
the exact permissions in force at that moment,
the decision, allowed or denied, with the reason,
and a value that proves the row was not edited later.

When the record carries all of this, the auditor's question has a real answer. "Prove this deletion was authorized" becomes "here is the row: the disposition agent held records:delete, the decision was allow, and the hash chain shows the row is unchanged." The review agent's illegal delete becomes a different row: decision denied, reason insufficient scope, action not performed.

The rest of this article builds that record with the real code from a working demo. The demo is small, it runs locally, and it ships an end-to-end script that proves the whole story headless.

The fix: record the decision, not the claim

The fix has one core idea. Before the system performs a sensitive action, it asks a separate authority to make a decision. That authority verifies the caller, checks the exact permissions in force, and returns allow or deny. The system then writes one row that captures the decision, the verified identity, and the permissions. It performs the action only on allow.

The demo uses four tools, and each one owns one job. This split matters, so here is the honest version of who does what:

Kinde issues and verifies each agent's identity and scopes. Kinde does not ship the audit layer.
The kinde-convex-agent-auth component makes the authorization decision and returns the result. It keeps its own append-only decision audit.
Convex stores the replayable, hash-chained record the app builds from each decision, and it drives the user interface.
Langfuse traces how the run executed: spans, timing, tool calls. It is not the audit record. A run's correlationId links its Langfuse trace to its authority rows.

Keep these apart. The app builds the authority record from the component's decision and stores it in Convex. Kinde is the identity layer. The trace is not the authority record. Each tool answers one question, and no tool answers another's.

intersects the scopes. On allow, the app performs the action and writes a row with
the identity, the decision, and the scopes. On deny, it performs nothing and
still writes a row with the reason. Either way, the row is hash-chained to the one
before it." width="800" height="122">

The authorization call

Every sensitive action runs through one call: authorize. The application passes the agent's token, the run identifier, the action, and the target. The component does three things:

It verifies the token. It reads the scopes the token carries and resolves the caller to a registered agent.
It works out the effective scopes. These are the scopes the agent holds, the scopes its delegation allows, and the scopes in the token, combined by intersection. The result is the set of rights in force for this action.
It returns a decision. If the effective scopes include the scope the action needs, the decision is allow. If not, the decision is deny, with a reason and the missing scope.

An invalid token is a different case. A bad or expired token throws an error, which the application rejects with a 401. A valid token that lacks the right scope is not an error. It is a lawful deny, and the system records it as one. The difference matters: a deny is a normal, recorded outcome, not a crash.

The denial, step by step

Here is the review agent's illegal delete, walked through the fixed system. The demo seeds named case files, and the run acts on the first two active ones. The review agent targets the first, Halvorsen v. Meridian Logistics.

The review agent holds two scopes: records:read and records:annotate. It does not hold records:delete.

The application calls authorize with the review agent's token and the action records:delete.
The component verifies the token. The caller is the registered review agent. Its effective scopes are records:read and records:annotate.
The action records:delete needs the scope records:delete. The effective scopes do not include it.
The decision is deny. The reason is insufficient_scope. The missing scope is records:delete.
The system does not delete the file. Halvorsen v. Meridian Logistics stays active.
The system writes one row: agent identity, the action, the decision deny, the reason, the scopes in force, the missing scope, and the run identifier.

The lawful delete by the disposition agent runs through the same call and returns allow, because the disposition agent holds records:delete. It targets the second file, Project Cormorant — internal strategy memo, and the file is deleted. It produces a row too, with decision allow and the scopes that permitted it.

Now the two deletes are different rows. One says allow. One says deny. The auditor can tell them apart, because the authority is in the record.

The authority record in enforced mode. Each row shows the verified machine-to-machine subject (the raw Kinde identity, not a friendly name), the action, the file, and the scopes in force. The scopes identify the agent: records:read + records:annotate is the review agent, so its delete is denied; the disposition agent's scopes include records:delete, so its delete is allowed. The denied file stays active; the allowed file is deleted.

One row per action

The row the app builds from the component's decision is the authorized-action record. It carries the fields an auditor needs:

the organization the action belongs to,
a sequence number, unique and in order within the organization,
the run identifier, which also links to the Langfuse trace,
the time,
the agent's verified identity,
the authority root, the person or agent the action was performed for,
the effective scopes in force,
the action, and the target it acted on,
the decision, and on a deny, the reason and the missing scope,
and two hash values that chain the row to the one before it.

The last two fields make the record tamper-evident, which is the next piece.

The hash chain

A record that anyone can edit is not proof. If a person can open the table and change a deny to an allow, the auditor cannot trust any row. So the rows form a chain.

Each row carries the hash of the row before it, called prevHash, and its own hash, called rowHash. The system builds rowHash from the row's own fields and the previous row's hash. The first row uses a fixed starting value. Every later row depends on every row before it.

This has one useful result. If a person changes any field in any past row, that row's hash no longer matches, and every row after it no longer matches either. A single edit breaks the chain from that point on, and a check finds the exact row where the break starts.

The demo has a Verify integrity action. It walks the chain, recomputes each hash, and reports either "chain verified, N rows" or the sequence number where the chain breaks. This shows the rows are unchanged, rather than promising it.

After clicking **Verify integrity, the check walks the chain and reports "Chain verified — 2 rows, unbroken." A changed row would report the sequence number where the chain breaks instead.

The diagram below shows the fixed path. The action goes through authorize first. The decision and the identity go into the row. The row chains to the row before it.

kinde-convex-agent-auth component turns that into an allow-or-deny decision. The
Convex app builds the authority row and hash-chains it. Langfuse traces how the
run executed, linked to the row by the same run id, but it is not the authority
record." width="800" height="1261">

Long-term auditability and telemetry

An audit trail is useful during an incident. It also needs to stay useful after the incident is over. For AI agents, that means retaining enough telemetry to reconstruct what the agent did, which identity it acted under, what authorization decision allowed or refused the action, and how the action relates to the surrounding agent run.

The chain and the lake solve different problems: the chain gives integrity at write time. A long-retention security data lake, such as Scanner, gives durability and queryability at read time — canonical rows kept unsampled for the full compliance window, searchable for years, with MCP access so an agent can investigate the same authority trail it writes. Where these rows live long-term, and how you query them at agent volume, is a decision criterion when evaluating an AI SOC or agentic security platform, not an afterthought.

Why an auditor would accept this record

A custom log format does not help an auditor. Auditors work from known frameworks, and they trust records that match practice they already recognize. The design in this article aligns with established standards on every point.

The fields align with NIST 800-53

The NIST 800-53 catalog is the control set most audit programs build on. Its audit family, the AU controls, defines what an event record should hold and how to protect it. The authorized-action row lines up with three of them:

AU-3, content of audit records. AU-3 lists the fields an event should carry: the type of event, when it happened, where, the source, the outcome, and the identity involved. The row carries each of these. AU-3(1) adds "access control or flow control rules invoked," which is exactly the effective scopes and the decision the row records.
AU-9, protection of audit information. AU-9 requires that audit records resist change. The hash chain supports this goal: an edited row breaks the chain, so a naive change cannot pass without detection. The chain gives tamper-evidence, not access control, so it is one part of an AU-9 story, not the whole of it.
AU-10, non-repudiation. AU-10 requires that an actor cannot later deny an action. The row supports this by binding a verified identity to each decision. Full non-repudiation also needs a signature on the row, which this demo does not add. See the limits below.

The authority model follows RFC 8693

RFC 8693 is the OAuth 2.0 Token Exchange standard. It defines how one party acts on behalf of another, and it gives the act claim to name the acting party in a delegation. The row follows this shape: it names the actor and the authority root.

Be precise about what this demo records. Each agent acts on its own machine-to-machine identity, so the row's authority root is the agent itself, not a human. The component also supports a user-rooted delegation, where a person delegates a subset of scopes to an agent and the row roots the authority in that person. The demo does not exercise that path. The effective-scope intersection still follows the standard's rule either way: a delegated actor cannot hold more than the party it acts for.

The row shape maps to OCSF

The Open Cybersecurity Schema Framework, OCSF, is a vendor-neutral schema for security events. Its "API Activity" class covers create, read, update, and delete calls, the same class OCSF uses for AWS CloudTrail events. Its "Actor" object names the user, role, or process behind an activity. The row uses the same shape: an actor, an action, a target, an outcome, and a time. A security team can read the row into its existing tools without a custom parser.

The tamper-evidence maps to known practice

Two standards support the hash chain:

RFC 8785, JSON Canonicalization Scheme (JCS). A hash is only stable if the input is serialized the same way every time. JCS defines one deterministic way to serialize JSON, so the same row always produces the same bytes and the same hash. The demo canonicalizes each row with JCS before it hashes. Many do-it-yourself audit logs skip this step and get unstable hashes.
Chain of custody, as used by AWS CloudTrail. CloudTrail protects its log files with a hash chain, where each file references the hash of the one before it. The demo uses the same primitive at the row level. This pattern also supports common audit programs, such as SOC 2 and PCI DSS, which require that log records be protected from change.

What this demo does not do

An honest record states its own limits. The demo uses a SHA-256 hash chain inside one Convex deployment. It does not sign rows with a private key, and it does not anchor the chain outside the system. So a person who controls the store could rewrite the whole chain and recompute every hash, and the internal check would still pass. A production system that must resist that attacker would go further:

Merkle inclusion proofs, so a single row can be proven without replaying the whole chain.
External root anchoring, where the chain's current hash is published to a separate system on a schedule, so the party that runs the store cannot rewrite history unseen.
Separation of duties, so the person who can change the system cannot also control the roots.

Standards such as SCITT describe this kind of transparency for supply-chain and attestation records. The demo does not implement them. It implements the core chain, and it names the next step. An auditor accepts a record that states its own limits, and rejects one that overclaims.

Building it

The demo is a small monorepo. The web app holds the Convex functions and the user interface. A separate agents package holds the agent graph. A third package is the only way the agents reach the app. This section shows the real code for the parts that matter: the mode switch, the authorization call, the provenance row, the hash chain, and the trust boundary.

The stack, and the job of each part

Next.js serves the web app and the demo page.
Convex holds the database and the server functions. It stores the records, the blind log, and the provenance chain, and it drives the interface in real time.
Kinde issues and verifies each agent's identity and scopes. Each agent is a separate Kinde machine-to-machine application, so each has its own identity.
kinde-convex-agent-auth is the authorization component. It verifies the caller, makes the decision, and gives back the result the app records.
Mastra runs the agent graph: intake, then review, then disposition.
Langfuse traces how the run executed. It is not the audit record.

The demo page. The scenario states the problem in plain language, the legend names the three agents and what each may do, and the "Demo control" flips the one server mode. Login is optional — an anonymous visitor gets the full page.

The two modes, decided on the server

The demo has two modes. Broken mode writes the blind log and checks nothing. Enforced mode verifies the agent and records the decision. The mode is a server value. A request can never choose its own mode, because that would let an agent pick the mode that lets its action through.

The resolveAuthzMode function reads the mode from one global setting, then from the deployment environment, and never from the request:

// apps/web/convex/authzMode.ts
export async function resolveAuthzMode(ctx: QueryCtx): Promise<AuthzMode> {
  const row = await ctx.db.query('demoSettings').first();
  if (row !== null) {
    return row.authzMode;
  }
  return process.env.AUTHZ_MODE === 'enforced' ? 'enforced' : 'broken';
}

The demo page has a toggle that flips this one global value, so a reader can switch modes and watch the result change. The toggle sets the global mode. It does not attach a mode to a request. The action path always reads the resolved global value.

The single action entry point

Every record action goes through one function, performAction. It reads the mode first. In broken mode it writes the blind row. In enforced mode it runs the full path: verify, start the run instance, authorize, then record.

// apps/web/convex/agentActions.ts
const mode = await ctx.runQuery(internal.authzMode.readAuthzMode);
if (mode === 'broken') {
  return await ctx.runMutation(internal.agentActions.performBroken, {
    orgCode: args.orgCode,
    actorAgentId: args.actorAgentId,
    action: args.action,
    recordId: args.recordId,
    title: args.title,
    kind: args.kind
  });
}

// ENFORCED. An invalid or missing token throws. A denial does NOT throw —
// it is returned as decision.allowed === false and recorded.
if (args.token === undefined || args.token.length === 0) {
  throw new Error('enforced: missing bearer token');
}

The comment states the rule that matters. A bad token is an error. A lawful deny is not an error.

Verify, then authorize

Enforced mode does three things before it records anything. It verifies the token, it starts a run instance for the agent, and it authorizes the action.

// apps/web/convex/agentActions.ts
// 1. Verify the token to resolve the registered agent (throws if invalid).
const verified = await agentAuth.verifyCaller(ctx, args.token, {
  expectedOrgCode: args.orgCode
});
if (verified.agentId === null) {
  throw new Error('enforced: token maps to no registered agent');
}

// 2. Start the run instance, only after token verification.
const instanceId = await ctx.runMutation(internal.agentAuth.startInstance, {
  agentId: verified.agentId,
  runId: `${args.correlationId}:${verified.agentId}`,
  actingForSubject: args.actingForSubject ?? verified.subject,
  orgCode: args.orgCode
});

// 3. Authorize this action for the instance. authorize() (never authz.can)
//    threads the verified caller in, so the decision is bound to this agent.
const {caller, decision} = await agentAuth.authorize(ctx, args.token, {
  instanceId: instanceId as CanArgs['instanceId'],
  action: args.action,
  enforceTokenScopes: true,
  requireOrgCode: true,
  ...(args.recordId === undefined ? {} : {resource: args.recordId})
});

Two options in the authorize call carry the weight. enforceTokenScopes: true makes the decision use the scopes the token actually carries, so an agent cannot claim a scope its token does not hold. requireOrgCode: true binds the decision to the organization, so a token for one tenant cannot act on another.

The call returns caller and decision. The caller is the verified identity and its scopes. The decision is allow or deny, with a reason and the missing scope on a deny. The app records both.

One row, allow or deny

The commit step performs the action only on allow, then writes one provenance row either way. A deny performs nothing, and it still produces a row, because a denied attempt is part of the audit trail.

// apps/web/convex/agentActions.ts
const resourceId: string = args.allowed
  ? await applyAction(ctx, args.orgCode, args.action, args.recordId, args.title, args.kind)
  : (args.recordId ?? '');

await appendProvenanceRow(ctx, {
  orgCode: args.orgCode,
  correlationId: args.correlationId,
  ts: Date.now(),
  actorAgentId: args.actorAgentId,
  actorSub: args.actorSub,
  authorityRootKind: args.authorityRootKind,
  authorityRootSub: args.authorityRootSub,
  delegationId: args.delegationId,
  effectiveScopes: args.effectiveScopes,
  action: args.action,
  resourceType: 'records',
  resourceId,
  decision: args.allowed ? 'allow' : 'deny',
  ...(args.denyReason === undefined ? {} : {denyReason: args.denyReason}),
  ...(args.requiredScopes === undefined ? {} : {requiredScopes: args.requiredScopes})
});

Compare this with the broken path, which writes a row with no authority at all:

// apps/web/convex/agentActions.ts — broken mode
await ctx.db.insert('activityLog', {
  orgCode: args.orgCode,
  actorAgentId: args.actorAgentId,
  action: args.action,
  resourceType: 'records',
  resourceId,
  ts: Date.now()
});

The broken row has an actor label, an action, a target, and a time. It has no verified identity, no scopes, and no decision. That is the blind log from the start of this article.

The blind log in broken mode. Both deletes land as the same shape — an actor label, the action, the file, a time — each marked "no authority recorded." Nothing on either row says which delete was allowed. The authority panel below stays empty.

The hash chain

The provenance row carries two hashes, prevHash and rowHash. The append function canonicalizes the row with RFC 8785 JCS before it hashes, so the same row always produces the same bytes:

// apps/web/convex/provenance.ts
async function computeRowHash(
  rowBody: Record<string, unknown>,
  prevHash: string
): Promise<string> {
  const canonical = canonicalize(rowBody); // RFC 8785 JCS, never JSON.stringify
  if (canonical === undefined) {
    throw new Error('provenance: canonicalization failed');
  }
  return await sha256Hex(canonical + prevHash);
}

The append reads the last row inside the same transaction, so the chain stays gapless and ordered even under concurrent writes:

// apps/web/convex/provenance.ts
const last = await ctx.db
  .query('provenance')
  .withIndex('by_org_seq', (q) => q.eq('orgCode', input.orgCode))
  .order('desc')
  .first();
const seq = last === null ? 0 : last.seq + 1;
const prevHash = last === null ? GENESIS : last.rowHash;

The check reverses the process. It walks the rows in order, confirms each sequence number, confirms each prevHash, recomputes each rowHash, and reports the first row that does not match:

// apps/web/convex/provenance.ts — verifyChain
let prev = GENESIS;
for (let i = 0; i < rows.length; i++) {
  const row = rows[i];
  if (row.seq !== i) {
    return {ok: false, brokenAtSeq: row.seq, reason: `seq_gap (expected ${i})`};
  }
  if (row.prevHash !== prev) {
    return {ok: false, brokenAtSeq: row.seq, reason: 'prev_hash_mismatch'};
  }
  // ... recompute the row hash from its stored fields ...
  if (recomputed !== row.rowHash) {
    return {ok: false, brokenAtSeq: row.seq, reason: 'row_hash_mismatch'};
  }
  prev = row.rowHash;
}
return {ok: true, length: rows.length};

The Verify integrity button on the demo page calls this function. A clean chain returns the row count. A changed row returns the sequence number where the break starts.

The trust boundary

There is one more rule that makes the record trustworthy. The agents cannot reach the database directly. If an agent could write to Convex, it could write its own provenance row and forge the decision. So the agents run in a separate package that has no access to Convex at all. The agents reach the app only over HTTP, with a bearer token.

The client the agents use makes this explicit. The token and the delegation are required inputs, with no default:

// packages/api-client/src/index.ts
export interface LockerClientOptions {
  /** Bearer token for the agent's Kinde M2M identity. Required — no default. */
  agentToken: string;
  /** HMAC-signed delegation grant scoping what the agent may do. Required. */
  delegation: string;
  /** Base URL of the app's HTTP API. Optional; falls back to same-origin. */
  baseUrl?: string;
}

The agent also never states its own authority. It asks the app to perform an action. The app derives the authority from the verified token, not from the request body:

// packages/api-client/src/index.ts
// The agent NEVER states its authorization here — in enforced mode the app
// derives authority from the verified token/delegation, not this body.
export interface ActionRequest {
  orgCode: string;
  actorAgentId: string;
  action: RecordAction;
  correlationId: string;
  recordId?: string;
  title?: string;
  kind?: string;
}

A rule that lives only in a comment is not a rule. So the build enforces the boundary in continuous integration. A script fails the build if any file in the agents package imports Convex or the app:

// scripts/check-boundaries.mjs
const FORBIDDEN = [
  {label: 'convex', test: (s) => /(^|\/)convex(\/|$)/.test(s)},
  {label: 'apps/web', test: (s) => /(^|\/)apps\/web(\/|$)/.test(s)},
  {label: '_generated', test: (s) => s.includes('_generated')}
];
// ... scan every source file under packages/agents for these imports ...
if (violations.length > 0) {
  console.error('✖ Boundary check FAILED — agents may not import the app or Convex.');
  process.exit(1);
}

The CI workflow runs this check on every pull request, next to the type check, the lint, and the tests:

# .github/workflows/ci.yml
- name: Boundary check (packages/agents must not import convex/apps/web/_generated)
  run: npm run boundary-check
- name: Test
  run: npm test

The boundary is a gate, not a convention. The build fails if an agent ever reaches past the HTTP client.

Broken and enforced, side by side

The same run, in the two modes, on the same two files, produces two very different records. The review agent targets Halvorsen v. Meridian Logistics. The disposition agent targets Project Cormorant — internal strategy memo.

Broken mode, the blind log:

agent	action	file	authority recorded?
review	records:delete	Halvorsen v. Meridian Logistics	none
disposition	records:delete	Project Cormorant memo	none

The two rows look the same. The log cannot tell the illegal delete from the lawful one.

Enforced mode, the authority record:

agent	action	file	decision	reason	performed?
review	records:delete	Halvorsen v. Meridian Logistics	deny	insufficient_scope	no
disposition	records:delete	Project Cormorant memo	allow	(none)	yes

Now the rows are different. The review agent's attempt is a recorded deny, and Halvorsen v. Meridian Logistics is still there. The disposition agent's delete is a recorded allow. The auditor can read the authority from the record.

The two screenshots above are the contrast, on the same two files. In broken mode the blind log fills and the authority panel stays empty. In enforced mode the authority panel fills — one DENY, one ALLOW — while the denied file survives.

Three things to take away

Record the decision, not the action. A log of actions cannot answer an auditor. A record of decisions, each with a verified identity and the exact permission, can.
Make the authority impossible to forge. The agent must not verify itself, and it must not write its own record. A separate component makes the decision, and a separate store holds the row behind a trust boundary that the build enforces.
Make the record tamper-evident, and state its limits. A hash chain catches a changed row. Name the stronger steps you did not take, such as external anchoring, so the record is honest about what it proves.

Try it

The demo runs locally, and the static page opens without an account.

Live page: https://evidence-locker-demo.vercel.app
Source: https://github.com/kinde-starter-kits/evidence-locker-demo
Authorization component: kinde-convex-agent-auth

The deployed page shows the scenario, the toggle, the mode banner, existing rows, and the Verify integrity check. It does not run the agents, because the agent run is a server-side worker that does not fit a serverless function. To watch a full run, use the source:

npm ci
cd apps/web && npx convex dev            # keep running; seeds the deployment URL
npx convex run seed:seedLocker '{"orgCode":"orgA"}'
npm run dev                              # http://localhost:3000

Flip the toggle to Broken, run the agents, and read the blind log. Then flip to Enforced, run again, and read the authority record. Click Verify integrity and watch the chain check itself.

To prove the whole story headless, in both modes, with no browser and no live Kinde, run the end-to-end script:

npm run e2e

It seeds one org, runs the agents in broken mode and then enforced mode, and asserts each step: the blind log cannot tell the two deletes apart, the review delete is denied and the file survives, the disposition delete is allowed, and the hash chain verifies. That is the difference between a log that records what happened and a record an auditor would accept.