A graduated response ladder where every rung is invisible

Detection produces a number. Something has to turn that number into a response,
and the response has two hard constraints that pull against each other:

1. It must be proportionate. A score of 55 is not a score of 95, and treating them the same means either blocking clients you shouldn't or serving attackers you shouldn't.
2. It must be invisible. An attacker who learns they were detected changes tactics, and you've converted a detection into a training signal for them.

The second constraint rules out most of the obvious implementations of the first.

The ladder

Six tiers, evaluated in Rego, driven by the pushed risk score plus the gateway's
own fast-path signals:

allow → log → throttle → step_up → deny → revoke

The policy is small enough to read in full, and mirrors the scorer's thresholds
exactly:

score_tier := "deny"     if { score_fresh; score_entry.score >= 85 }
score_tier := "step_up"  if { score_fresh; score_entry.score >= 70; score_entry.score < 85 }
score_tier := "throttle" if { score_fresh; score_entry.score >= 50; score_entry.score < 70 }
score_tier := "log"      if { score_fresh; score_entry.score >= 30; score_entry.score < 50 }

final_tier := fast_tier  if { tier_rank[fast_tier] >= tier_rank[score_tier] }
final_tier := score_tier if { tier_rank[score_tier] > tier_rank[fast_tier] }

The final decision is the more severe of two independent signals: the scorer's
composite, and the gateway's own sub-second guardrails. The second exists because
windowed scoring cannot react faster than one window, and a flood needs stopping
before then.

Fail open, deliberately

# FAIL-OPEN: when the risk-score data is missing or stale we allow (with a
# logged reason) rather than deny.

This is the choice most likely to get an argument, so here's the reasoning.

This is a detection layer bolted onto a production API. If the scorer, the
pipeline, or the data path between them breaks, failing closed takes the real
API down for every legitimate integration: a self-inflicted outage far more
damaging than the marginal enumeration an attacker achieves during the gap.

The tradeoff is explicit and bounded: the gateway's fast-path guardrails
(cardinality, miss storm, honeytokens) are independent of the score data and keep
working when it's stale. Failing open on the statistical layer is not failing
open entirely.

Covert enforcement

Here's where the second constraint bites.

Every blocking decision returns a response byte-identical to a normal upstream
404: same status, same body, same headers. The attacker sees the objects they
request "not existing".

export const MISS_BODY = JSON.stringify({ error: 'not_found' });

One body, used for a genuine 404, a scope-violating 403, and a covert block
alike. This reaches deeper than it first appears:

• The resource API's 404 and 403 are byte- and timing-identical to each other, with a test that measures both distributions and fails if they're distinguishable. A 403 that's distinguishable from a 404 is an existence oracle: "this object exists but you can't see it" is exactly the information an enumerator wants.
• Honeytoken recognition happens gateway-side (part 6) so the resource API's response never varies at all.
• Ownership metadata is returned on hits and misses, empty when absent, for the same reason.

The two rungs that did nothing

For a long time this system decided throttle and step_up, recorded them,
displayed them on a dashboard, and then served the request anyway. Only deny
and revoke were enforced.

A six-rung ladder with two decorative rungs is not a graduated response. It's a
binary one with extra logging.

Implementing them meant answering: what do these mean when the client must not
notice?

throttle is a budget, not a delay

The instinct is to slow the client down. That's wrong here for two reasons: added
latency is directly measurable by an attacker timing their own requests, and it
defeats the constant-time work in the resource API.

So throttle serves a budget of 60 requests per rolling minute and returns the
same byte-identical miss beyond it:

if (input.tier === 'throttle' && input.reqCount1m > THROTTLE_BUDGET_PER_MIN) {
  return 'miss';
}

The budget sits comfortably above the legitimate integration's ~60 req/min,
because throttle is the middle of the ladder where a false positive is still
plausible. It cuts an enumerator's extraction rate by an order of magnitude while
barely touching a client that landed there by mistake.

step_up is indistinguishable from token expiry

A step-up challenge that announces itself is useless. So instead: require a token
minted after the challenge was raised, and treat anything older as an ordinary
expired token.

Re-authentication is routine for a client_credentials integration, tokens live
300 seconds here, so a 401 at this point is indistinguishable from the expiry the
client already handles. The legitimate client re-mints and never notices it was
challenged.

What step-up buys is a distinction no other tier can draw: between holding a
stolen bearer token and holding the credentials. A token thief cannot mint a
replacement and stops dead.

It does not stop an attacker with the client secret, and every attacker in
this demo is modelled that way: they simply re-authenticate. Worth stating
plainly rather than dressing up. The residual value is that forced re-issuance
drives the token-issuance rate up, which the scorer weights: the mitigation feeds
the detector even when it fails to block.

That claim went untested for far too long, which was the wrong way round for the
argument the whole tier rests on. It's now pinned directly:

it('lets the credential holder through and stops the thief permanently', async () => {
  tier = 'step_up';
  tokenIssuedAtMs = stolenTokenIssuedAt;
  expect((await get(app)).statusCode).toBe(401);      // challenge raised

  tokenIssuedAtMs = stolenTokenIssuedAt + 5_000;      // re-authenticated
  expect((await get(app)).statusCode).toBe(200);

  tokenIssuedAtMs = stolenTokenIssuedAt;              // thief, same token
  for (let i = 0; i < 5; i++) {
    expect((await get(app)).statusCode).toBe(401);
  }
});

Plus one more: the stolen token stays refused even after the client's tier drops
back, or a thief could simply wait out the score.

Two ordering bugs, both found by getting them wrong

The challenge cleared itself. JWT iat has one-second resolution, so "was
this token minted after the challenge?" is ambiguous within the raising second.
My first implementation granted a second of grace, which accepted the very token
the challenge was meant to retire. The challenge raised and satisfied itself on
the same request.

The fix is to round the challenge timestamp up to the next whole second.
Ambiguity resolves in the safe direction: a token from the challenge's own second
is treated as too old, and the client mints another.

A recoverable 401 pre-empted a hard block. An earlier version answered an
outstanding challenge immediately, before the upstream call. That both leaked
timing: a challenged request returned faster than a served one, and let a
recoverable 401 take precedence over a deny.

Now nothing short-circuits. Every post-authentication request takes the same
path, proxy upstream, then overwrite the response, so served, throttled, denied
and challenged requests are indistinguishable by timing. That's the same pattern
deny already used; I'd introduced an inconsistency rather than following it.

A challenge is not a block

One measurement distinction that turned out to matter a lot.

The demo's ambiguous automation reaches step_up, is challenged, re-authenticates
transparently, and carries on, 597 of 604 requests served. Early on my assertion
counted step_up responses alongside deny as "blocked", so this reported as a
failure.

It was reporting the ladder working correctly as though it had blocked an
innocent client.

denied and challenged are now separate columns, and the verdict asserts the
client was never denied:

✓ PASS  ambiguous automation escalated to log/throttle/step_up but was never denied
        (5 challenges, all recovered), and still had 598/603 requests served

Being challenged and recovering is the ladder succeeding. If your metrics can't
express that difference, a graduated response will look like a false-positive
generator and someone will simplify it back into a binary one.

What the ladder is for

The demo's ambiguous client is legitimate. Rigid machine timing, several egress
hosts, scaling out. Genuinely ambiguous evidence.

It climbs log → throttle → step_up, keeps every request served, and is never
hard-blocked. It's an honest false positive, and the demo counts it as one.

That's the argument for graduated response in one client: you will be wrong
about someone, and the design question is what happens when you are. A binary
detector answers "cut them off". A ladder answers "slow them down, ask them to
prove it, and let them carry on when they do", which is survivable for both
sides.

---

Next, and last: when your detector lies to you: the bug class that dominated
this project, where a component does nothing, raises no error, and leaves every
number looking plausible.