Your API is being enumerated by a client with a perfectly valid token

Here is a request:

GET /users/4821
Authorization: Bearer eyJhbGciOiJSUzI1NiIs...

The token is valid. It is not expired, the signature verifies, the issuer is
yours. The client it belongs to: a third-party integration your customer
installed eighteen months ago, has the users:read scope, and this endpoint
requires exactly that. The user record 4821 exists and is one your customer's
organisation is entitled to see.

Every authorization check you have passes. They should pass. The request is,
individually, correct.

Now here is the problem: that integration was compromised last Tuesday, and this
is request 40,000 of a sweep through your entire user table.

Authorization is the wrong tool, structurally

The instinct is to reach for authz. Tighten the scopes. Add object-level checks.
Adopt whatever the latest OWASP API list calls it this year.

None of it helps, and it's worth being precise about why. Authorization is a
per-request predicate: given this principal, this action, this object, yes or
no? A compromised-but-authenticated integration satisfies that predicate on every
single request, because it is using exactly the access it was legitimately
granted. There is no individual request you can point at and call wrong.

What's wrong is the sequence. Forty thousand individually-correct requests,
in a particular order, at a particular rate, hitting a particular distribution of
object IDs, is an exfiltration. But "the sequence is wrong" is not a statement
per-request authorization is capable of making. It doesn't have the shape.

This is a general principle worth internalising: you cannot detect an attack
whose signal lives at a scale your check doesn't operate at. Rate limiting
operates per-client-per-minute, so it catches bursts. Authorization operates
per-request, so it catches privilege violations. Enumeration by a valid
credential lives at neither scale.

What's left: the shape of the sequence

If you can't judge requests individually, judge them collectively. Aggregate
requests by credential over a window of time, compute features describing the
shape of that window, and score the shape.

Concretely, the thing I built (apileak, TypeScript, runs under
docker compose) aggregates every request by client_id into tumbling windows
of 1 minute, 5 minutes and 1 hour, and computes ~20 features per window:

• how many distinct object IDs were touched
• what fraction of them had never been touched before
• what fraction of requests were misses
• whether the IDs walked in order
• the coefficient of variation of the inter-request gaps
• how many source IPs and ASNs the credential appeared from
• whether any decoy object was touched

...and so on. A score comes out; a policy turns the score into a decision.

That is the whole thesis. Everything else is detail, but the detail is where all
the interesting failures live, which is why this series is eight articles and not
one.

The architecture, briefly

  agents ──► api-gateway ──► resource-api
               │  ├──► OPA          (decision; reads pushed risk scores as data)
               │  └──► Redis        (fast-path counters, sub-second guardrails)
               └──► Redpanda: access.events / token.events
                         │
                   feature-pipeline   (windowed aggregation, 1m/5m/1h)
                         │  └──► ClickHouse (events, features, scores)
                   scorer (guardrails + statistical + isolation forest)
                         ├──► OPA data push
                         └──► WebSocket ──► CLI + dashboard

Two properties of this shape matter more than the component choices.

Scoring is asynchronous. No machine learning runs in the request path. The
gateway consults a precomputed score plus its own cheap in-Redis counters, so
its p99 overhead stays under 15ms: asserted in a load test, because a detection
layer that adds 200ms to every request will be switched off by whoever is on call
the first time latency alerts fire.

The gateway has a fast path of its own. Windowed scoring cannot react faster
than one window. A client that opens with 600 requests in fifteen seconds needs
stopping before the first window closes, so the gateway keeps rolling per-minute
counters (a request count, a miss count, a HyperLogLog of distinct IDs) in Redis
and the policy can fire on those alone. In practice this matters enormously: in
one run the dictionary-attack profile was enforced at 9 seconds by the fast
path and only flagged at 44 seconds by the windowed scorer. Enforcement
preceding detection is not an anomaly, it's the fast path doing its job.

What makes this hard is not the attackers

If the only requirement were "catch someone reading the whole user table", this
would be a weekend project. Set a threshold on distinct object IDs per hour, page
someone, go home.

The requirement that makes it hard is the other one: a legitimate bulk
operation must not trip it.

Consider a post-deploy backfill. Your customer's integration ships a new feature
and needs to populate a cache, so it reads 9,000 user records it has never read
before, over five minutes, as fast as your API will serve them.

Now compare it to an attacker dumping the same table:

	backfill	enumeration
distinct objects touched	thousands	thousands
share never seen before	~100%	~100%
repeat ratio	~0	~0
request rate	elevated	elevated
scope used	exactly as granted	exactly as granted

On every metric you'd reach for first, they are identical. I tried blocking
on cardinality anyway, with corroboration from other signals. It fired on backfill
windows whenever the timing jitter happened to dip. That approach is not
salvageable by tuning; the signals genuinely do not carry the distinction.

Finding signals that do carry it is the subject of article 3, and it turns on a
simple reframing: stop asking how much was read and start asking how it was
accessed. A backfill reads records in its own key order and they all exist. An
enumerator walks IDs, or guesses them, or probes for ones that don't.

What "detection" has to mean

One more constraint, and it changes the design more than anything else.

Detecting the attack is useless if the attacker can tell they were detected. An
attacker who learns that request 40,000 was blocked simply changes tactics,
slows down, rotates credentials, switches to a different ID scheme, and you have
converted a detection into a training signal for them.

So every blocking decision in this system returns a response byte-identical to
an ordinary 404: same status, same body, same headers, same timing. The
attacker's requests just start coming back as "that object doesn't exist". This
constraint reaches surprisingly deep: it's why the 404 and 403 in the
resource API are constant-time and byte-identical to each other (there's a test
that measures both distributions and fails if they're distinguishable), and it's
why honeytoken recognition happens at the gateway rather than in the resource API,
so that the resource API's response never varies at all.

Article 7 covers what covert enforcement does to a graduated response ladder,
including the two rungs that turned out to be doing nothing for months.

What this series covers

1. This article, why authorization can't help, and what's left.
2. The cast: the attacker profiles, and the much harder problem of modelling legitimate traffic convincingly enough to trust your own false-positive rate.
3. Features, which signals discriminate, which only look like they do, and the ones deliberately computed but never scored.
4. Scoring: three layers, why they combine by maximum rather than sum, and why per-feature attribution is the whole point.
5. Baselines: four separate ways the baseline quietly destroyed the detector built on top of it. The most transferable article in the series.
6. Honeytokens: decoys derived so they recognise themselves and attribute their own trips, plus rotation and the limits of allowlisting.
7. Enforcement, a six-rung ladder where every rung has to be invisible.
8. When your detector lies to you: the bug class that dominated this project: components that do nothing, raise no error, and leave every number looking plausible.

A warning about what this is: a demonstration artifact, built to be read and
argued with. It is not a product. Where it doesn't work I've said so, in the
articles as well as the repo: the failures turned out to be the most useful
material.