Simulating attackers is easy. Simulating legitimate users is the hard part

In part 1 I argued that enumeration from a valid credential can only be
caught from the shape of the request sequence, and that the hard requirement is
not catching attackers but not catching everyone else.

That requirement has a consequence people skip: to have any confidence in a false
positive rate, you need legitimate traffic that is genuinely hard to distinguish
from an attack. Traffic that is obviously benign proves nothing. The detector in
this project drives ten synthetic clients through one gateway, all holding real
tokens with identical scopes, and only five of them are attackers.

Here is the whole cast, and what each one is for.

The legitimate integration

integration-acme models a real third-party integration:

• Zipf-distributed reads of a hot object set: real integrations re-read the same handful of records constantly, they don't sample uniformly
• human-ish jitter between requests (300–1700ms), because a real system has queues and retries and network variance
• a small burst at the top of each hour from a batch sync
• occasional genuine 404s from stale references it still holds
• on-behalf-of reads: when it fetches user 812's record it does so on behalf of user 812

That last point matters and I'll come back to it in article 3.

And then, deliberately, the thing that makes the whole project difficult:

• a benign bulk backfill, a post-deploy job that reads 9,000 records it has never touched, over five scenario-minutes, as fast as the API will serve them.

This is the false-positive stress test, and every detection rule has to survive
it. As shown in part 1, on cardinality, novelty ratio and repeat ratio a backfill
is indistinguishable from an attacker dumping the same table.

The demo asserts integration-acme stays in allow for the entire run, backfill
included. Zero blocked requests. If that assertion ever fails, the detector is
worthless regardless of what it catches.

The attackers

Five profiles, all using a credential of the same type and scopes as the real
integration. None of them exploit a bug; every one is using access it legitimately
holds.

sequential: walks integer IDs in order, fast and machine-regular. The
naive attacker. Trips the fast-path cardinality guardrail and the windowed
sequentiality guardrail within seconds. It's in the cast mainly as a control: if
your detector can't catch this, stop reading.

dictionary: probes /users?email= from a wordlist of plausible corporate
addresses. Structurally different: there is no ID ordering to detect, so
id_sequentiality never fires. What gives it away is the miss ratio (almost every
guess is wrong) and the route mix pivoting onto the search endpoint. About one
guess in twenty lands on a real account, which is exactly why the profile is worth
running: a blind guesser still harvests real data.

slow-and-low: one request every 30 seconds, random IDs across a large
range so ~99% miss. Its whole design is to stay below every rate threshold. It
must be caught by accumulation over the coarser windows, never by rate. This
profile later exposed the most consequential bug in the project (article 8).

distributed: one credential replayed from six IPs across six ASNs,
re-authenticating from each. Per-origin rate limiting sees nothing: each IP
issues a trickle. The credential is the only thing tying the requests together,
which is precisely the axis the detector aggregates on.

mimicry: the hard one. Enumeration hidden inside replayed legitimate
traffic on the same credential: for every probe it issues several requests that
look exactly like the real integration's work, with the same Zipf re-reads, the
same jitter, the same route mix. Rate rules see nothing, the miss ratio is diluted
below the guardrail threshold, and interleaved hot reads destroy the ordering
signal.

Measured against the detector as it stood before I designed for it: mimicry
scored zero on every single window. Not "low": zero. Every feature described
the shape of one window, and in any one window mimicry is legitimate traffic.
Article 3 covers the feature that finally caught it, which had to be about
duration rather than shape.

obo-sweep: a compromised integration holding one genuine delegation. Every
request carries a valid token, valid scopes, and a subject it really is entitled
to act for. It uses that single delegation to read everybody else's records. No
enumeration signal fires at all: the IDs are valid so there's no miss ratio, they
are strided so there's no ordering, the rate is ordinary. This one is caught by a
signal nothing else in the system provides.

honeytoken breach: requests a decoy object from a cold start, with zero
accumulated score. Tests the instant-revoke path (article 6).

The two that aren't attackers, and aren't quite normal either

These are the interesting ones, and both were added late, after the detector
started behaving in ways I couldn't explain.

The ambiguous automation

integration-paced is legitimate. It re-reads a small fixed set of records it
is entitled to, never probes a missing ID, never walks IDs in order. What makes it
look wrong is that it is obviously a machine, rigid timing, zero jitter, running
from several egress hosts at once, and gradually adding more of them as it scales
out. A monitoring bot, or a sync worker being horizontally scaled.

That is genuinely ambiguous evidence, and it exists to answer a question the rest
of the cast doesn't ask: what should the system do when it isn't sure?

The answer should not be a block. It climbs log → throttle → step_up, keeps
working the whole time, and is never hard-blocked. When it reaches step_up it
answers the challenge by re-authenticating: transparently, because that's what
any competent OAuth2 client does when its token is rejected, and carries on.

It is also, honestly, a false positive, and the demo counts it as one. A
detector that never admits to any is not being measured.

The internal scanner

internal-scanner is your own security team's tooling, and it exists because of
a specific failure mode: a scanner's job is to probe for exactly the things
honeytokens are: identifiers that shouldn't resolve. Without an exemption, its
first sweep trips a decoy and revokes your own security team's credential, and
the failure looks like a successful detection.

So honeytoken recognition is allowlisted for scanner identities. Verified in a
run: 21 decoy requests, 0 recognised, 0 honeytoken hits, never revoked, while
the non-allowlisted breach client tripped 3 hits and was revoked immediately.

But the exemption turned out to be narrower than its name suggests, and the demo
now says so out loud. The scanner is still denied, for sweeping an ID range,
which is the mimicry signature. That's correct: a scanner sweeping for
non-existent records is genuinely indistinguishable from an enumerator. The
operational lesson is worth more than a green tick: allowlisting an identity for
honeytokens buys it no blanket immunity, and scanners need exempting at the policy
level too.

Why the boring clients matter more

If I'd built only the attackers, I would have shipped a detector that scores 100%
recall and is unusable.

Three of this project's most significant bugs were found only because a
non-attacking client was in the cast:

• The ambiguous automation existing at all is what forced me to check whether the middle tiers of the response ladder did anything. They didn't, throttle and step_up were being decided, recorded, displayed on a dashboard, and then the request was served anyway.
• Trying to make that client land reliably in the middle band exposed four separate baseline defects (article 5), including one where changing nothing but the request interval flipped the outcome between allow and step_up.
• The scanner exposed that a defensive feature wired into the gateway since the beginning had no test and no traffic exercising it.

None of those are findable with attackers alone. Attackers make the detector fire;
only realistic non-attackers tell you whether firing meant anything.

Determinism

Everything is seeded. Every agent takes an explicit RNG seed, the dataset is
generated from a fixed seed, and the isolation forest is seeded per window size.
Two runs of the same scenario produce the same traffic.

This matters more than it sounds. When you are chasing a detector that behaves
differently between runs, you need to know the traffic is not the variable.
That confidence is what let me establish, in article 5, that an inexplicable
flip-flop between allow and step_up was a defect in the baseline rather than
noise in the input.

What is not deterministic is timing under load, and I'll be honest about it here
because it bit repeatedly: all ten agents share one Node event loop. On an idle
machine the sequential profile is caught in 29–50 scenario-seconds; on a box at
load average 8.5 the same code reports 74–76 and fails its bound. That's the
harness, not the detector, but it means "did my change cause a regression?" has
to be answered on a quiet machine.

---

Next: which features actually discriminate, why the most obvious ones don't,
and the ones this system computes and deliberately never scores.