DEV Community: DarkEdges

My detector caught the attacker and never once stopped it, and reported PASS

DarkEdges — Fri, 07 Aug 2026 09:57:39 +0000

The most consequential bug in this project had been there since the beginning,
survived several full end-to-end runs, and was reported as a PASS every time.

✓ PASS  slow-and-low detected within 30m (7.3m), never exceeding legit rate

That line is true. The scorer flagged it correctly, well inside the bound. What
the line doesn't say is that the attacker was served every single request it
ever made. Zero non-allow decisions, across the entire scenario. Detected and
never once stopped.

This is one instance of a pattern that accounts for more real bugs in this
project than every other cause combined: a component contributes nothing, no
error is raised, and every surrounding number stays plausible.

The bug

The scorer computes windows at 1m, 5m and 1h, and publishes each result to a
per-client key in Redis and OPA.

Each result. To the same key.

So the last writer won. And 1-minute windows close most often, so they always
won.

slow-and-low issues about two requests a minute. Its 1m windows fall below the
minimum request count and score zero. Its 5m and 1h windows accumulate the miss
ratio that earns a deny. Every one of those zeroes immediately overwrote the
deny.

The entire premise of a multi-scale pipeline, that different attacks are visible
at different scales, was silently violated by the publication step. Any detection
that only appeared at a coarser scale was discarded.

The fix is a roll-up: publish the most severe verdict across window sizes within
the freshness horizon the policy already uses. Afterwards, the same attacker is
denied on 24–31 of its 44 requests.

Why it survived so long

Because the report could not express it.

Detection latency was computed as the earlier of two very different facts: the
scorer's first non-allow window, and the gateway's first non-allow decision.
Printed under one heading, detected, a client that was noticed but never
touched looked identical to one that was noticed and blocked.

A report that averages over the distinction you are trying to verify cannot
verify it.

The report now derives two facts from two sources and asserts both:

detected: the scorer's first non-allow window, from risk_scores
enforced: the first request the gateway applied a non-allow tier to, from the access log, which is the record of what actually happened

✓ PASS  slow-and-low responded within 30.0m (6.3m; scorer 6.3m, gateway 6.5m)
✓ PASS  slow-and-low ENFORCED — 31/44 requests stopped

Splitting them paid for itself immediately, and not only on the bug it was built
for. Look at the dictionary profile:

✓ PASS  dictionary responded within 60s (9s; scorer 76s, gateway 9s)

Enforced at 9 seconds, detected at 76. The gateway's fast path stopped it long
before the windowed scorer produced a single verdict. Enforcement preceding
detection is not an anomaly: it's the fast path doing its job, and the old
report was structurally incapable of showing it.

The same shape, five more times

Once I had a name for it, it was everywhere.

Tables with no writer. access_events: the raw log the whole "every decision
is reconstructable" claim rests on, was a schema definition and a materialized
view with no producer. Discovered only when I built a dataset export on top of it
and it came back empty. Then detection_labels, same thing. Then
client_minute_card, a materialized view aggregating cardinality per client per
minute, populating on every single event, queried by nothing.

An environment-dependent derivation. The label-joining query took its window
length from WINDOW_MS, which is divided by DEMO_SPEED. The demo sets that to
8; the export script doesn't set it at all. So the export bucketed events into
60-second windows against 7.5-second data and matched almost nothing: 172 of 197
labels silently vanished, and the CSV still looked perfectly well-formed, just
smaller.

The fix generalises. Derive window lengths from the data, since
client_features records both ends of every window, and never from the
environment that happens to be reading it.

Untyped tests. The package builds exclude **/*.test.ts, correctly, so tests
aren't emitted into dist/. But that also excluded them from type checking.
Three times a test fixture went stale after a field was added: the missing
property became undefined, Number(undefined) became NaN, and the failure
surfaced as an apparent detector regression.

The verification is the part worth keeping. Delete one field from one fixture:

$ tsc -p tsconfig.tests.json
ml.test.ts(17,3): error TS2719: Types of property 'route_profile' are
  incompatible. Type 'string | undefined' is not assignable to type 'string'.

$ vitest run packages/scorer/src/ml.test.ts
      Tests  15 passed (15)

The type checker names the field and the file. The test suite reports fifteen
passes. A test asserting on a NaN that flows through the arithmetic without
throwing is not a test failure: it's a test quietly measuring nothing.

A component that produced nothing at all. At DEMO_SPEED=8 a 1-hour window is
450 real seconds and the scenario runs for 30 scenario-minutes, half of one
window. Whether any 1h window closes depends on where the run falls relative to
a boundary. Most runs produce zero. The isolation forest can never fit a 1h model.
A third of the advertised 1m/5m/1h pipeline contributes nothing, and said
nothing about it.

A duplicated definition that drifted. The dashboard keeps its own list of
which clients are hostile, because it's a separate build. That copy drifted twice
while I was adding clients, and the failure is quiet in the dangerous direction:
an unlisted attacker defaults to "legitimate", so its every escalation is scored
as a false positive and the confusion matrix reports numbers that never happened.

Why this class specifically

Two structural reasons, and both are common.

The pipeline swallows errors. Insert failures are caught and dropped, because
a detection layer should not take down the API it protects. That's the right
call, and it converts every write failure into silence.

The report was derived from the components it was meant to verify. Detection
latency came from the scorer's own records. If the scorer says it flagged
something, the report said it was detected. The one thing that could have
contradicted it, what the gateway actually did, wasn't consulted.

What actually helped

Make absence visible. The report now prints how many windows each size
produced, and says so explicitly when one produced none:

labelled windows (detection_labels): 197   [1mx160  5mx37  1hx0]
  note: 1h produced no verdicts — the scenario is 30 scenario-minutes long,
  shorter than those windows. Run a longer one with e.g. SCENARIO_MIN=130.

Silent absence is indistinguishable from silent breakage. A component that
contributes nothing should say so.

Assert against the independent record. Not the detector's own output. The
access log is what the gateway did; risk_scores is what the scorer concluded.
They are different facts and deserve different assertions.

Guard the guard. The report is now only as trustworthy as the access log it
reads: a new single point of failure. So the agents keep their own request counts
purely to cross-check it, and the report warns when the two disagree.

Pin duplication you can't remove. The dashboard's copy of ground truth can't
be eliminated without restructuring the build graph. The drift can:

it('lists exactly the attack clients, no more and no fewer', () => {
  expect(dashboardAttackers().sort()).toEqual([...ATTACK_CLIENT_IDS].sort());
});

Type-check the code you excluded from the build. tsconfig.tests.json, with
noEmit, wired into pnpm test. It has caught a stale fixture in every round
since.

The question worth asking first

The habit I'd most like to pass on is a change in first instinct.

When something looks off in a system like this, the reflex is "where's the logic
error?" For six consecutive bugs here, the productive question was different:

Which component is silently doing nothing?

Not what is wrong with this calculation, but is this calculation happening at
all. Every one of these bugs produced plausible output. Several produced output
that was internally consistent. One reported PASS on the exact scenario it was
failing.

The corollary, if you build detection systems: your instrumentation is a component
too, and it fails the same way. Ask what your report cannot express, because
that's the shape of the bug you won't find.

That's the series. The code is a demonstration artifact, deliberately readable,
deliberately argued with, and honest about the parts that don't work. The
docs/DECISIONS.md file in the repo is 28 sections, and most of them record
something that was wrong and how it was found. That turned out to be the most
useful thing in it.

A graduated response ladder where every rung is invisible

DarkEdges — Fri, 07 Aug 2026 09:57:18 +0000

Detection produces a number. Something has to turn that number into a response,
and the response has two hard constraints that pull against each other:

It must be proportionate. A score of 55 is not a score of 95, and treating them the same means either blocking clients you shouldn't or serving attackers you shouldn't.
It must be invisible. An attacker who learns they were detected changes tactics, and you've converted a detection into a training signal for them.

The second constraint rules out most of the obvious implementations of the first.

The ladder

Six tiers, evaluated in Rego, driven by the pushed risk score plus the gateway's
own fast-path signals:

allow → log → throttle → step_up → deny → revoke

The policy is small enough to read in full, and mirrors the scorer's thresholds
exactly:

score_tier := "deny"     if { score_fresh; score_entry.score >= 85 }
score_tier := "step_up"  if { score_fresh; score_entry.score >= 70; score_entry.score < 85 }
score_tier := "throttle" if { score_fresh; score_entry.score >= 50; score_entry.score < 70 }
score_tier := "log"      if { score_fresh; score_entry.score >= 30; score_entry.score < 50 }

final_tier := fast_tier  if { tier_rank[fast_tier] >= tier_rank[score_tier] }
final_tier := score_tier if { tier_rank[score_tier] > tier_rank[fast_tier] }

The final decision is the more severe of two independent signals: the scorer's
composite, and the gateway's own sub-second guardrails. The second exists because
windowed scoring cannot react faster than one window, and a flood needs stopping
before then.

Fail open, deliberately

# FAIL-OPEN: when the risk-score data is missing or stale we allow (with a
# logged reason) rather than deny.

This is the choice most likely to get an argument, so here's the reasoning.

This is a detection layer bolted onto a production API. If the scorer, the
pipeline, or the data path between them breaks, failing closed takes the real
API down for every legitimate integration: a self-inflicted outage far more
damaging than the marginal enumeration an attacker achieves during the gap.

The tradeoff is explicit and bounded: the gateway's fast-path guardrails
(cardinality, miss storm, honeytokens) are independent of the score data and keep
working when it's stale. Failing open on the statistical layer is not failing
open entirely.

Covert enforcement

Here's where the second constraint bites.

Every blocking decision returns a response byte-identical to a normal upstream
404: same status, same body, same headers. The attacker sees the objects they
request "not existing".

export const MISS_BODY = JSON.stringify({ error: 'not_found' });

One body, used for a genuine 404, a scope-violating 403, and a covert block
alike. This reaches deeper than it first appears:

The resource API's 404 and 403 are byte- and timing-identical to each other, with a test that measures both distributions and fails if they're distinguishable. A 403 that's distinguishable from a 404 is an existence oracle: "this object exists but you can't see it" is exactly the information an enumerator wants.
Honeytoken recognition happens gateway-side (part 6) so the resource API's response never varies at all.
Ownership metadata is returned on hits and misses, empty when absent, for the same reason.

The two rungs that did nothing

For a long time this system decided throttle and step_up, recorded them,
displayed them on a dashboard, and then served the request anyway. Only deny
and revoke were enforced.

A six-rung ladder with two decorative rungs is not a graduated response. It's a
binary one with extra logging.

Implementing them meant answering: what do these mean when the client must not
notice?

`throttle` is a budget, not a delay

The instinct is to slow the client down. That's wrong here for two reasons: added
latency is directly measurable by an attacker timing their own requests, and it
defeats the constant-time work in the resource API.

So throttle serves a budget of 60 requests per rolling minute and returns the
same byte-identical miss beyond it:

if (input.tier === 'throttle' && input.reqCount1m > THROTTLE_BUDGET_PER_MIN) {
  return 'miss';
}

The budget sits comfortably above the legitimate integration's ~60 req/min,
because throttle is the middle of the ladder where a false positive is still
plausible. It cuts an enumerator's extraction rate by an order of magnitude while
barely touching a client that landed there by mistake.

`step_up` is indistinguishable from token expiry

A step-up challenge that announces itself is useless. So instead: require a token
minted after the challenge was raised, and treat anything older as an ordinary
expired token.

Re-authentication is routine for a client_credentials integration, tokens live
300 seconds here, so a 401 at this point is indistinguishable from the expiry the
client already handles. The legitimate client re-mints and never notices it was
challenged.

What step-up buys is a distinction no other tier can draw: between holding a
stolen bearer token and holding the credentials. A token thief cannot mint a
replacement and stops dead.

It does not stop an attacker with the client secret, and every attacker in
this demo is modelled that way: they simply re-authenticate. Worth stating
plainly rather than dressing up. The residual value is that forced re-issuance
drives the token-issuance rate up, which the scorer weights: the mitigation feeds
the detector even when it fails to block.

That claim went untested for far too long, which was the wrong way round for the
argument the whole tier rests on. It's now pinned directly:

it('lets the credential holder through and stops the thief permanently', async () => {
  tier = 'step_up';
  tokenIssuedAtMs = stolenTokenIssuedAt;
  expect((await get(app)).statusCode).toBe(401);      // challenge raised

  tokenIssuedAtMs = stolenTokenIssuedAt + 5_000;      // re-authenticated
  expect((await get(app)).statusCode).toBe(200);

  tokenIssuedAtMs = stolenTokenIssuedAt;              // thief, same token
  for (let i = 0; i < 5; i++) {
    expect((await get(app)).statusCode).toBe(401);
  }
});

Plus one more: the stolen token stays refused even after the client's tier drops
back, or a thief could simply wait out the score.

Two ordering bugs, both found by getting them wrong

The challenge cleared itself. JWT iat has one-second resolution, so "was
this token minted after the challenge?" is ambiguous within the raising second.
My first implementation granted a second of grace, which accepted the very token
the challenge was meant to retire. The challenge raised and satisfied itself on
the same request.

The fix is to round the challenge timestamp up to the next whole second.
Ambiguity resolves in the safe direction: a token from the challenge's own second
is treated as too old, and the client mints another.

A recoverable 401 pre-empted a hard block. An earlier version answered an
outstanding challenge immediately, before the upstream call. That both leaked
timing: a challenged request returned faster than a served one, and let a
recoverable 401 take precedence over a deny.

Now nothing short-circuits. Every post-authentication request takes the same
path, proxy upstream, then overwrite the response, so served, throttled, denied
and challenged requests are indistinguishable by timing. That's the same pattern
deny already used; I'd introduced an inconsistency rather than following it.

A challenge is not a block

One measurement distinction that turned out to matter a lot.

The demo's ambiguous automation reaches step_up, is challenged, re-authenticates
transparently, and carries on, 597 of 604 requests served. Early on my assertion
counted step_up responses alongside deny as "blocked", so this reported as a
failure.

It was reporting the ladder working correctly as though it had blocked an
innocent client.

denied and challenged are now separate columns, and the verdict asserts the
client was never denied:

✓ PASS  ambiguous automation escalated to log/throttle/step_up but was never denied
        (5 challenges, all recovered), and still had 598/603 requests served

Being challenged and recovering is the ladder succeeding. If your metrics can't
express that difference, a graduated response will look like a false-positive
generator and someone will simplify it back into a binary one.

What the ladder is for

The demo's ambiguous client is legitimate. Rigid machine timing, several egress
hosts, scaling out. Genuinely ambiguous evidence.

It climbs log → throttle → step_up, keeps every request served, and is never
hard-blocked. It's an honest false positive, and the demo counts it as one.

That's the argument for graduated response in one client: you will be wrong
about someone, and the design question is what happens when you are. A binary
detector answers "cut them off". A ladder answers "slow them down, ask them to
prove it, and let them carry on when they do", which is survivable for both
sides.

Next, and last: when your detector lies to you: the bug class that dominated
this project, where a component does nothing, raises no error, and leaves every
number looking plausible.

Honeytokens that recognise themselves: stateless decoys with automatic attribution

DarkEdges — Fri, 07 Aug 2026 09:57:06 +0000

Every signal in the previous articles is statistical. They weigh evidence, they
have thresholds, they can be argued with.

Honeytokens are different in kind. A honeytoken is a record that does not
exist and was never given to anyone. Nothing legitimate can ask for it, because
nothing legitimate has ever held a reference to it. A request for one isn't
suspicious, it's proof that someone is guessing or working from a stolen list.

That makes it the highest-confidence signal available, and worth building
carefully.

Derivation: make the decoy recognise itself

The obvious implementation is a table. Generate decoy IDs, store them, and check
every miss against the table.

That has two problems. It puts a database lookup in the request path on every
miss, and misses are exactly what a flood produces. And it doesn't tell you
whose decoy was tripped without another join.

Instead, derive them:

export function honeytokenFor(clientId: string, n: number, generation = 0): string {
  const mac = createHmac('sha256', config.honeytokenSecret)
    .update(`${clientId}:${generation}:${n}`)
    .digest();
  return uuidFromBytes(mac.subarray(0, 16));
}

Three properties fall out of this, and they're the whole design:

Recognition is stateless. Given any ID and any client, recompute the client's
decoy set and check membership. No lookup, no cache, no round trip. The gateway
precomputes each known client's set at startup into a Set and membership is
O(1).

Attribution is automatic. The client ID is inside the derivation. There is
no "which client did this decoy belong to?" question, a decoy for
integration-acme is not a decoy for anyone else, and cannot be. If a decoy
seeded into acme's scope is requested by a different credential, that's
information too.

They're format-identical to real IDs. The output is shaped as a v4 UUID,
with correct version and variant nibbles, so it is indistinguishable from a real
documents identifier:

export function uuidFromBytes(bytes: Buffer): string {
  const b = Buffer.from(bytes.subarray(0, 16));
  b[6] = (b[6]! & 0x0f) | 0x40; // version 4
  b[8] = (b[8]! & 0x3f) | 0x80; // variant 10
  …
}

If your decoys are distinguishable from real identifiers, they are not decoys.
An attacker who can filter them out gets a free map of what to avoid.

Recognition happens at the gateway, and only on a miss

Two placement decisions, both load-bearing.

Only on a miss. The check runs after the upstream call, only when the
response is ≥ 400. A honeytoken is not in the dataset, so requesting one produces
an ordinary miss from the resource API. Checking before would mean checking every
request; checking after means checking only the ones that could possibly be a
decoy.

At the gateway, not in the resource API. This is the subtle one, and it comes
out of the covert-enforcement requirement from part 1.

If the resource API recognised honeytokens, its response would have to do
something different, and any difference, even in timing, is observable. By
keeping recognition in the gateway, a honeytoken request is, to the resource API,
an ordinary miss. Its response never varies. The gateway then makes a sub-second
deterministic decision on top of a response that carries no signal at all.

if (
  isMiss &&
  match.resource_type === 'documents' &&
  match.resource_id &&
  !isAllowlisted(clientId) &&
  honeyIndex.get(clientId)?.has(match.resource_id)
) {
  honeytokenHit = true;
}

A hit is an immediate revoke, regardless of accumulated score. The demo's breach
client has a score of essentially zero when it trips one, that's the point. This
is the one signal that doesn't need history.

Rotation

rotated_at sat in the schema for a long time with nothing writing to it. That's
worth fixing, because rotation isn't hygiene theatre, it addresses a specific
failure.

A decoy list leaks. Through a backup, a support ticket, a former employee, a
misconfigured export. And a decoy an attacker can identify is worse than none,
because they route around it silently and you now have a tripwire you believe in
that will never fire.

Generations make retirement wholesale: bump HONEYTOKEN_GENERATION and every
decoy changes, because the generation is inside the HMAC input.

The design decision worth arguing with is what happens to the old set. The
gateway keeps recognising the previous generation even though it is no longer
seeded anywhere:

export function activeHoneytokenSet(clientId: string, count: number, generation = 0): string[] {
  const current = honeytokenSet(clientId, count, generation);
  return generation > 0
    ? [...current, ...honeytokenSet(clientId, count, generation - 1)]
    : current;
}

The reasoning: nothing legitimate ever requests a retired decoy. It was never a
real record, and no client holds a reference to one. Something asking for one is
replaying a stolen list or guessing: both worth knowing about. Retiring
recognition at the same instant as seeding would hand an attacker a window in
which yesterday's decoys are free to probe.

Recognition stops at two generations, so it can't grow without bound.

The allowlist, and what it does not buy

Here's a failure mode that's easy to miss until it happens to you.

Your security team runs a scanner. A scanner's job is to probe for exactly the
things honeytokens are: identifiers that shouldn't resolve. Without an
exemption, its first sweep trips a decoy and revokes your own security team's
credential, and the failure looks like a successful detection, so nobody
investigates.

So recognition is allowlisted for scanner identities. Verified on a real run: 21
decoy requests, 0 recognised, 0 honeytoken hits, never revoked, while a
non-allowlisted client tripped 3 and was revoked immediately.

But, and this is the part I got wrong first, the exemption is much narrower
than its name suggests.

When I added a scanner agent and asserted it was never blocked, the assertion
failed. Twice. The scanner was still denied: first by the miss-ratio guardrail
when it probed aggressively, then, after I made its behaviour more realistic, by
novelty_run_length, because sweeping an ID range is the mimicry signature.

Both are correct. A scanner sweeping for non-existent records is genuinely
indistinguishable from an enumerator. There is no clever feature that separates
them, because there is no difference in the traffic.

So the demo now asserts what the allowlist actually promises, decoy probes are
not recognised and never cause a revoke, and prints a note when the scanner is
denied on other grounds:

✓ PASS  internal scanner's decoy probes were not recognised (0 honeytoken hits, never revoked)
        note: it was still denied on other grounds (113/418 requests) — sweeping an ID
        range looks like enumeration, and the allowlist does not cover that

The operational lesson is worth more than a green tick: allowlisting an identity
for honeytokens buys it no blanket immunity. Scanners need exempting at the
policy level too, or they need to not look like attackers.

Honeytokens as training labels

One last use, which is the most valuable and the least obvious.

If you're exporting a dataset to train a model on, you have a labelling problem:
in production you don't know which windows were attacks. Simulation labels are
available only in simulation and are useless for a model you intend to deploy.

Honeytoken hits are different in kind. They are a high-confidence positive label
that a real deployment also has, because nothing but an enumerator ever requests
an ID that was never issued to anyone.

So the exported dataset carries them as a separate column from the simulation
label:

window_start, …, predicted_tier, ml_anomaly, label, honeytoken_positive

label is the simulation's ground truth. honeytoken_positive is the one you'd
actually have. A model trained on the second is a model you could ship.

That reframes what honeytokens are for. They're not only a tripwire, they're the
one source of ground truth an anomaly detector in production can get for free.

Next: enforcement, a six-rung graduated ladder where every rung has to be
invisible to the client it's applied to, and the discovery that two of them were
doing nothing at all.

Four ways a baseline quietly destroys the anomaly detector built on it

DarkEdges — Fri, 07 Aug 2026 09:56:56 +0000

Every anomaly detector answers one question: compared to what?

That comparison, the baseline, is where I lost the most time on this project,
and every failure had the same signature. Nothing errored. No test went red. The
numbers stayed plausible. The detector just quietly stopped detecting.

Four of them, in the order I found them.

1. The peer group contained the client it was judging

Cold-start clients have no history, so they're compared against a pool of other
clients' recent benign windows. Reasonable.

The pool was keyed by feature:

private readonly peer = new Map<FeatureKey, number[]>();

Every benign window every client produced went into the pool that client was
later compared against. Including itself.

So a client could define its own normality. Feed in enough windows and any
behaviour becomes unremarkable, which is precisely the cold-start attacker the
layer exists to catch.

What made me look was not reasoning, it was an experiment that wouldn't sit
still. I was trying to build a demo client that reliably landed in the middle of
the response ladder, and holding the traffic shape fixed while changing only the
request interval flipped the outcome between allow and step_up:

gap=500ms  origins=5  →  allow  (peak 0)
gap=700ms  origins=5  →  step_up (peak 83)
gap=800ms  origins=5  →  allow  (peak 0)

A knife edge like that is never a tuning problem. The outcome depended on a race
between a client's own samples reaching the pool and the pool being consulted.

Fix: key the pool per client, and exclude the client under evaluation.

for (const [clientId, values] of byClient) {
  if (clientId === excludeClientId) continue;   // this is what "peer" means
  …
}

Afterwards the behaviour became monotone in the actual evidence, and identical at
every request interval:

origins	1	3	4	5	6
peak score	17	35	59	83	100
tier	allow	log	throttle	step_up	deny

Lesson: if a parameter that shouldn't matter changes the outcome, stop tuning
and go find the defect. Knife edges are symptoms.

2. Baselines weren't keyed by window size

Same investigation, second defect. The per-client history was keyed by
(clientId, feature), with no window size.

req_rate over a minute and req_rate over an hour are not the same quantity.
Neither is distinct_src_ips, or anything else. All three window sizes were
pooling into one array, producing a baseline that described nothing.

It also filled the "enough samples to use own history" threshold three times too
fast, so clients switched off the peer baseline long before they had a meaningful
one of their own.

An unglamorous bug, and the kind that hides indefinitely because every individual
number still looks sane.

3. The boiling frog

This one is my favourite, because the code documented the correct behaviour and
then didn't implement it.

scorer.ts said:

// A window is treated as benign (and folded into baselines) only well below the
// throttle tier, so attack windows never poison the peer baseline, a client's
// own history, or the isolation forest's training set.
const BENIGN_MAX_SCORE = 40;

And learn() did this:

learn(fv: FeatureVector, benign: boolean): void {
  for (const feature of …) {
    ownHist.push(value);          // ← unconditional
    if (benign) { peerHist.push(value); }
  }
}

Own history was updated regardless. The peer pool was protected; the client's own
baseline was not.

The consequence is a textbook boiling frog. A sustained anomaly gets absorbed
into the client's own history within MIN_OWN_SAMPLES windows, the robust
z-score collapses toward zero, and the layer falls silent on exactly the thing
it was watching.

Measured before and after, on an anomalous client run for 26 windows:

before: flagged for 6 windows, then never again
after: flagged for 23 of 26

The fix is one line, if (!benign) return;, and the tradeoff is real and worth
stating: a persistently flagged client now never rebuilds its own baseline and
keeps being measured against its peers until it behaves normally again. For a
false positive that means staying flagged. That's survivable here only because
the response is graduated: throttle and step_up are recoverable, so a
wrongly-flagged client is slowed and challenged, not cut off.

Lesson: when a comment states an invariant, that's a test waiting to be
written. This one was a lie for months.

4. The all-time set that should have been a trailing window

novelty_ratio, the share of object IDs a client hasn't seen before, needs a
set of previously-seen IDs. Mine was a Set<string> that only ever grew.

It was wrong twice over.

Unbounded memory. One entry per distinct resource ID, per client, per window
size. The sequential attacker alone adds ~4,800 in a single demo run. Every
other baseline in the pipeline was capped: the route mix, the seasonal history,
the peer pool, the z-score history, the token tracker's eviction. This one, the
largest of them, was not.

Wrong semantics. The spec defines novelty against the trailing 7 days. Mine
was all-time, so nothing ever became novel again and novelty decayed monotonically
toward zero.

The second one is the dangerous one, and it's a slow-acting poison:
novelty_run_length is the only thing that catches mimicry, and it requires
novelty_ratio ≥ 0.15. On a long-running deployment that threshold quietly stops
being reachable, and mimicry detection degrades to nothing. A thirty-minute demo
cannot show it. Nothing would ever have raised a complaint.

Fix: a trailing horizon with a hard cap behind it.

And writing the test for it immediately found a second bug: the horizon was
applied after the window was scored, so a window could be measured against IDs
that had already expired. It's applied first now; the question a window asks is
"what has this client seen in the trailing 7d as of now".

That's worth pausing on. The test didn't just confirm the fix; it found a
different bug in the fix. The behaviour was subtle enough that I'd have shipped it.

5. (Bonus) The feedback loop I created myself

Not a baseline exactly, but the same family, and the most instructive because I
built it deliberately and it took a measurement to notice.

The step_up tier forces a client to re-authenticate. Re-authentication raises
token_issuance_rate. The scorer weights token_issuance_rate. A higher score
escalates the client to deny.

So: the system challenges a client → the client complies → the system convicts it
for complying.

The demo's ambiguous automation was being denied at a score of 88, with +21 of
it coming from the token issuance its own challenge had caused.

The fix is to mark issuances made under an outstanding challenge and exclude
them: the same reasoning that keeps the policy's own decision out of the
detection projection. Feeding your own output back in as a feature lets the
detector confirm its own suspicions: one spurious escalation justifies the next,
and the score drifts away from the traffic it's supposed to describe.

After the fix that client peaked at 66 instead of 88, and stopped being denied.

What these have in common

None of the five raised an error. None turned a test red. In every case the
system kept producing scores that looked entirely reasonable.

Three practical habits came out of it:

Measure before you tune. Three of these were found because a number moved when
it shouldn't have, or didn't move when it should. Every time I reached for a
threshold first, the threshold was the wrong tool.

Treat comments as unwritten tests. The boiling frog was documented correctly
and implemented wrongly, and lived for months in a file I'd read a dozen times.

Ask what the baseline contains. Almost every failure here was the baseline
absorbing the thing it was meant to measure against: the client itself, its own
anomalies, its own history without bound, or the system's own output. If you build
one of these, write down explicitly what the comparison population is, what it
excludes, and how it ages. Then check the code agrees.

Next: honeytokens: decoys derived so they recognise themselves and attribute
their own trips, why recognition lives in the gateway, and what an allowlist
does and doesn't buy.

Three detection layers that disagree usefully, and why they combine by max, not sum

DarkEdges — Fri, 07 Aug 2026 09:56:37 +0000

Features get you a vector per window. Turning that into a decision is where the
design choices are.

This system scores every window three independent ways and takes the strongest
single case. Each layer covers a failure mode of the others.

Layer 1: guardrails

Deterministic thresholds, no baseline of any kind:

// Honeytoken hit — highest-confidence signal. Immediate revoke.
if (fv.honeytoken_hits > 0) add(100, 'honeytoken_hits', '…');

// High miss ratio — guessing IDs that mostly do not exist.
if (fv.miss_ratio >= 0.4 && fv.req_count >= 10) add(88, 'miss_ratio', '…');

// Sequential walk — near-adjacent IDs in order.
if (fv.id_sequentiality >= 0.8 && fv.distinct_resource_ids >= 10)
  add(90, 'id_sequentiality', '…');

// Working set that expands and never stops — the mimicry signature.
if (fv.window_size === '1m' && fv.novelty_run_length >= 20)
  add(86, 'novelty_run_length', '…');

Being baseline-free is the point: they fire on a client's first window. A
statistical layer needs history to say anything, so a brand-new compromised
integration, one that never had a quiet period to learn from, is invisible to
it. Guardrails cover exactly that gap.

The deliberate omission is cardinality. There is no "distinct IDs > N" guardrail
in the scorer, for the reasons in part 3: it false-positives on legitimate
bulk reads and no threshold fixes that. (The gateway's fast path does have a
cardinality rule, at 150 distinct/minute: well above any realistic backfill, and
it exists to stop a flood before the first window closes.)

Layer 2: robust statistics

Per client, per feature, per window size: keep a bounded history and score new
values with a median/MAD robust z-score.

Median and MAD rather than mean and standard deviation, because mean and σ are
themselves distorted by the outliers you're hunting. One 5,000-request window
drags a mean enough to make the next one look normal.

Three things make this work in practice, and each was added after watching it
misbehave without them.

A scale floor per feature. A client whose interarrival_cv has been exactly
0.55 for a week has a MAD of ~0, so any deviation divides by nearly zero and
produces an enormous z. The floor is the smallest deviation worth treating as
meaningful:

export function robustZFloored(x: number, history: number[], floor: number): number {
  if (history.length < 2) return 0;
  const med = median(history);
  const scale = Math.max(medianAbsoluteDeviation(history), floor);
  return scale > 0 ? (x - med) / scale : 0;
}

Plain robustZ returns 0 when the history has no spread at all, which is exactly
backwards for the case you care about: a perfectly regular client that jumps 10×
is the strongest possible signal, not the weakest.

A gate and a cap. Deviations below |z| = 2.5 contribute nothing; above 8 they
stop growing. Normal variation is not evidence, and one absurd outlier shouldn't
dominate.

A single-feature cap. If only one feature is anomalous, the total is capped
below the escalation threshold. One lone signal doesn't escalate; it needs
corroboration.

Directions are explicit per feature, because they aren't uniform:

const DIRECTION: Partial<Record<FeatureKey, Dir>> = {
  interarrival_cv:        { sign: -1, weight: 10, floor: 0.1  }, // LOW is suspicious
  id_sequentiality:       { sign:  1, weight:  8, floor: 0.15 },
  miss_ratio:             { sign:  1, weight:  8, floor: 0.05 },
  subject_mismatch_ratio: { sign:  1, weight:  9, floor: 0.05 },
  distinct_src_ips:       { sign:  1, weight:  6, floor: 0.5  },
  // …
};

Peer groups

A client with no history of its own gets compared against its peers. But "peers"
has to mean something, pooling every other client means a search integration
is judged against a bulk sync and both look odd.

Clients are grouped by route profile: the smallest set of routes covering
most of their traffic, sorted. Two candidate keys were rejected:

Request volume, it would place a flooding attacker among the busiest clients, which is exactly where it looks least remarkable. Grouping by something the attacker controls hands them the choice of jury.
The full route set: too sparse. Nearly every client ends up alone, and a peer group of one is the client itself under another name.

The profile is taken from a client's benign history, not the window being
scored, so an attacker can't change groups by changing endpoints mid-attack.

Layer 3: an extended isolation forest

The first two layers each ask a one-dimensional question. Both are structurally
blind to a window that is unremarkable on every axis individually but sits
somewhere no legitimate client has ever been jointly. That joint structure is
the only reason to add a third layer.

The obvious choice is a classic isolation forest (Liu et al.): build trees that
split on one randomly-chosen feature at a random value, and measure how quickly a
point gets isolated. Anomalies are few and different, so they isolate near the
root.

I measured it on this project's own feature space rather than assuming:

	inliers	jointly-novel window
axis-parallel splits	0.50	0.51
hyperplane splits	0.44	0.66

The classic algorithm gave no separation at all on the case the layer exists
for. That's not a tuning failure, it's structural: isolating a point between
two clusters requires a specific conjunction of axis cuts that a random tree
rarely finds, so it scores no higher than the edges of the clusters themselves.

The fix is the Extended Isolation Forest (Hariri et al.): split on a random
hyperplane, a random normal vector and a random intercept, so one cut can
carve an oblique region.

// Draw a random hyperplane. Only dimensions that vary in this subsample get a
// non-zero coefficient — a constant dimension would just shift the intercept.
const normal = new Array<number>(dims).fill(0);
for (const d of active) {
  if (max > min) { normal[d] = gaussian(rand); any = true; }
}

Because a hyperplane mixes dimensions, features must be standardised first.
Without that, req_count (in the thousands) dominates every dot product and the
forest collapses back into a one-dimensional detector.

Calibration, not constants

An isolation-forest score has no absolute meaning. It shifts with dimensionality,
subsample size, the depth limit, and the shape of whatever benign population
exists. My first implementation used a literal threshold of 0.62 and was simply
wrong for this feature space: every real anomaly landed below it.

So the threshold is read off the model's own training scores: the 99th
percentile, with a floor at 0.5 (the forest's own reference point, below which a
window isolates more slowly than average and cannot be an outlier).

const trainScores = rows.map((r) => forest.score(r)).sort((a, b) => a - b);
const gate = Math.max(quantile(trainScores, 0.99), GATE_FLOOR);
const saturation = gate + Math.max(MIN_RAMP, 1.5 * (p99 - p50));

Roughly 1% of benign windows clear it, whatever the absolute numbers turn out to
be.

Two deliberate constraints

The model is frozen once fitted. Continuously refitting on whatever currently
looks benign is how an anomaly detector gets poisoned: a patient attacker whose
early windows score below the benign threshold gets absorbed into the definition
of normal, and the model then defends their behaviour. The cost is drift, in
production this needs a periodic, human-reviewed refresh against a vetted clean
period.

The ML score is capped below the deny tier. A forest tells you how strange,
never why. It may escalate for attention; a blocking decision should rest on a
signal a human can read off the attribution.

And an honest blind spot: the forest cannot learn "this feature must stay
constant", because a zero-variance dimension offers nothing to split on.
distinct_src_ips is 1 for every window of a well-behaved integration, so 25 of
them is invisible here. That's a division of labour rather than a gap,
constant-in-baseline features are exactly where a median/MAD z-score with a
scale floor excels, and layer 2 weights them.

Combining: max, not sum

const composite = clamp(Math.max(g.score, s.score, m.score), 0, 100);

This surprises people, so: the three layers are three readings of the same
evidence, not three independent pieces of it. A sequential walk shows up as a
guardrail trip, a timing anomaly, and an isolation-forest outlier, all
describing one fact. Summing them triple-counts it.

Taking the max means the composite is always "the strongest case any single layer
can make". That's also what keeps attribution honest: the score and the reason for
it come from the same place. With a sum you get 94 points from three partial
explanations and nothing you can put in front of a human.

Attribution is the point

Every score carries per-feature attribution: which features contributed how many
points, with their values and z-scores. It's persisted to ClickHouse so any past
decision can be reconstructed.

score=88 tier=deny
  miss ratio 0.90 over 280 reqs
  miss_ratio anomalous (z=18.1, +64)

For the forest, which has no native notion of per-feature contribution, this is
done by ablation: replace one feature with its training median ("what if this
had been typical?") and measure how much of the anomaly disappears. Contributions
are normalised to the points actually awarded, so the attribution always adds up
to the score it explains.

The reason this matters isn't UX. An unexplained score cannot be argued with, and
a detection nobody can argue with is a detection nobody will act on. The first
time an on-call engineer is paged at 3am by "client X scored 87", the only useful
next question is why, and if the answer is "the model said so", the system gets
turned off.

Tiers

The composite maps to a graduated ladder, mirrored exactly in the Rego policy:

  0–29  allow
 30–49  log
 50–69  throttle
 70–84  step_up
 85+    deny
        revoke   (never from score alone — reserved for hard signals)

revoke is unreachable by score. It's reserved for honeytoken hits and explicit
revocation, because those are categorically different kinds of evidence.

What each rung actually does, and the discovery that two of them did nothing at
all for months, is article 7.

Next: the four ways a baseline quietly destroyed the detector built on it.
The most transferable article in the series.

The most obvious signals for detecting enumeration don't work

DarkEdges — Fri, 07 Aug 2026 09:55:42 +0000

Every window of traffic, per credential, this system computes about twenty
features. Six of them drive decisions. The rest are computed, stored, displayed,
and deliberately never allowed to escalate anything.

That split is the most useful thing in this article, so let's get to why.

The trap

You want to detect someone reading your whole user table. The obvious signals:

distinct_resource_ids, how many distinct objects were touched
novelty_ratio, what share had never been touched by this client before
repeat_ratio, what share had been

An enumerator maxes all three: thousands of objects, nearly all new, almost no
repeats. It looks damning.

Now run a legitimate post-deploy backfill through the same features. Thousands of
objects. Nearly all new. Almost no repeats.

They are identical. Not similar, identical in shape, because both are "read a
large number of records this client has not read before". Cardinality is a measure
of how much was read, and how much is exactly the thing the two have in common.

I tried it anyway, with corroboration from other signals, because it felt like it
should be salvageable. It fired on backfill windows whenever the timing jitter
happened to dip. There is no threshold that separates them, because the
distributions overlap by construction.

The reframe

Stop asking how much was read. Ask how it was accessed.

A backfill reads records in its own key order, whatever order its job queue produced. An enumerator walks the ID space.
A backfill reads records that exist; it has a list. An enumerator guesses, and guesses wrong most of the time.
A backfill is driven by a system with queues and retries, so its timing has jitter. A tight loop does not.
A backfill runs from one place. A credential being replayed doesn't.
A backfill ends.

Each of those is a feature, and each carries the distinction that cardinality
doesn't.

`id_sequentiality`

The fraction of consecutive requested IDs whose absolute gap is ≤ 2, in arrival
order.

export function idSequentiality(idsInOrder: string[]): number {
  const nums: number[] = [];
  for (const id of idsInOrder) if (/^\d+$/.test(id)) nums.push(Number(id));
  if (nums.length < 2) return 0;
  let adjacent = 0;
  for (let i = 1; i < nums.length; i++) {
    if (Math.abs(nums[i]! - nums[i - 1]!) <= 2) adjacent++;
  }
  return adjacent / (nums.length - 1);
}

Near 1.0 for an ID walk, near 0 for random access or hot-object reads. Note it
only applies to integer-like IDs, which is the whole argument for opaque
identifiers, and why the demo's dataset deliberately has both integer (users,
orders) and UUID (documents) resource types.

`miss_ratio` and `not_found_ratio`

Share of responses ≥ 400, and share that are 404. An integration reading records
it has references to has a miss ratio near zero. A guesser's is near one. This is
the single most reliable signal in the system and it's almost embarrassingly
simple.

`interarrival_cv`

Coefficient of variation (σ/μ) of the gaps between requests. Low CV means
machine-regular. Legitimate traffic sits around 0.5–0.7; a tight loop is
essentially 0.

Note the direction is inverted here: low is suspicious. That's worth
flagging because most anomaly-detection plumbing assumes high-is-bad, and this
feature is a reminder to make direction an explicit per-feature property rather
than an assumption.

`distinct_src_ips` / `distinct_asns`

One credential appearing from many origins. Catches the distributed profile,
where per-origin rate limiting sees nothing because each IP issues a trickle.

`endpoint_kl_divergence`

Kullback–Leibler divergence between this window's route mix and the client's own
trailing route mix, in bits. A dictionary attack pivots traffic onto the search
endpoint, and KL measures exactly that pivot.

The implementation detail that matters is smoothing. A route the client has
never used before would produce infinite divergence, which would swamp every
other feature forever. Additive smoothing on the baseline makes it large but
finite:

const alpha = 0.5;
const qTotal = [...q.values()].reduce((a, b) => a + b, 0) + alpha * keys.size;

An integration hitting a brand-new endpoint should register as surprising, not
as an unbounded outlier.

`subject_mismatch_ratio`

The one that catches an attack nothing else sees.

Consider a compromised integration holding a genuine delegation for one user.
Every request carries a valid token, valid scopes, and a subject it really is
entitled to act for. Per-request authorization has nothing to object to: a token
that may act for Alice is acting for Alice.

What it cannot see is that Alice's delegation is being used to read everyone
else's records.

So the resource API reports who owns each record, the gateway records that beside
the subject, and the feature is the share of delegated requests where the two
differ. An honest on-behalf-of integration reads the record of the person it's
acting for, so this sits at zero.

Measured on the first window of the obo-sweep profile: miss_ratio 0,
subject_mismatch_ratio 1.0, caught on subject_mismatch_ratio (z=20.0, +72)
alone, with no other signal contributing anything.

One implementation note with a security edge: the owner header is set on hits
and misses, empty when there's nothing to report. Present only on hits, it
would turn the response shape into an existence oracle.

`novelty_run_length`, the feature about time

This is the one that finally caught mimicry, and it's the most interesting
feature in the system because it isn't about the shape of a window at all.

Mimicry hides enumeration inside replayed legitimate traffic. Normal rate, normal
timing, valid IDs, no ordering, miss ratio diluted below every threshold. Against
the feature set above it scored zero on every window, because every one of
those features describes a single window, and in any single window mimicry is
legitimate traffic.

What separates it from a benign bulk read is not shape but duration. A
backfill ends. An ongoing compromise does not.

So novelty_run_length counts consecutive windows in which the client's working
set kept growing, and a guardrail fires at 20: far beyond any plausible backfill
episode. Detection is correspondingly slow, and the project claims no latency
bound for it.

Two design details worth arguing with:

It is deliberately not corroborated by a low repeat ratio. Mimicry's method is to keep re-reading the hot set as cover, which holds repeat ratio high; requiring a low one would exclude precisely the traffic it exists to catch. (I had this wrong first, and the guardrail excluded its own target.)
It is evaluated only on the 1-minute window. Over longer windows a legitimate client's occasional stale references accumulate, each is an ID never seen before, until its novelty ratio never returns to baseline and the run stops distinguishing anything.

The honest limitation: an integration whose legitimate job is to page through an
ever-growing corpus trips this, and needs an explicit exemption rather than a
cleverer threshold.

The features that are computed and never scored

This is the part I'd most like to convince you of.

Several features are computed every window, written to ClickHouse, exported in
the training set, and shown in the attribution breakdown, but are excluded from
the score. That is a design decision, not an unfinished TODO.

distinct_resource_ids, novelty_ratio, repeat_ratio, the trap from the
top of this article. They establish that a bulk access is happening, which is
useful context for a human, and they cannot separate the two hypotheses.

hour_of_day_zscore, how unusual this window's volume is for this client at
this time of week. Tempting, and wrong: an off-schedule volume spike is precisely
what a legitimate post-deploy backfill looks like. Weighting it would buy nothing
but false positives on the one case that most needs to be right.

distinct_subjects: how many principals a credential acted for. Saturates
for any legitimate bulk job, exactly as cardinality does.

replayed_jti: the share of traffic riding a token seen from more than one
host. This is the shape of a stolen bearer token, and I very much wanted it to
work. It's excluded because I measured it: weighted at all, a perfectly benign
client, one that mints a single token and shares it across its own workers,
scored 100 and was denied, with +53 of that coming from this feature alone.

One token from several hosts is the shape of theft and the shape of a shared
token cache. Separating them needs origin-relationship data (AS ownership,
geography) this system doesn't model. So it's recorded for a human to act on,
which is not the same as the system acting on it.

That last one is the general pattern, and it's worth stating plainly: the
signals that look most damning are often the ones that saturate for some entirely
ordinary behaviour. The only reason this one didn't ship as a false-positive
generator is that the weight was measured against a benign client before being
kept.

Windows

Everything above is computed at 1m, 5m and 1h, per client, in tumbling windows.

Multiple scales are not redundancy. slow-and-low issues about two requests a
minute, so its 1m windows fall below the minimum request count and score nothing
at all, its signal only exists at 5m and 1h. Conversely novelty_run_length is
only meaningful at 1m.

Different attacks are visible at different scales, and that is the entire
premise of the pipeline. It's also the premise that a bug quietly violated for
months, which is article 8.

Next: three scoring layers, why they combine by maximum rather than sum, and
why per-feature attribution is the point rather than a nice-to-have.

Simulating attackers is easy. Simulating legitimate users is the hard part

DarkEdges — Fri, 07 Aug 2026 09:55:05 +0000

In part 1 I argued that enumeration from a valid credential can only be
caught from the shape of the request sequence, and that the hard requirement is
not catching attackers but not catching everyone else.

That requirement has a consequence people skip: to have any confidence in a false
positive rate, you need legitimate traffic that is genuinely hard to distinguish
from an attack. Traffic that is obviously benign proves nothing. The detector in
this project drives ten synthetic clients through one gateway, all holding real
tokens with identical scopes, and only five of them are attackers.

Here is the whole cast, and what each one is for.

The legitimate integration

integration-acme models a real third-party integration:

Zipf-distributed reads of a hot object set: real integrations re-read the same handful of records constantly, they don't sample uniformly
human-ish jitter between requests (300–1700ms), because a real system has queues and retries and network variance
a small burst at the top of each hour from a batch sync
occasional genuine 404s from stale references it still holds
on-behalf-of reads: when it fetches user 812's record it does so on behalf of user 812

That last point matters and I'll come back to it in article 3.

And then, deliberately, the thing that makes the whole project difficult:

a benign bulk backfill, a post-deploy job that reads 9,000 records it has never touched, over five scenario-minutes, as fast as the API will serve them.

This is the false-positive stress test, and every detection rule has to survive
it. As shown in part 1, on cardinality, novelty ratio and repeat ratio a backfill
is indistinguishable from an attacker dumping the same table.

The demo asserts integration-acme stays in allow for the entire run, backfill
included. Zero blocked requests. If that assertion ever fails, the detector is
worthless regardless of what it catches.

The attackers

Five profiles, all using a credential of the same type and scopes as the real
integration. None of them exploit a bug; every one is using access it legitimately
holds.

sequential: walks integer IDs in order, fast and machine-regular. The
naive attacker. Trips the fast-path cardinality guardrail and the windowed
sequentiality guardrail within seconds. It's in the cast mainly as a control: if
your detector can't catch this, stop reading.

dictionary: probes /users?email= from a wordlist of plausible corporate
addresses. Structurally different: there is no ID ordering to detect, so
id_sequentiality never fires. What gives it away is the miss ratio (almost every
guess is wrong) and the route mix pivoting onto the search endpoint. About one
guess in twenty lands on a real account, which is exactly why the profile is worth
running: a blind guesser still harvests real data.

slow-and-low: one request every 30 seconds, random IDs across a large
range so ~99% miss. Its whole design is to stay below every rate threshold. It
must be caught by accumulation over the coarser windows, never by rate. This
profile later exposed the most consequential bug in the project (article 8).

distributed: one credential replayed from six IPs across six ASNs,
re-authenticating from each. Per-origin rate limiting sees nothing: each IP
issues a trickle. The credential is the only thing tying the requests together,
which is precisely the axis the detector aggregates on.

mimicry: the hard one. Enumeration hidden inside replayed legitimate
traffic on the same credential: for every probe it issues several requests that
look exactly like the real integration's work, with the same Zipf re-reads, the
same jitter, the same route mix. Rate rules see nothing, the miss ratio is diluted
below the guardrail threshold, and interleaved hot reads destroy the ordering
signal.

Measured against the detector as it stood before I designed for it: mimicry
scored zero on every single window. Not "low": zero. Every feature described
the shape of one window, and in any one window mimicry is legitimate traffic.
Article 3 covers the feature that finally caught it, which had to be about
duration rather than shape.

obo-sweep: a compromised integration holding one genuine delegation. Every
request carries a valid token, valid scopes, and a subject it really is entitled
to act for. It uses that single delegation to read everybody else's records. No
enumeration signal fires at all: the IDs are valid so there's no miss ratio, they
are strided so there's no ordering, the rate is ordinary. This one is caught by a
signal nothing else in the system provides.

honeytoken breach: requests a decoy object from a cold start, with zero
accumulated score. Tests the instant-revoke path (article 6).

The two that aren't attackers, and aren't quite normal either

These are the interesting ones, and both were added late, after the detector
started behaving in ways I couldn't explain.

The ambiguous automation

integration-paced is legitimate. It re-reads a small fixed set of records it
is entitled to, never probes a missing ID, never walks IDs in order. What makes it
look wrong is that it is obviously a machine, rigid timing, zero jitter, running
from several egress hosts at once, and gradually adding more of them as it scales
out. A monitoring bot, or a sync worker being horizontally scaled.

That is genuinely ambiguous evidence, and it exists to answer a question the rest
of the cast doesn't ask: what should the system do when it isn't sure?

The answer should not be a block. It climbs log → throttle → step_up, keeps
working the whole time, and is never hard-blocked. When it reaches step_up it
answers the challenge by re-authenticating: transparently, because that's what
any competent OAuth2 client does when its token is rejected, and carries on.

It is also, honestly, a false positive, and the demo counts it as one. A
detector that never admits to any is not being measured.

The internal scanner

internal-scanner is your own security team's tooling, and it exists because of
a specific failure mode: a scanner's job is to probe for exactly the things
honeytokens are: identifiers that shouldn't resolve. Without an exemption, its
first sweep trips a decoy and revokes your own security team's credential, and
the failure looks like a successful detection.

So honeytoken recognition is allowlisted for scanner identities. Verified in a
run: 21 decoy requests, 0 recognised, 0 honeytoken hits, never revoked, while
the non-allowlisted breach client tripped 3 hits and was revoked immediately.

But the exemption turned out to be narrower than its name suggests, and the demo
now says so out loud. The scanner is still denied, for sweeping an ID range,
which is the mimicry signature. That's correct: a scanner sweeping for
non-existent records is genuinely indistinguishable from an enumerator. The
operational lesson is worth more than a green tick: allowlisting an identity for
honeytokens buys it no blanket immunity, and scanners need exempting at the policy
level too.

Why the boring clients matter more

If I'd built only the attackers, I would have shipped a detector that scores 100%
recall and is unusable.

Three of this project's most significant bugs were found only because a
non-attacking client was in the cast:

The ambiguous automation existing at all is what forced me to check whether the middle tiers of the response ladder did anything. They didn't, throttle and step_up were being decided, recorded, displayed on a dashboard, and then the request was served anyway.
Trying to make that client land reliably in the middle band exposed four separate baseline defects (article 5), including one where changing nothing but the request interval flipped the outcome between allow and step_up.
The scanner exposed that a defensive feature wired into the gateway since the beginning had no test and no traffic exercising it.

None of those are findable with attackers alone. Attackers make the detector fire;
only realistic non-attackers tell you whether firing meant anything.

Determinism

Everything is seeded. Every agent takes an explicit RNG seed, the dataset is
generated from a fixed seed, and the isolation forest is seeded per window size.
Two runs of the same scenario produce the same traffic.

This matters more than it sounds. When you are chasing a detector that behaves
differently between runs, you need to know the traffic is not the variable.
That confidence is what let me establish, in article 5, that an inexplicable
flip-flop between allow and step_up was a defect in the baseline rather than
noise in the input.

What is not deterministic is timing under load, and I'll be honest about it here
because it bit repeatedly: all ten agents share one Node event loop. On an idle
machine the sequential profile is caught in 29–50 scenario-seconds; on a box at
load average 8.5 the same code reports 74–76 and fails its bound. That's the
harness, not the detector, but it means "did my change cause a regression?" has
to be answered on a quiet machine.

Next: which features actually discriminate, why the most obvious ones don't,
and the ones this system computes and deliberately never scores.

Your API is being enumerated by a client with a perfectly valid token

DarkEdges — Fri, 07 Aug 2026 09:53:09 +0000

Here is a request:

GET /users/4821
Authorization: Bearer eyJhbGciOiJSUzI1NiIs...

The token is valid. It is not expired, the signature verifies, the issuer is
yours. The client it belongs to: a third-party integration your customer
installed eighteen months ago, has the users:read scope, and this endpoint
requires exactly that. The user record 4821 exists and is one your customer's
organisation is entitled to see.

Every authorization check you have passes. They should pass. The request is,
individually, correct.

Now here is the problem: that integration was compromised last Tuesday, and this
is request 40,000 of a sweep through your entire user table.

Authorization is the wrong tool, structurally

The instinct is to reach for authz. Tighten the scopes. Add object-level checks.
Adopt whatever the latest OWASP API list calls it this year.

None of it helps, and it's worth being precise about why. Authorization is a
per-request predicate: given this principal, this action, this object, yes or
no? A compromised-but-authenticated integration satisfies that predicate on every
single request, because it is using exactly the access it was legitimately
granted. There is no individual request you can point at and call wrong.

What's wrong is the sequence. Forty thousand individually-correct requests,
in a particular order, at a particular rate, hitting a particular distribution of
object IDs, is an exfiltration. But "the sequence is wrong" is not a statement
per-request authorization is capable of making. It doesn't have the shape.

This is a general principle worth internalising: you cannot detect an attack
whose signal lives at a scale your check doesn't operate at. Rate limiting
operates per-client-per-minute, so it catches bursts. Authorization operates
per-request, so it catches privilege violations. Enumeration by a valid
credential lives at neither scale.

What's left: the shape of the sequence

If you can't judge requests individually, judge them collectively. Aggregate
requests by credential over a window of time, compute features describing the
shape of that window, and score the shape.

Concretely, the thing I built (apileak, TypeScript, runs under
docker compose) aggregates every request by client_id into tumbling windows
of 1 minute, 5 minutes and 1 hour, and computes ~20 features per window:

how many distinct object IDs were touched
what fraction of them had never been touched before
what fraction of requests were misses
whether the IDs walked in order
the coefficient of variation of the inter-request gaps
how many source IPs and ASNs the credential appeared from
whether any decoy object was touched

...and so on. A score comes out; a policy turns the score into a decision.

That is the whole thesis. Everything else is detail, but the detail is where all
the interesting failures live, which is why this series is eight articles and not
one.

The architecture, briefly

  agents ──► api-gateway ──► resource-api
               │  ├──► OPA          (decision; reads pushed risk scores as data)
               │  └──► Redis        (fast-path counters, sub-second guardrails)
               └──► Redpanda: access.events / token.events
                         │
                   feature-pipeline   (windowed aggregation, 1m/5m/1h)
                         │  └──► ClickHouse (events, features, scores)
                   scorer (guardrails + statistical + isolation forest)
                         ├──► OPA data push
                         └──► WebSocket ──► CLI + dashboard

Two properties of this shape matter more than the component choices.

Scoring is asynchronous. No machine learning runs in the request path. The
gateway consults a precomputed score plus its own cheap in-Redis counters, so
its p99 overhead stays under 15ms: asserted in a load test, because a detection
layer that adds 200ms to every request will be switched off by whoever is on call
the first time latency alerts fire.

The gateway has a fast path of its own. Windowed scoring cannot react faster
than one window. A client that opens with 600 requests in fifteen seconds needs
stopping before the first window closes, so the gateway keeps rolling per-minute
counters (a request count, a miss count, a HyperLogLog of distinct IDs) in Redis
and the policy can fire on those alone. In practice this matters enormously: in
one run the dictionary-attack profile was enforced at 9 seconds by the fast
path and only flagged at 44 seconds by the windowed scorer. Enforcement
preceding detection is not an anomaly, it's the fast path doing its job.

What makes this hard is not the attackers

If the only requirement were "catch someone reading the whole user table", this
would be a weekend project. Set a threshold on distinct object IDs per hour, page
someone, go home.

The requirement that makes it hard is the other one: a legitimate bulk
operation must not trip it.

Consider a post-deploy backfill. Your customer's integration ships a new feature
and needs to populate a cache, so it reads 9,000 user records it has never read
before, over five minutes, as fast as your API will serve them.

Now compare it to an attacker dumping the same table:

	backfill	enumeration
distinct objects touched	thousands	thousands
share never seen before	~100%	~100%
repeat ratio	~0	~0
request rate	elevated	elevated
scope used	exactly as granted	exactly as granted

On every metric you'd reach for first, they are identical. I tried blocking
on cardinality anyway, with corroboration from other signals. It fired on backfill
windows whenever the timing jitter happened to dip. That approach is not
salvageable by tuning; the signals genuinely do not carry the distinction.

Finding signals that do carry it is the subject of article 3, and it turns on a
simple reframing: stop asking how much was read and start asking how it was
accessed. A backfill reads records in its own key order and they all exist. An
enumerator walks IDs, or guesses them, or probes for ones that don't.

What "detection" has to mean

One more constraint, and it changes the design more than anything else.

Detecting the attack is useless if the attacker can tell they were detected. An
attacker who learns that request 40,000 was blocked simply changes tactics,
slows down, rotates credentials, switches to a different ID scheme, and you have
converted a detection into a training signal for them.

So every blocking decision in this system returns a response byte-identical to
an ordinary 404: same status, same body, same headers, same timing. The
attacker's requests just start coming back as "that object doesn't exist". This
constraint reaches surprisingly deep: it's why the 404 and 403 in the
resource API are constant-time and byte-identical to each other (there's a test
that measures both distributions and fails if they're distinguishable), and it's
why honeytoken recognition happens at the gateway rather than in the resource API,
so that the resource API's response never varies at all.

Article 7 covers what covert enforcement does to a graduated response ladder,
including the two rungs that turned out to be doing nothing for months.

What this series covers

This article, why authorization can't help, and what's left.
The cast: the attacker profiles, and the much harder problem of modelling legitimate traffic convincingly enough to trust your own false-positive rate.
Features, which signals discriminate, which only look like they do, and the ones deliberately computed but never scored.
Scoring: three layers, why they combine by maximum rather than sum, and why per-feature attribution is the whole point.
Baselines: four separate ways the baseline quietly destroyed the detector built on top of it. The most transferable article in the series.
Honeytokens: decoys derived so they recognise themselves and attribute their own trips, plus rotation and the limits of allowlisting.
Enforcement, a six-rung ladder where every rung has to be invisible.
When your detector lies to you: the bug class that dominated this project: components that do nothing, raise no error, and leave every number looking plausible.

A warning about what this is: a demonstration artifact, built to be read and
argued with. It is not a product. Where it doesn't work I've said so, in the
articles as well as the repo: the failures turned out to be the most useful
material.

Cross-App Access Without Leaking Credentials: A Hands-On ID-JAG POC

DarkEdges — Wed, 29 Jul 2026 10:47:50 +0000

Most "identity federation" demos stop at login. The harder problem is what happens after login: how does App B get a token to call App C on a user's behalf, without App B ever holding the user's original credentials — and without the user re-authenticating at every hop?

That's the problem xaa-id-jag-poc is built to explore.

What it is

A working proof-of-concept for cross-app access (XAA) using the emerging ID-JAG (Identity Assertion JWT Authorization Grant) draft spec, layered on top of standard OAuth 2.0 building blocks:

RFC 8693 — OAuth 2.0 Token Exchange
RFC 7523 — JWT Bearer Grant for OAuth
RFC 7636 — PKCE
ID-JAG (draft) — exchanging an ID Token from a trusted IdP for an access token scoped to a downstream Resource Authorization Server

In plain terms: a user authenticates once at an Identity Provider, and their identity assertion can then be exchanged — hop by hop — for scoped access tokens at other services, without those services ever seeing a password or a long-lived credential.

Architecture

Spring Authorization Server — acts as both the Identity Provider and the Resource Authorization Server
Spring Boot — protected Resource API validating the exchanged tokens
Next.js + TypeScript — the web UI driving the flows
Nginx — reverse proxy handling hostname routing between services
Docker Compose / Helm — local orchestration and a full Kubernetes deployment path

There's also MCP (Model Context Protocol) resource URI support, which lets you point the same token-exchange machinery at MCP-style endpoints — relevant if you're thinking about how AI agents should authenticate to tools and downstream APIs rather than being handed a static bearer token.

What's been shaking out recently

The repo has been getting hardened with negative-path test coverage — the boring but critical part of any auth system:

Correct propagation of MCP auth failures as HTTP 401/403 (instead of leaking through as 500s or ambiguous errors)
Explicit incorrect-scope test cases across both the MCP and web test suites
Fixes to JWT bearer issuer extraction, plus stabilized REST paths for the positive-flow tests

If you've ever shipped a token-exchange flow, you know the negative cases are where the real bugs live — a wrong scope or a bad issuer claim needs to fail loudly and specifically, not silently degrade into "well, it kind of works."

Try it

Everything runs locally via Docker Compose with pre-configured dev credentials and hostname routing, so you can watch the full token-exchange chain happen end-to-end without standing up real infrastructure. A Helm chart is included if you want to push it onto Kubernetes with a private registry.

Repo: https://github.com/darkedges/xaa-id-jag-poc

If you're working on agent-to-agent auth, MCP security, or just want to see RFC 8693 token exchange actually wired up rather than described in a spec doc, it's worth a clone and a poke around. Issues and PRs welcome.

Proving Zero Trust Actually Works: Entra ID + Cloudflare Access over Both OIDC and SAML

DarkEdges — Sun, 26 Jul 2026 22:32:19 +0000

Most Zero Trust write-ups stop at "user signs in, user gets access." That's not
where these integrations actually break. They break on group membership that
almost, but doesn't quite, satisfy a policy. They break on admin consent that
was never granted, silently, until the first real sign-in. They break on the
one country rule that was supposed to be an OR and was actually an AND.

I built a small, fully Terraform-managed demo to make those failure modes
visible instead of theoretical: Microsoft Entra ID driving Cloudflare Access
authorisation over both OIDC and SAML, against the same three users, with the
resulting identity provably verified at the origin, not just echoed.

The setup

Everything is provisioned by Terraform: the Entra app registrations, the demo
users and groups, the admin consent grant, both Cloudflare identity providers,
a dedicated tunnel with Cloudflare-managed ingress, DNS, and the Access
applications and policies themselves. One make apply and it's all live.

Entra is registered with Cloudflare twice. Once through the native azureAD
OIDC connector, where Cloudflare resolves group membership with a live
Microsoft Graph call at sign-in. Once as a generic SAML 2.0 app, where group
membership rides inside the signed assertion instead. Two applications point
at OIDC, one points at SAML, so the protocols sit side by side against
identical identities instead of being described in the abstract.

Three users, three apps, one deliberate trap

user	Entra groups	echo-basic	echo-secure	echo-saml
alice	engineering, oncall, finance	yes	yes	yes
carol	engineering	yes	no	no
bob	contractors	no	no	no

Alice and bob are the boring cases: pass everything, or fail everything. If
that's all you test, a single-group policy and a compound one look identical
from the outside. Carol is the point of the whole demo. She satisfies
echo-secure's include block because she's in engineering, and she's still
refused, because its require block also demands on-call membership and she
doesn't have it. That's the only way to actually show compound policy logic
rather than just assert it in a blog post.

That maps onto the three operators Cloudflare Access policies are built from,
and where hand-built policies most often go wrong:

block	logic	in echo-secure
include	OR, match at least one	engineering or finance
require	AND, match every one	and on-call, and in-country
exclude	NOT, match none	and not a contractor

There's also a deny policy at precedence 1 that short-circuits the chain
before the allow rule is ever evaluated, a purpose-justification prompt, a
30-minute session, and a path-scoped bypass policy that lets /health
through unauthenticated on a hostname that's otherwise fully protected.

Identity you can actually verify, not just see

The brief for the origin was simple: echo the request headers back. But raw
echo only proves a request arrived, it can't distinguish a request that
passed policy from one that reached the origin some other way, and the
Cf-Access-Jwt-Assertion header stays an opaque blob unless something
decodes it.

So the origin does three things instead of one:

Verifies the JWT signature against the team's JWKS, with both issuer and audience pinned, audience being that specific application's AUD tag. A valid token minted for a different Access application in the same account gets rejected here. This is the check any real origin sitting behind Access should be doing.
Calls /cdn-cgi/access/get-identity with the session cookie, but only after verification succeeds. An unproven session shouldn't cause the origin to make upstream calls on the user's behalf.
Resolves the group GUIDs both protocols return into readable names, and labels where each one came from: the identity endpoint for OIDC, the SAML assertion for SAML. That provenance label is the visible difference between the two protocols on the page.

A /raw route still gives the literal, unverified header dump for scripting.

Design decisions worth stealing

No circular dependency between Entra and Cloudflare. Both integrations
key off one URL, https://<team>.cloudflareaccess.com/cdn-cgi/access/callback,
which is derivable from the team name alone, before either side of the
integration exists. For OIDC it's the redirect URI. For generic SAML,
Cloudflare uses the same URL as both the Entity ID and the ACS endpoint.
Nothing has to exist before anything else, so the whole configuration applies
in a single Terraform pass.

Admin consent is configuration, not a portal click. support_groups = true makes Cloudflare call Microsoft Graph after the code exchange, which
needs Directory.Read.All and GroupMember.Read.All with tenant-wide
consent. This is the step most often missed when an integration like this is
built by hand in the portal, and it fails at user sign-in, not at setup, so
the integration looks fine right up until someone actually tries to use it.
azuread_service_principal_delegated_permission_grant grants it
declaratively, followed by a 30 second time_sleep, because Entra's
directory is eventually consistent and an IdP created the instant the grant
returns can fail its first few sign-ins for no visible reason.

Group identity is always a GUID, never a display name, on both
protocols. Policies reference azuread_group.demo[...].object_id directly
rather than hand-copied GUIDs, and the origin gets a Terraform-generated
GUID to name map purely for display. Nothing security-relevant depends on
that map.

required_country is deliberately a single string, not a list. Every
entry in an Access require block is ANDed together. Two country rules
wouldn't permit either country, they'd make the policy unsatisfiable. That's
a genuinely easy trap to fall into by hand, which is why the Terraform
variable has a validation rule enforcing a single value.

The SAML email claim is name, not emailaddress. Entra sources the
emailaddress claim from the mail attribute, which is empty for cloud-only
users that have never been assigned a mailbox, exactly the users this project
creates. Pointing Cloudflare at it produces a session with a blank identity.
The name claim comes from userPrincipalName instead and is always
populated.

What this demo is not

It's a demonstration, not a production template, and it says so out loud:
Terraform state is local and unencrypted, the tunnel token lands in
.env.generated in plaintext, the three demo users share a password with no
forced rotation, there's no SCIM provisioning, and the SAML IdP doesn't sign
AuthnRequests. Every one of those is a deliberate trade for a demo that
stands up in one command instead of an afternoon, and docs/ARCHITECTURE.md
in the repo spells out what each one would need to become for anything
longer lived.

Why bother

Zero Trust vendors sell the diagram. What actually separates a policy that
works from one that only looks like it works is whether you tested the case
that's supposed to fail for a specific reason, not just any reason. Carol is
that case. If your own Access policies don't have a Carol, they haven't
really been tested yet.

Github Repository

The code is available at https://github.com/darkedges/cloudflare-zerotrust-entra

From vexctl scripts to a governed VEX platform: building vex-ui with Next.js, keyless signing, and a Trivy-consumable repo

DarkEdges — Tue, 07 Jul 2026 21:17:17 +0000

un Trivy against almost any vendor container image and you'll get a wall of
findings. Most of them don't matter, the vulnerable code path is never
executed in your deployment. The problem is that knowing that has
traditionally lived in spreadsheets and Jira comments, where no scanner can
see it.

VEX (Vulnerability Exploitability eXchange) turns that triage into
machine-readable statements that scanners apply automatically. This post
walks through a complete, runnable workflow, baseline scan → OpenVEX
generation → a hosted VEX repository → suppressed re-scan, and covers three
gotchas I hit that the docs don't mention.

Everything here is in a one-script demo repo:
github.com/darkedges/trivy-vex-demo

⚠️ The demo marks every CVE not_affected mechanically to exercise the
plumbing. In real life the assessment is the valuable part, don't ship
VEX statements you can't defend.

The setup

Target: pingidentity/pingaccess:8.3.4-edge (digest
sha256:51689e8c…). Tools, pinned and current at time of writing: Trivy
v0.71.0, vexctl v0.4.1, OpenVEX v0.2.0, and the
VEX Repository Specification v0.1.

A small Alpine-based toolchain image carries everything:

FROM alpine:latest
ARG TRIVY_VERSION=0.71.0
ARG VEXCTL_VERSION=0.4.1
RUN apk update && apk upgrade --no-cache && \
    apk add --no-cache ca-certificates curl jq
RUN curl -fsSL -o /tmp/trivy.tar.gz \
      "https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION}/trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz" && \
    tar -xzf /tmp/trivy.tar.gz -C /usr/local/bin trivy && rm /tmp/trivy.tar.gz
RUN curl -fsSL -o /usr/local/bin/vexctl \
      "https://github.com/openvex/vexctl/releases/download/v${VEXCTL_VERSION}/vexctl-linux-amd64" && \
    chmod +x /usr/local/bin/vexctl

Step 1, Baseline

trivy image --format json --output baseline-report.json \
  pingidentity/pingaccess:8.3.4-edge

Result: 70 findings (2 CRITICAL / 18 HIGH / 42 MEDIUM / 8 LOW) across 51
unique CVE/GHSA IDs, all in bundled Java jars and git-lfs. The Alpine OS
layer itself: zero.

Step 2, Generate OpenVEX statements

One statement per CVE with vexctl. The critical detail is the product
identifier: a pkg:oci purl pinned to the image digest, never the tag,
tags move, VEX assertions shouldn't.

vexctl create \
  --product="pkg:oci/pingaccess@sha256:51689e8ccf1ec6bef28c855a2f2fafdd3556f753609adad2e258580e3bc9397c" \
  --vuln="CVE-2022-46337" \
  --status="not_affected" \
  --justification="vulnerable_code_not_in_execute_path" \
  --status-note="DEMO assessment for a suppression-workflow POC." \
  --author="DarkEdges Security <nirving@darkedges.com>" \
  --file="CVE-2022-46337.openvex.json"

Which produces:

{
  "@context": "https://openvex.dev/ns/v0.2.0",
  "@id": "https://darkedges.com/vex/demo/pingaccess/CVE-2022-46337",
  "author": "DarkEdges Security <nirving@darkedges.com>",
  "statements": [
    {
      "vulnerability": { "name": "CVE-2022-46337" },
      "products": [
        { "@id": "pkg:oci/pingaccess@sha256:51689e8c…" }
      ],
      "status": "not_affected",
      "justification": "vulnerable_code_not_in_execute_path",
      "status_notes": "DEMO assessment for a suppression-workflow POC."
    }
  ]
}

Loop that over the 51 IDs from the baseline JSON, then consolidate:

vexctl merge statements/*.openvex.json > pingaccess-8.3.4-edge.openvex.json

Tip: leave the purl qualifiers off in statement products. Per Trivy's
matching rules, a purl without qualifiers matches regardless of arch or
repository_url; with qualifiers, you're signing up for exact matching.

Step 3, The instant win: `--vex` flag

trivy image --show-suppressed \
  --vex pingaccess-8.3.4-edge.openvex.json \
  pingidentity/pingaccess:8.3.4-edge

70 findings → 0 reported, 70 suppressed. Each suppression is recorded in
the JSON report under Results[].ExperimentalModifiedFindings with
Type: ignored and the justification, auditable, not deleted.

Step 4, Publish via a VEX repository

A VEX repository is just static files: a manifest at
/.well-known/vex-repository.json and a tar.gz archive containing an index
plus the documents.

vex-repo/                       # webroot
├── .well-known/vex-repository.json
└── v0.1/vex-data.tar.gz        # contains: index.json + pkg/oci/pingaccess/vex.json

The manifest:

{
  "name": "DarkEdges Demo VEX Repository",
  "versions": [{
    "spec_version": "0.1",
    "locations": [{ "url": "http://vex-server/v0.1/vex-data.tar.gz" }],
    "update_interval": "1h"
  }]
}

repositories:
  - name: darkedges-demo
    url: http://vex-server
    enabled: true

Then trivy vex repo download fetches it, and scanning with --vex repo
gives the same result: 0 findings, 70 suppressed, except now the
statements are centrally hosted, versioned, and every Trivy in your org picks
them up automatically on the manifest's update interval.

The three gotchas

1. The repository index needs `repository_url` for OCI purls

This one cost me an hour. The vex-repo-spec says index ids are purls
"without version and qualifiers", so pkg:oci/pingaccess, right?

No match. Silently. Trivy's index lookup (pkg/vex/repo.go) makes an
exception for OCI purls and keeps the repository_url qualifier as part
of the package identity. The index entry must be:

{
  "id": "pkg:oci/pingaccess?repository_url=index.docker.io%2Fpingidentity%2Fpingaccess",
  "location": "pkg/oci/pingaccess/vex.json",
  "format": "openvex"
}

If your repo downloads fine but suppresses nothing, check this first.

2. Embedding VEX in the image does nothing (for scanning)

Docker's Hardened Images article mentions copying VEX documents into the
image at build time:

FROM pingidentity/pingaccess:8.3.4-edge
COPY pingaccess-8.3.4-edge.openvex.json /usr/share/vex/

I tested it: Trivy does not auto-discover VEX from the image filesystem.
A plain scan of the derived image still reports all 70 findings. Embedding is
a distribution mechanism, the consumer must extract the file and pass it
via --vex. (What Trivy can auto-discover is VEX attached as a signed OCI
registry attestation, trivy image --vex oci, which requires pushing.)

3. VEX binds to the digest, derived images don't inherit it

The statements identify the product by the base image's digest. My derived
image (built locally, never pushed) has no repo digest at all, so neither
the local file nor the repository suppressed anything on it. Expected, but
worth internalizing: issue VEX for the digest you actually ship, and
regenerate when it changes.

mechanism	original image (digest-pinned)	derived image
`--vex <file>`	✅ 70/70 suppressed	❌ 0
`--vex repo`	✅ 70/70 suppressed	❌ 0
embedded file alone	n/a	❌ 0

What about Wiz?

Wiz ingests OpenVEX automatically, zero config, but from registry
attestations, not files or repositories. The hand-off from this workflow is
one command against the image in a Wiz-scanned registry:

docker scout attestation add \
  --file pingaccess-8.3.4-edge.openvex.json \
  --predicate-type https://openvex.dev/ns/v0.2.0 \
  <registry>/<org>/pingaccess:8.3.4-edge
# or: cosign attest --type openvex --predicate <file> <image@digest>

Same OpenVEX documents, two consumers: Trivy via the repository, Wiz via the
attestation. That's the point of a standard.

Try it

The repo runs the whole thing, toolchain build, baseline, 51 statements,
hosted repository, all the re-scans, with one command and colored output
(there's a GIF of the full run in the README):

git clone https://github.com/darkedges/trivy-vex-demo
cd trivy-vex-demo
./run.sh        # pauses after each step; ./run.sh -y to run unattended

Nothing is pushed anywhere; the only network traffic is Docker Hub pulls and
Trivy DB downloads. PowerShell users get an equivalent run.ps1.

If you've hit other VEX edge cases, CycloneDX VEX, CSAF, Grype's matching
behaviour, I'd love to hear about them in the comments.

Building CIS-Hardened, SBOM-Attested CentOS 9 Golden Images with Packer, QEMU and PingAccess - entirely on WSL2

DarkEdges — Sun, 28 Jun 2026 22:50:53 +0000

Most "golden image" pipelines assume a cloud builder or a beefy Linux box. This one runs end-to-end on a Windows laptop, inside WSL2 with nested KVM, and produces a CentOS 9 Stream QCOW2 image that is:

CIS Level 1 hardened (via the official ansible-lockdown role)
Pre-loaded with PingAccess 8.3.5 on a JRE 17 runtime
Shipped with a signed SBOM and VEX attestations so a Trivy scan tells you what's actually exploitable - not just what's theoretically vulnerable

Everything is driven by a single ./build.sh. Here's how it fits together, and the WSL2-specific gotchas I had to solve along the way.

The shape of the pipeline

The build is split into two Packer sources that run sequentially, because the second one boots the artifact the first one produced.

CentOS 9 Stream ISO
        │  Build 1 - "harden"
        ▼
  kickstart install ──► CIS Level 1 hardening (Ansible) ──► base SBOM
        │
        ▼
  output/base/centos9-base.qcow2
        │  Build 2 - "pingaccess"
        ▼
  JRE 17 + PingAccess ──► VEX/SBOM artifacts ──► app SBOM
        │
        ▼
  output/app/centos9-app.qcow2
        │  "sbom" target
        ▼
  Trivy scan (VEX-suppressed) + cosign-signed VEX + CVE report

build.sh exposes each stage as a target so you can iterate on just the part you're changing:

./build.sh harden        # ISO → CIS-hardened base image
./build.sh pingaccess    # base image → Java + PingAccess + security artifacts
./build.sh sbom          # scan + sign an existing app image
./build.sh               # all of the above, end to end

Build 1: a CIS-compliant base image from a kickstart

The foundation is an automated CentOS 9 Stream install. The interesting part is the partition layout - CIS expects several mount points to be isolated filesystems with restrictive flags, so the kickstart lays them out as separate LVM logical volumes:

part /boot            xfs   512 MiB
volgroup centos_vg
  lv_root      /                6 GiB   xfs
  lv_tmp       /tmp            512 MiB  xfs   nodev,nosuid,noexec
  lv_var       /var              2 GiB  xfs
  lv_varlog    /var/log        512 MiB  xfs
  lv_audit     /var/log/audit  512 MiB  xfs
  lv_home      /home           512 MiB  xfs   nodev,nosuid
  lv_swap      swap              1 GiB

SELinux is left enforcing, the firewall is enabled with only SSH open, and unused network services are stripped out.

Hardening itself is delegated to the maintained ansible-lockdown.rhel9_cis role rather than a hand-rolled script. A few overrides keep it Packer-friendly:

rhel9cis_level_1: true
rhel9cis_level_2: false
os_check: false                 # role asserts != CentOS; Stream == RHEL 9 controls
skip_reboot: true               # Packer powers off after all provisioners
rhel9cis_rule_5_2_2: false      # "Defaults use_pty" hangs sudo under pipelining
rhel9cis_sudoers_exclude_nopasswd_list: [packer]   # keep build sudo mid-play

That last group is the kind of thing you only learn by watching a build hang at 80%: CIS rule 5.2.2 adds Defaults use_pty to sudoers, which deadlocks sudo when Ansible pipelining is on and no PTY is allocated.

Build 2: PingAccess, the noexec-/tmp problem, and a lean image

Build 2 boots the hardened base over SSH (key-based now - password auth is gone) and layers on Java 17 and PingAccess.

The first surprise: /tmp is 512 MiB and mounted noexec thanks to the CIS layout. That breaks two assumptions at once - the PingAccess archive doesn't fit, and you can't execute staged binaries from there. The fix is to stage and extract under /opt (which lives on the 6 GiB root LV):

- name: "PA | Extract PingAccess archive"
  ansible.builtin.unarchive:
    src: /opt/pa_staging.zip
    dest: /opt/pa_extract/
    remote_src: true

To keep the image lean, the role deletes the PingAccess SDK samples after install (nobody runs the Maven sample projects in production), and the final Packer step does dnf clean all + fstrim -av. Combined with disk_discard = "unmap" and disk_compression = true, the TRIM is the single biggest lever on final image size.

The interesting part: shipping security context inside the image

A plain vulnerability scan of any real product image is mostly noise - hundreds of CVEs in transitive dependencies that aren't reachable, aren't loaded, or are already mitigated. The goal here was an image that carries its own answer to "is this actually exploitable?"

SBOMs generated inside the VM

Rather than scanning the QCOW2 from the host (which gives you the host's view, and made my base and app SBOMs come out identical at one point), SBOM generation runs inside each VM during its build via a small generate_sbom Ansible role. It downloads syft, scans /, writes the SBOM to /opt/security/sbom/, then fetches a copy back to the host and removes the binary:

- name: "SBOM | Generate SBOM"
  ansible.builtin.shell:
    cmd: >
      syft / --output spdx-json
      --file {{ sbom_output_dir }}/{{ sbom_filename }}
      {% for e in sbom_excludes %}--exclude {{ e }} {% endfor %}

- name: "SBOM | Fetch SBOM to build host"
  ansible.builtin.fetch:
    src: "{{ sbom_output_dir }}/{{ sbom_filename }}"
    dest: "{{ sbom_host_dest }}"
    flat: true

Now you get a true base SBOM (OS only) and a true app SBOM (OS + JRE + PingAccess), and both are embedded in the image for later audit.

VEX: turning "vulnerable" into "not affected"

OpenVEX statements describe whether a product is actually affected by a given CVE. The pipeline collects *.openvex.json statements, merges them with vexctl, signs the consolidated document with cosign, and embeds it at /opt/security/vex/:

cosign generate-key-pair --output-key-prefix cosign   # COSIGN_PASSWORD="" for unattended
cosign sign-blob --key cosign.key --yes consolidated.openvex.json

(Keyless OIDC signing is tempting but hangs in an unattended build - a local key with an empty password is the right call for CI.)

Trivy + VEX = signal, not noise

Finally, Trivy scans the mounted image filesystem with the VEX document applied, so suppressed findings are shown but explained:

trivy fs --vex consolidated.openvex.json \
  --severity CRITICAL,HIGH,MEDIUM,LOW \
  --show-suppressed --format json image-root/

A small Python summariser (no jq dependency) turns the JSON into a per-severity CVE count, including how many were suppressed by VEX. There's a NO_VEX=1 switch to see the raw, unsuppressed picture when you want it.

Mounting a QCOW2 offline without a running VM

To scan the image's filesystem (not just packages), build.sh mounts the QCOW2 read-only via qemu-nbd. Two tricks make this reliable:

qemu-nbd --connect=/dev/nbd0 --snapshot app.qcow2   # CoW overlay → no exclusive lock
kpartx -av /dev/nbd0
pvscan --cache
vgchange -ay centos_vg
udevadm settle
mount -t xfs -o ro,norecovery /dev/centos_vg/lv_root /mnt/packer-scan

--snapshot writes changes to a throwaway overlay, so the original file is never locked exclusively - you can even scan while being careful about a running VM. And because the root filesystem lives on LVM, you can't just mount the partition; you have to pvscan/vgchange to activate the volume group first. The norecovery mount option avoids XFS log replay on a read-only device.

Running it

A companion run.sh boots either image under QEMU/KVM with port forwarding wired up:

./run.sh           # boot the app image (PingAccess)
./run.sh ssh       # boot + drop into an SSH session
./run.sh stop      # graceful shutdown over SSH, SIGTERM fallback

2222  → 22     SSH
19000 → 9000   PingAccess Admin UI
13000 → 3000   PingAccess Engine

WSL2 lessons worth stealing

Use -cpu host. The default qemu64 vCPU lacks SSE4.2/XSAVE and the CentOS 9 kernel panics on boot.
Build to an ext4 path, not /mnt/c or /mnt/e. QCOW2 I/O over the 9P/NTFS bridge is unreliable; build on the Linux filesystem and copy out afterward.
q35 machine type is required - CentOS 9's initramfs uses AHCI for CD-ROM access.
scp_extra_args = -O forces legacy SCP mode, which Ansible's file transfers need against the hardened sshd.
Trivy reads embedded SBOM/SPDX files and will report CVEs for components listed there even after you've deleted them from disk. Skip those directories in the scan, or it'll look like your cleanup didn't work.

Takeaway

The result is a reproducible, single-command build that turns an ISO into a hardened, application-ready image whose security posture travels with it: an embedded SBOM that says what's inside, signed VEX attestations that say what's actually exploitable, and a Trivy report that separates signal from noise - all on a laptop.

If you're building golden images, consider making the SBOM and VEX first-class build artifacts rather than an afterthought bolted on at scan time. The difference between "412 CVEs" and "412 CVEs, 6 actually relevant, here's the signed proof" is the difference between a report nobody reads and one a security team can act on.

GitHub

The repository is at https://github.com/darkedges/centos9-qemu

DEV Community: DarkEdges

My detector caught the attacker and never once stopped it, and reported PASS

The bug

Why it survived so long

The same shape, five more times

Why this class specifically

What actually helped

The question worth asking first

A graduated response ladder where every rung is invisible

The ladder

Fail open, deliberately

Covert enforcement

The two rungs that did nothing

throttle is a budget, not a delay

step_up is indistinguishable from token expiry

Two ordering bugs, both found by getting them wrong

A challenge is not a block

What the ladder is for

Honeytokens that recognise themselves: stateless decoys with automatic attribution

Derivation: make the decoy recognise itself

Recognition happens at the gateway, and only on a miss

Rotation

The allowlist, and what it does not buy

Honeytokens as training labels

Four ways a baseline quietly destroys the anomaly detector built on it

1. The peer group contained the client it was judging

2. Baselines weren't keyed by window size

3. The boiling frog

4. The all-time set that should have been a trailing window

5. (Bonus) The feedback loop I created myself

What these have in common

Three detection layers that disagree usefully, and why they combine by max, not sum

Layer 1: guardrails

Layer 2: robust statistics

Peer groups

Layer 3: an extended isolation forest

Calibration, not constants

Two deliberate constraints

Combining: max, not sum

Attribution is the point

Tiers

The most obvious signals for detecting enumeration don't work

The trap

The reframe

id_sequentiality

miss_ratio and not_found_ratio

interarrival_cv

distinct_src_ips / distinct_asns

endpoint_kl_divergence

subject_mismatch_ratio

novelty_run_length, the feature about time

The features that are computed and never scored

Windows

Simulating attackers is easy. Simulating legitimate users is the hard part

The legitimate integration

The attackers

The two that aren't attackers, and aren't quite normal either

The ambiguous automation

The internal scanner

Why the boring clients matter more

Determinism

Your API is being enumerated by a client with a perfectly valid token

Authorization is the wrong tool, structurally

What's left: the shape of the sequence

The architecture, briefly

What makes this hard is not the attackers

What "detection" has to mean

What this series covers

Cross-App Access Without Leaking Credentials: A Hands-On ID-JAG POC

What it is

Architecture

What's been shaking out recently

Try it

Proving Zero Trust Actually Works: Entra ID + Cloudflare Access over Both OIDC and SAML

The setup

Three users, three apps, one deliberate trap

Identity you can actually verify, not just see

Design decisions worth stealing

What this demo is not

Why bother

`throttle` is a budget, not a delay

`step_up` is indistinguishable from token expiry

`id_sequentiality`

`miss_ratio` and `not_found_ratio`

`interarrival_cv`

`distinct_src_ips` / `distinct_asns`

`endpoint_kl_divergence`

`subject_mismatch_ratio`

`novelty_run_length`, the feature about time

Step 3, The instant win: `--vex` flag

1. The repository index needs `repository_url` for OCI purls