My detector caught the attacker and never once stopped it, and reported PASS

The most consequential bug in this project had been there since the beginning,
survived several full end-to-end runs, and was reported as a PASS every time.

✓ PASS  slow-and-low detected within 30m (7.3m), never exceeding legit rate

That line is true. The scorer flagged it correctly, well inside the bound. What
the line doesn't say is that the attacker was served every single request it
ever made. Zero non-allow decisions, across the entire scenario. Detected and
never once stopped.

This is one instance of a pattern that accounts for more real bugs in this
project than every other cause combined: a component contributes nothing, no
error is raised, and every surrounding number stays plausible.

The bug

The scorer computes windows at 1m, 5m and 1h, and publishes each result to a
per-client key in Redis and OPA.

Each result. To the same key.

So the last writer won. And 1-minute windows close most often, so they always
won.

slow-and-low issues about two requests a minute. Its 1m windows fall below the
minimum request count and score zero. Its 5m and 1h windows accumulate the miss
ratio that earns a deny. Every one of those zeroes immediately overwrote the
deny.

The entire premise of a multi-scale pipeline, that different attacks are visible
at different scales, was silently violated by the publication step. Any detection
that only appeared at a coarser scale was discarded.

The fix is a roll-up: publish the most severe verdict across window sizes within
the freshness horizon the policy already uses. Afterwards, the same attacker is
denied on 24–31 of its 44 requests.

Why it survived so long

Because the report could not express it.

Detection latency was computed as the earlier of two very different facts: the
scorer's first non-allow window, and the gateway's first non-allow decision.
Printed under one heading, detected, a client that was noticed but never
touched looked identical to one that was noticed and blocked.

A report that averages over the distinction you are trying to verify cannot
verify it.

The report now derives two facts from two sources and asserts both:

• detected: the scorer's first non-allow window, from risk_scores
• enforced: the first request the gateway applied a non-allow tier to, from the access log, which is the record of what actually happened

✓ PASS  slow-and-low responded within 30.0m (6.3m; scorer 6.3m, gateway 6.5m)
✓ PASS  slow-and-low ENFORCED — 31/44 requests stopped

Splitting them paid for itself immediately, and not only on the bug it was built
for. Look at the dictionary profile:

✓ PASS  dictionary responded within 60s (9s; scorer 76s, gateway 9s)

Enforced at 9 seconds, detected at 76. The gateway's fast path stopped it long
before the windowed scorer produced a single verdict. Enforcement preceding
detection is not an anomaly: it's the fast path doing its job, and the old
report was structurally incapable of showing it.

The same shape, five more times

Once I had a name for it, it was everywhere.

Tables with no writer. access_events: the raw log the whole "every decision
is reconstructable" claim rests on, was a schema definition and a materialized
view with no producer. Discovered only when I built a dataset export on top of it
and it came back empty. Then detection_labels, same thing. Then
client_minute_card, a materialized view aggregating cardinality per client per
minute, populating on every single event, queried by nothing.

An environment-dependent derivation. The label-joining query took its window
length from WINDOW_MS, which is divided by DEMO_SPEED. The demo sets that to
8; the export script doesn't set it at all. So the export bucketed events into
60-second windows against 7.5-second data and matched almost nothing: 172 of 197
labels silently vanished, and the CSV still looked perfectly well-formed, just
smaller.

The fix generalises. Derive window lengths from the data, since
client_features records both ends of every window, and never from the
environment that happens to be reading it.

Untyped tests. The package builds exclude **/*.test.ts, correctly, so tests
aren't emitted into dist/. But that also excluded them from type checking.
Three times a test fixture went stale after a field was added: the missing
property became undefined, Number(undefined) became NaN, and the failure
surfaced as an apparent detector regression.

The verification is the part worth keeping. Delete one field from one fixture:

$ tsc -p tsconfig.tests.json
ml.test.ts(17,3): error TS2719: Types of property 'route_profile' are
  incompatible. Type 'string | undefined' is not assignable to type 'string'.

$ vitest run packages/scorer/src/ml.test.ts
      Tests  15 passed (15)

The type checker names the field and the file. The test suite reports fifteen
passes. A test asserting on a NaN that flows through the arithmetic without
throwing is not a test failure: it's a test quietly measuring nothing.

A component that produced nothing at all. At DEMO_SPEED=8 a 1-hour window is
450 real seconds and the scenario runs for 30 scenario-minutes, half of one
window. Whether any 1h window closes depends on where the run falls relative to
a boundary. Most runs produce zero. The isolation forest can never fit a 1h model.
A third of the advertised 1m/5m/1h pipeline contributes nothing, and said
nothing about it.

A duplicated definition that drifted. The dashboard keeps its own list of
which clients are hostile, because it's a separate build. That copy drifted twice
while I was adding clients, and the failure is quiet in the dangerous direction:
an unlisted attacker defaults to "legitimate", so its every escalation is scored
as a false positive and the confusion matrix reports numbers that never happened.

Why this class specifically

Two structural reasons, and both are common.

The pipeline swallows errors. Insert failures are caught and dropped, because
a detection layer should not take down the API it protects. That's the right
call, and it converts every write failure into silence.

The report was derived from the components it was meant to verify. Detection
latency came from the scorer's own records. If the scorer says it flagged
something, the report said it was detected. The one thing that could have
contradicted it, what the gateway actually did, wasn't consulted.

What actually helped

Make absence visible. The report now prints how many windows each size
produced, and says so explicitly when one produced none:

labelled windows (detection_labels): 197   [1mx160  5mx37  1hx0]
  note: 1h produced no verdicts — the scenario is 30 scenario-minutes long,
  shorter than those windows. Run a longer one with e.g. SCENARIO_MIN=130.

Silent absence is indistinguishable from silent breakage. A component that
contributes nothing should say so.

Assert against the independent record. Not the detector's own output. The
access log is what the gateway did; risk_scores is what the scorer concluded.
They are different facts and deserve different assertions.

Guard the guard. The report is now only as trustworthy as the access log it
reads: a new single point of failure. So the agents keep their own request counts
purely to cross-check it, and the report warns when the two disagree.

Pin duplication you can't remove. The dashboard's copy of ground truth can't
be eliminated without restructuring the build graph. The drift can:

it('lists exactly the attack clients, no more and no fewer', () => {
  expect(dashboardAttackers().sort()).toEqual([...ATTACK_CLIENT_IDS].sort());
});

Type-check the code you excluded from the build. tsconfig.tests.json, with
noEmit, wired into pnpm test. It has caught a stale fixture in every round
since.

The question worth asking first

The habit I'd most like to pass on is a change in first instinct.

When something looks off in a system like this, the reflex is "where's the logic
error?" For six consecutive bugs here, the productive question was different:

Which component is silently doing nothing?

Not what is wrong with this calculation, but is this calculation happening at
all. Every one of these bugs produced plausible output. Several produced output
that was internally consistent. One reported PASS on the exact scenario it was
failing.

The corollary, if you build detection systems: your instrumentation is a component
too, and it fails the same way. Ask what your report cannot express, because
that's the shape of the bug you won't find.

---

That's the series. The code is a demonstration artifact, deliberately readable,
deliberately argued with, and honest about the parts that don't work. The
docs/DECISIONS.md file in the repo is 28 sections, and most of them record
something that was wrong and how it was found. That turned out to be the most
useful thing in it.