226 Scenarios, 43 Detectors, 1 Clean Run: Building a Test Harness That Can't Lie

The detectors I didn't trust were the ones that stayed quiet. A rule that fires is easy to sanity-check. A rule that does nothing could mean "checked, found nothing," or "crashed before it checked," and from the outside those look the same.

I ended up with 43 detectors in agentwatch, part of the agentic-suite suite, and this is the harness I built to stop confusing the two. It runs 226 scenarios in about twelve seconds.

The shape of a scenario

Every case is a small dataclass that knows how to build a trace and what should happen to it:

@dataclass
class Scenario:
    id: str
    detector: type
    expect_fire: bool
    build: Callable                     # -> (RunSummary, list[SpanNode])
    severity: str | None = None
    pool: dict[str, Any] | None = None  # scripted DB values
    phase: str = "matrix"               # matrix | precision | llm | clean

The traces are built by hand, not sampled from a corpus, so there's exactly one thing being tested at a time:

add("L1", LoopDetector, True,
    lambda: (summ(total_tool_calls=6), [tool("search_kb") for _ in range(6)]),
    severity="warning")

Six identical calls, threshold of five, should fire a warning. No ambiguity to argue about later.

The traces are hand-built for a reason. A corpus tells you whether the detector fires on real data, but it can't tell you whether it fires for the right reason — corpus traces don't have ground-truth labels. A hand-built trace has exactly one property being tested, and the expected outcome is known before the detector runs. That's the only layer that can compute true TPR and FPR.

Faking the database

Six detectors query Postgres for baselines. Rather than spin up Postgres for the tests, they query a dict:

class ScriptedPool:
    def __init__(self, values=None):
        self.conn = _Conn(values or {})
    def acquire(self):
        return _Acq(self.conn)

The trick is in how the detectors decide whether they have a real pool. _has_valid_pool() checks for an acquire method and not a Mock's return_value — a heuristic that distinguishes a real asyncpg pool from a unittest.mock.Mock. ScriptedPool has acquire but no return_value, so it passes for the real thing and the detectors run their async queries against a dictionary. No Docker, no Postgres, no asyncpg — and about a thousand times faster than spinning up a database container.

The detectors check for an acquire method and not a Mock's return_value, so a ScriptedPool passes for the real thing and runs its async queries against a dictionary.

add("CV1", CostVsBaselineDetector, True,
    lambda: (summ(agent_version="v1", estimated_cost=6.0), []),
    severity="warning", pool={"avg_cost": 2.5})   # baseline $2.50, run $6.00

No Docker, no Postgres, and about a thousand times faster.

Real calls, not mocks

The twelve LLM scenarios hit a real model. This matters, because a mock always returns something valid and can never show you an endpoint that's down — which is exactly the bug that had been hiding in my suite for weeks. So each scenario snapshots the client's counters before and after:

stats_before = dict(client._stats)
anomaly = await det.detect_async(summary, spans, pool=None)
stats_after = dict(client._stats)

llm_calls = stats_after["chat_calls"] - stats_before["chat_calls"]
embed_calls = stats_after["embed_calls"] - stats_before["embed_calls"]

That delta is what lets the harness ask the question that matters.

Six checks, not one

Most detector suites check one thing. This one checks six: did it fire, was the severity right, was it the right detector, was there an explanation, was there an evidence dict, and — the one I care about most — was the model actually called.

tip: A negative case where the model was never called is not a pass. It's a skip wearing a pass. Assert that the dependency ran, not just that the output looked right.

There's a seventh for LLM cases too: parse the model's JSON and confirm its verdict matches what was expected, so a detector can't fire for the wrong reason. For a positive case, the model should say {"hallucination": true}. For a negative, {"hallucination": false}. If the detector fires but the model's verdict disagrees, the detector has a logic bug — it fired for the wrong reason. Verdict matching catches that.

The phase design

The 226 scenarios split into four phases, each catching a different class of bug:

Phase	Cases	What it catches
matrix	166 (154 ported + 12 hook patterns)	Per-detector fire/no-fire and severity
precision	48 (n−1 vs n for 24 thresholds)	Off-by-one and floating-point at boundaries
llm	12 (6 LLM detectors, real calls)	Silent skips, verdict mismatches
clean	1 (43 detectors, 1 normal trace)	False positives on clean input

The matrix phase is the ported predecessor suite — 154 scenarios from the original 35-detector field-test plan, plus 12 new Claude Code hook-pattern detectors. These are the "does it fire when it should" cases, and they're the layer most suites stop at.

The precision phase exists because numeric thresholds live at the boundary. Testing only at n+5 misses off-by-one errors; testing only at n hits floating-point. Two cases per threshold — one below, one at the line — catch both. Forty-eight cases across twenty-four thresholds, and that's the layer that caught the 6/9 ÷ 3/9 = 1.9999... floating-point bug.

The LLM phase is the one that needs real calls, and the one the llm_called_ok check lives in. Mocks would make this phase pass vacuously — a mock always returns a valid response, so a detector that silently fails looks like it works.

The clean phase is one case — all 43 detectors against a single known-normal trace, expecting zero fires. It's the cheapest test in the harness and the most valuable, and it gets its own article in this series.

Boundary cases

For every numeric threshold, two cases — one just below and one at the line:

prec("P-loop-4", LoopDetector, False, _tools(["search_kb"] * 4))              # under
prec("P-loop-5", LoopDetector, True,  _tools(["search_kb"] * 5), "warning")   # at
prec("P-cost-500", CostSpikeDetector, False, lambda: (summ(estimated_cost=5.0), []))
prec("P-cost-501", CostSpikeDetector, True,  lambda: (summ(estimated_cost=5.01), []))

The CostSpikeDetector uses a strict > comparison, so 5.0 doesn't fire (not > 5.0) and 5.01 does. Testing at exactly 5.0 confirms the strictness; testing at 5.01 confirms the threshold. Without both, a >= that should be > slips through.

tip: Test at n−1 and at n for every numeric threshold. The exact boundary is where off-by-one and floating-point bugs live. A test at n+5 proves the detector can fire; it says nothing about where the threshold actually is.

The clean-run check

All 43 detectors against a single known-normal trace, expecting zero fires. The clean trace has three different tools, one-second gaps, a success status, twelve cents of cost, and a 200-character output — every value far from every threshold. If anything fires, it's a false positive, full stop.

async def run_clean():
    summary, spans = _clean_trace()
    fired = [det.anomaly_type
             for det in create_all_detectors()
             if await det.detect_async(summary, spans, pool=None) is not None]
    return {"detectors": 43, "fired": fired, "ok": not fired}

It comes back empty. The full story of why this check matters — and the predecessor data that made it non-negotiable — is in the next piece in this series.

Why it holds up

The rule-based cases run in about four milliseconds. The twelve model calls take a second each and dominate the total, which is still fine for CI. The whole thing ends at 226 of 226, with a hundred percent true-positive rate and zero false positives on the clean trace.

The severity distribution across the matrix cases: 47 warning, 26 critical, 4 info, 77 silent (no anomaly). Every severity matched. The LLM telemetry across the 12 real-call scenarios: 12 chat calls, 100% JSON parse success, latency p50 ≈ 606 ms, p99 ≈ 1427 ms. All LLM request/response I/O is captured in llm-scenario-io.jsonl — 13 records with prompts, system messages, raw responses, model names, and timing. That's the audit trail: the exact thing the model said for each scenario, preserved.

The design choices that did the work were unglamorous: hand-built traces instead of a corpus, a fake database instead of a real one, real model calls instead of mocks, two cases per threshold, and the boring clean trace almost nobody writes.

The harness is checked in at scenario_validation.py. The full results are in detector-results.json.

The design question worth asking of any detector suite: does it check that the dependency ran, or just that the output looked right? The first can't be satisfied by accident. The second can — and that's where silent failures hide.

References

• Release: agentwatch v0.1.0 on GitHub · on PyPI
• Suite: agentic-suite — the full set of agent-security tools
• Field test report (FT-15, 226 scenarios)
• Detector results: detector-results.json