Too fast is wrong

probatum is a test-oriented check runner I build (in Rust): one TOML file, embedded curl and grep, process ownership, only the failures that matter. This post stands on its own.

A login endpoint verifies a password with Argon2. The security property is not that the right password passes and the wrong one fails — any string comparison does that. The property is that a wrong password costs the attacker real time. Argon2's parameters exist to be slow; that is what makes a wordlist impractical.

Measured on the app under test: forty wrong-password attempts, 13.4 seconds, about 336 ms each.

Now picture a refactor that swaps the hash for a string comparison. Same 401. Same response body. Every functional check stays green. The only visible difference is that the endpoint got two orders of magnitude faster — and nobody looks at that. It is exactly the regression nobody notices.

Here is what I wanted to write in probatum, and could not:

[[check]]
name = "a wrong password costs real time"
post = "http://127.0.0.1:3000/auth/login"
body = '{"username":"editor","password":"wrong"}'
expect = 401
min_ms = 20        # what I want and cannot write

max_ms had existed since v0.5.0: a response that is too slow fails. The reverse did not exist. Issue #8 was opened on August 15th.

Why not a workaround

probatum has an escape hatch: run, an arbitrary command. To measure time, you reach for date +%s%N. I tried, and it does not survive contact with the image:

$ docker run --rm --entrypoint sh ghcr.io/probatum-org/probatum:0.9.0 \
    -c 'command -v date seq curl; date +%s%N'
/bin/date
1786805331

No curl, no seq, and BusyBox date does not implement %N: %s%N yields whole seconds, and any elapsed-time arithmetic built on it is nonsense. That is not a complaint about the image — a minimal image is the right call. But it means run cannot stand in for a timing assertion the way it stands in for other gaps. It had to be a keyword.

Forty-two days

Every previous probatum issue had closed the same day: post: (#1), the cookie jar (#5), absent on HTTP (#6), put/patch/delete (#7). Each produced its release within the day. #8 waited five weeks, during which the repo received no commit at all.

Then, in three days, four releases.

v0.12.0 closes #8. min_ms mirrors max_ms, with the distinction that matters: a response that is too fast is a failure (exit 1, with the measured time as evidence), not a "couldn't observe" (exit 2). A login answering in 2 ms is wrong, not unavailable. And the README adds a fair remark: a floor is a sturdier bound than a ceiling, because noise makes a measurement slower and almost never faster. Declaring min_ms above max_ms is a config error.

v0.11.0 is the big one, and it breaks the JSON schema — from version 2 to version 4. Checks now group into named scenarios, with numbered steps, an OS scope, and captured values that one step produces and another consumes. Two decisions in it worth pointing at:

• Steps must be declared in ascending order. [auth.2] written after [auth.10] is an error, not something the runner politely reorders. The file always reads in the order it runs.
• A scenario excluded by its OS scope is still fully validated before being filtered out. An exclusion cannot hide a typo.

The schema change is breaking and stated as such: a consumer that reads log_file unconditionally, or switches on the status, has to handle the new Excluded status and the null. No silent compatibility.

v0.13.0 is the dependabot group. v0.13.1 is the release that taught me the most.

The first 100,000 lines

Since v0.6.0, a command's captured output is capped in memory. It is a good rule: a run that writes gigabytes must not blow up the runner.

Except the contains and absent rules were evaluated on that copy, not on what the command had really produced. Past the bound it lied in both directions: a contains failed on output that did contain the pattern — a false alarm — and an absent passed over a FATAL — a false green. A service's crash filter and a passing command's summary had the same blind spot.

That is the defect I spent the summer writing about, in cidx, in Verbose: success reported for work that never happened. And it sat in the one tool whose only job is to prevent that. v0.13.1 evaluates the rules line by line as output arrives; the bounded copy serves diagnosis only. The cap limits what you keep as evidence, never what you check. Two dogfooding checks lock both directions.

I know no better illustration of why you build these tools for yourself first: you do not find this by reading the code. You find it by using it.

Meanwhile, Verbose

The same week, Verbose had its own — 25 PRs, 49 commits, the largest in its history — and one of them says the same thing as min_ms, from the other end of the stack.

proofs.native_stack: 96 declares that a rule will consume no more than 96 bytes of native stack. Not a comment, not a wish: the compiler counts — input slots, the saved rbp, formatting temporaries — and refuses to emit the binary if the total exceeds it. 96 passes a bound of 96 and fails at 95. A sufficient declaration leaves the emitted code byte-identical; a wrong one gets no binary at all. And the analyses the compiler cannot do — self-hosted stack, WASM — are refused rather than guessed.

A floor on a time, a ceiling on a stack. In both cases the author writes a number, and the tool either holds it or refuses. There is no third state where it pretends.

And the usual limit: Verbose still has no tag since v0.10.0, ~133 commits ahead. probatum cut four in three days — after five weeks of cutting none. Two rhythms, neither of them a cadence. They are windows.

Too fast is wrong. It holds for a login, and it holds for a test that passes without having read everything.

French original on arcker.org.