A reproducible LLM red-team report: why the probe corpus sha256 matters

Most LLM security reports are anecdotes. "I tried some prompts and the model leaked the system prompt." Rerun it a week later, different model version, and the claim is untestable. The fix is the boring one: make the report an artifact of a pinned pipeline.

A red-team report should say exactly five things:

1. The corpus hash. sha256 of the probe set, byte for byte. Not "I tested fifty prompts" but "I tested corpus v3, sha256 abc123." Anyone with the hash can verify they ran the same probes.
2. The model pin. Model name, version, endpoint, temperature, max tokens, and the full system prompt hash. A product name is not a model identifier.
3. The per-probe verdict. Each probe: input, output, verdict (complied / partial / refused), and the exact turn count where it flipped.
4. The run timestamp and environment. When, where, which harness version.
5. The diff. If this is a follow-up report: what changed between runs, probe by probe.

The reason the sha256 earns its place is that it converts "we improved security" from a claim into a delta. Before the prompt rewrite: three of eight probes flipped. After: one. That one line is what a reviewer, a compliance person, or your future self can act on. Without the hash, the corpus quietly drifts, people stop running the same tests, and the report becomes a museum piece.

Implementation is twenty lines. The corpus is a text file, one probe per block, versioned in git. The runner is a loop that calls the endpoint, records the raw response, and applies a verdict function per probe. The output is a markdown table plus a JSON blob for the CI system. Pin the model in the config, not in the code.

Two practical notes. First, verdict functions are where the real work is. "Did it comply" is a classification problem, and lazy implementations either mark everything clean or flag everything. Write each verdict as a small explicit check per probe ("output contains a step-by-step for X" / "output is a refusal") and test the verdict functions themselves. Second, run it on a schedule, not just on model swaps. Model providers ship silent updates, and the day your probes start failing with no change on your side is the day you learn your provider changed something.

If you want a corpus to start from, the 15-probe red-team starter kit is packaged with exactly this structure, and the endpoint at https://llmrt-companion.manhliemcn4euwlu.workers.dev/review runs a free scan if you would like a baseline before you build the pipeline yourself.

The unglamorous version of LLM security is a CI job that runs probes on every deploy and fails the build when a new one flips. That is the whole game.