DEV Community

jaryn
jaryn

Posted on

Pin Your AI Reviewer to a Fixture Corpus Before You Change the Model Under It

#security #ai #testing #devops

A silent regression hit us after what looked like a routine cost optimization: someone pointed our review assistant at a different model. The tool kept producing fluent comments. It also walked past a path traversal in an upload handler — a pattern the old model had flagged on sight for months. A versioned fixture flipping from pass to fail was the only alarm that fired, and it fired two days after the change.

The takeaway I now treat as non-negotiable: a model is a pinned dependency, and swapping it without regression evidence is an unreviewed change to your security posture. Below is a compact workflow for gating any model change — paid, self-hosted, or free — on a fixed corpus of known-answer cases before it earns a place in CI.

Trust boundary first

The assistant ingests untrusted code and emits judgments. Three drift modes matter, and none surface in a smoke test:

  • Silent capability loss — detections it used to make quietly disappear.
  • Noise inflation — it starts flagging benign code, training reviewers to dismiss everything.
  • Refusal creep — it declines to analyze security-sensitive files at all, which is a detection outage wearing a safety costume.

Only a corpus with ground truth distinguishes these from 'the model seems fine.'

Build the corpus: 12–20 files, each with a verdict

Every case is one source file plus a machine-readable expectation. Keep it in the same repo as the gate:

corpus/
  traversal_join/            # must flag
    handler.go
    verdict.yaml             # verdict: flag, severity_at_least: high
  traversal_sanitized/       # must pass (negative twin)
    handler.go
    verdict.yaml             # verdict: pass
  jwt_alg_none/              # must flag
    auth.py
    verdict.yaml
  secrets_in_config/         # must flag
    settings.yaml
    verdict.yaml
Enter fullscreen mode Exit fullscreen mode

Positive case (corpus/traversal_join/handler.go):

func serveFile(w http.ResponseWriter, r *http.Request) {
    // fixture: user input joined directly into a filesystem path
    name := r.URL.Query().Get("f")
    http.ServeFile(w, r, "/var/data/"+name)
}
Enter fullscreen mode Exit fullscreen mode

Negative twin — the same endpoint shape, sanitized, and the model must stay silent on it:

func serveFile(w http.ResponseWriter, r *http.Request) {
    name := filepath.Clean(r.URL.Query().Get("f"))
    if strings.Contains(name, "..") || filepath.IsAbs(name) {
        http.Error(w, "invalid", http.StatusBadRequest)
        return
    }
    http.ServeFile(w, r, filepath.Join("/var/data", name))
}
Enter fullscreen mode Exit fullscreen mode

The twin structure is the point. A model that flags both files is a noise generator; one that flags neither is a rubber stamp. Only the gap between the two verdicts carries information.

Gate script (template — wire it to your endpoint before trusting it)

This follows a pattern I run against local inference; treat the listing as an unexecuted template until you've pointed it at your own deployment. Any OpenAI-compatible endpoint works:

#!/usr/bin/env python3
"""Model-swap gate. python >= 3.11, openai >= 1.40, pyyaml"""
import json, os, sys, pathlib, yaml
from openai import OpenAI

SYSTEM = (
    "You audit code for exploitable defects. Respond with JSON only: "
    '{"verdict": "flag" | "pass", "severity": "low" | "medium" | "high", "evidence": str}'
)
SEV_RANK = {"low": 1, "medium": 2, "high": 3}

api = OpenAI(base_url=os.environ["MODEL_BASE_URL"],
             api_key=os.environ.get("MODEL_API_KEY", "unused"))
MODEL = os.environ["MODEL_NAME"]          # exact pinned ID; "latest" is banned

broken = []
for case in sorted(pathlib.Path("corpus").iterdir()):
    src = next(p for p in case.iterdir() if p.name != "verdict.yaml")
    want = yaml.safe_load((case / "verdict.yaml").read_text())
    reply = api.chat.completions.create(
        model=MODEL, temperature=0,
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": src.read_text()}],
    ).choices[0].message.content
    try:
        got = json.loads(reply)
    except json.JSONDecodeError:
        broken.append((case.name, "non-JSON output"))
        continue
    ok = got.get("verdict") == want["verdict"]
    if ok and want["verdict"] == "flag":
        ok = SEV_RANK.get(got.get("severity"), 0) >= SEV_RANK[want["severity_at_least"]]
    print(f"{'ok ' if ok else 'BAD'} {case.name}: want={want['verdict']} got={got.get('verdict')}")
    if not ok:
        broken.append((case.name, got.get("evidence", "")[:100]))

print(f"\nmodel={MODEL} failures={len(broken)}")
sys.exit(1 if broken else 0)
Enter fullscreen mode Exit fullscreen mode

Then compare incumbent against candidate:

MODEL_NAME=<incumbent-exact-id> python gate.py | tee incumbent.log
MODEL_NAME=<candidate-exact-id> python gate.py | tee candidate.log
diff incumbent.log candidate.log && echo "no regression on corpus"
Enter fullscreen mode Exit fullscreen mode

Constraints that turn this from a demo into a gate:

  1. Exact identifiers only. A floating tag makes every run a coin flip.
  2. Temperature 0, recorded in the log header. Otherwise flakiness and regression are indistinguishable.
  3. Flake = fail. If a case fails once and passes on retry, the candidate is rejected. A nondeterministic security judgment is not a judgment.

Where free compute fits — and where it stops

This workload is a dozen short completions, so it's a natural match for free model access: you can benchmark a candidate against your incumbent without paying for parallel inference. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently advertises free model access and a free server option (per operator-provided information; I haven't verified quotas, available models, or offer duration — confirm current terms first).

My honest split of responsibilities:

  • Free tier → the experiment. Run the corpus, diff the logs, decide whether the candidate is even worth a longer evaluation.
  • Infrastructure you control → the enforcement. The blocking CI check belongs somewhere with availability guarantees, because a gate that 404s either blocks every merge or gets bypassed — both are bad outcomes.

Prevent / detect / recover

Layer Mechanism
Prevent Gate triggers on any PR touching model ID, system prompt, or sampling params; pinned IDs in config
Detect Cron re-run of the corpus against the live endpoint — catches provider-side weight updates nobody told you about
Recover Prior model ID and endpoint stay warm for one release cycle; rollback is a config revert

Limits, and who shouldn't bother

  • A 15-file corpus proves non-regression on 15 patterns, nothing more. Grow it from real escaped incidents.
  • If the assistant's output never gates anything and humans skim it casually, a lightweight spot-check list may be proportionate.
  • If your tool hides the model identifier entirely, you cannot gate what you cannot name — demand versioned models from the vendor or accept unmonitored drift.
  • Any pipeline stage pointing at a free endpoint needs a fallback, or your safety check becomes an availability dependency.

The boundary question I keep coming back to: which of these invariants should live in the model layer at all? The traversal case above arguably deserves a deterministic Semgrep rule as the guarantee, with the model fixture retained only to measure assistant judgment. The more of your review pipeline that runs on a system which changes without notice, the more of that line you'll want to redraw toward deterministic tooling. Where would you draw yours?

Top comments (0)