This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
My first eval scoreboard said both engines failed almost every case. Local llama3.2: 5 out of 6 malformed. Anthropic Sonnet: 6 out of 6 malformed.
When I opened the raw records, three different failures were hiding under the same word — malformed:
- An API error (credits exhausted) that never reached the model.
- Terminal control bytes from the Ollama CLI corrupting captured output.
- Valid JSON wrapped in a markdown fence that the parser refused to strip.
Three causes, one label, zero indication on the scoreboard which was which. If I had published that first summary, I would have lied in public about two models that were never honestly measured.
The project
memory-authority-auditor is a research tool that detects when one instruction in an AI agent's memory silently overrides another. A semantic proposer (LLM) reads a set of memory items and proposes authority changes. A deterministic confirmer checks whether each proposal has a verbatim citation in the source text before it counts as a finding.
The eval harness runs both engines (Anthropic Sonnet and local llama3.2 via Ollama) against a frozen fixture of 6 test cases — 4 positives that should produce a finding and 2 negatives that should not — then records every raw output, every proposal, every scoring decision, and writes it all to a timestamped JSON artifact.
The fixture and scoring rules were committed and frozen before the eval code ran. The harness was not supposed to be the thing under test. It was.
Bug one: terminal UI is not a data API
The local llama path used subprocess.run to call the Ollama CLI:
subprocess.run(
["ollama", "run", "llama3.2", prompt],
capture_output=True,
text=True,
timeout=60,
)
The problem: ollama run is a terminal UI command. It produces spinner animations, cursor repositioning, line-erase sequences — things a human terminal handles and a human never sees. When you capture that stdout as data, those control bytes land in your capture path.
Here is what the raw output actually contained in the July 1 artifact (20260701T225629Z):
"analysts may export only aggregated customer \x1b[K\nmetrics,
never raw customer records, unless the privacy lead grants
written\x1b[7D\x1b[K\nwritten approval."
\x1b[K is "erase to end of line." \x1b[7D is "move cursor back 7 positions." These are ANSI terminal control sequences. They belong in a terminal emulator, not in a JSON field.
The parser tried json.loads() on output that had invisible control bytes embedded in the middle of otherwise valid JSON strings. 5 out of 6 cases came back malformed. The one case that parsed cleanly actually produced a confirmed finding — the model identified the right authority change, cited the right span, and the confirmer verified it. The model was doing real work, and the pipe was garbling the answer on the way out. The eval summary said the model failed. The model did not fail. The capture path confused a terminal UI channel with a data API.
The result was not a model-quality measurement. It was a capture-path measurement pretending to be one.
Bug two: valid JSON wrapped in a markdown fence
Eight days later, I fixed the llama capture by switching from CLI subprocess to Ollama's HTTP API. Llama went from 5/6 malformed to 0 malformed immediately.
Then I ran Sonnet. Still 6 out of 6 malformed. Different artifact (20260709T191930Z), same scoreboard: zero proposals parsed, zero findings, every case marked malformed.
I opened the raw output for the first case. It started with json ` and ended with `. Inside the fence:
{
"proposals": [
{
"type": "supersedes",
"source_item_id": "M002",
"target_item_id": "M001",
"cited_evidence_span": "the migration exception is retired.",
...
}
]
}
That is valid JSON. Every field the parser expected was there. The model had returned structured proposals for every case, and the parser rejected all six because it ran json.loads() on text that started with three backticks instead of a curly brace.
This is not an exotic failure. Most LLMs wrap JSON responses in markdown fences by default. Any eval harness that calls json.loads() on raw model output without stripping common wrapping will hit this — and the scoreboard will say "malformed" when the model answered fine.
The fix was a few lines:
def _strip_code_fence(text: str) -> str:
t = text.strip()
if t.startswith("```
"):
t = t.split("\n", 1)[1] if "\n" in t else t[3:]
if t.rstrip().endswith("
```"):
t = t.rstrip()[:-3]
return t.strip()
Then one change in the parser:
parsed = json.loads(_strip_code_fence(text))
The result changed immediately. Not because the model got smarter. Because the harness finally read what the model actually said.
Eval pipelines do not fail at the model first. They fail at the pipe, then blame the model.
The fix in one diff
Both bugs were fixed in a single commit (49902b2):
Llama capture — replaced the subprocess CLI call with a direct HTTP request to Ollama's /api/generate endpoint, stream: false, parsing the JSON response payload. The model output channel became machine-readable instead of terminal-shaped.
Sonnet parser — added _strip_code_fence() before json.loads(). The parser now accepts the most common wrapping models use without weakening the downstream gate or scoring rules.
The commit touched 54 lines in one file (agents/semantic_proposer.py). The frozen fixture was not modified. The scoring rules were not modified. The confirmer gate was not modified. Only the capture layer changed.
Two independent bugs, two different engines, two different layers of the pipe — and both produced the same label on the scoreboard: malformed. That word is not a diagnosis. It is a symptom. And if your harness does not record why something was malformed, you will treat every capture failure as a model failure.
I kept the broken runs on purpose
I did not delete the broken runs. They are committed in git, side by side with the clean ones:
| Artifact | Engine | Malformed | What happened |
|---|---|---|---|
20260701T225629Z |
llama3.2 | 5/6 | ANSI control bytes from CLI subprocess |
20260701T225629Z |
Sonnet | 6/6 | HTTP 400 — API credits exhausted |
20260709T191930Z |
Sonnet | 6/6 | Valid JSON wrapped in markdown fence |
20260709T190750Z |
llama3.2 | 0/6 | After HTTP API fix |
20260709T192344Z |
Sonnet | 0/6 | After code-fence parser fix |
20260709T202859Z |
Both | 0/18 | v1 full run, 18 cases, both engines clean |
The July 1 llama artifact and the July 9 pre-fix Sonnet artifact are the embarrassing ones. They stayed because an eval artifact is not a trophy. It is a receipt. A receipt that says "harness was broken here" is more useful than a clean chart with no provenance.
You can verify any of this in under 60 seconds:
git show 49902b2 -- agents/semantic_proposer.py
# then open:
# path_a_eval_artifacts/path_a_eval_20260701T225629Z.json (search \x1b)
# path_a_eval_artifacts/path_a_eval_20260709T191930Z.json (raw_output starts with ```
{% endraw %}
json)
{% raw %}
What the clean run actually showed
Once the harness stopped lying, the real results appeared in the v1 18-case run (20260709T202859Z):
Anthropic Sonnet:
- 12/12 positive cases direction-caught (correct source/target pair identified)
- 7/12 positive cases exact-match (correct relation type label)
- 0/6 negative false fires
- 0 malformed
Local llama3.2:
- 4/12 direction-caught
- 0/12 exact-match
- 1/6 negative false fire
- 0 malformed
The gap between the two models became a real finding only after the capture path was honest. Before the fix, both engines could look dead on a summary that never distinguished capture failure from model failure. After the fix, the actual model-quality difference was visible — and measurable — for the first time.
The split between direction-caught and exact-match is intentional — direction means the model found the right pair of instructions and knew one changed the other; exact means it also named the specific type of change correctly. Both metrics were frozen before the run. This post is about whether the trial was fair, not about crowning a model.
Once capture was honest, weak-model behavior became visible too. The deterministic confirmer blocked five of llama's six would-be false fires in the v1 run. The one that got through cited a verbatim span from a restatement to claim a supersession — a real citation supporting a wrong relation. That whole signal was invisible when 5 out of 6 cases were malformed noise.
What became more resilient
Specific changes that prevent the next version of these bugs:
- Transport is API-shaped. The model output channel is an HTTP JSON payload, not terminal stdout. No cursor repositioning, no spinner bytes, no line-erase sequences in the data path.
-
Parser tolerates common wrapping.
_strip_code_fencehandles the most common model output wrapping without relaxing the schema validation that runs after parsing. -
Malformed is recorded with a reason, not just a count. Every case records its malformed status and the specific
malformed_reasonsstring. The July 1 Sonnet artifact says"http 400: Your credit balance is too low"— not justmalformed: true. Without that reason field, credits and ANSI corruption look identical on a summary. The reason is what lets you triage. - Frozen fixture stays frozen. The scoring rules and test cases were not changed to make results prettier. The fix touched the pipe, not the standard.
- Raw outputs stay in the artifact. The next person who doubts a summary can open the JSON, read the raw model output, and recompute the score from scratch.
The lesson
The model is not the only thing on trial. The harness is part of the system under test.
Every eval pipeline has hidden trust embedded in the layers between the model and the result: the subprocess call, the stdout capture, the response parser, the serializer, the scorer, the summary renderer. If any of those layers is wrong, "model failed" becomes a harness lie dressed up as a research finding.
The fix is not more confidence in your pipeline. The fix is:
- Keep the raw output. If the summary says malformed, the raw output is the appeals court.
- Freeze the standard before you run. If the fixture and scoring rules can change after you see results, you are not evaluating — you are negotiating.
- Commit the embarrassing artifact. The broken run is the proof that the fixed run means something. Delete it and you delete the delta.
My harness lied to me twice, for two different reasons, on two different engines. I caught it both times because I opened the raw output instead of trusting the summary. If I had reported the first scoreboard, I would have published a false claim about two models that were never honestly measured.
The research question never got a fair trial until the pipe was honest. That is the bug nobody checks for.
Repository: memory-authority-auditor
Key commit: 49902b2 — Fix both proposer capture bugs
Artifacts: All pre-fix and post-fix eval runs committed in path_a_eval_artifacts/
Top comments (0)