The Scoreboard Lied. Now Sentry Shows Which Layer Broke

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Two capture-layer bugs made a working eval harness report false failures. Fixing them exposed a third integrity seam in the scoreboard.

My first eval scoreboard looked catastrophic.

• Local llama3.2: 5 out of 6 malformed.
• Anthropic Sonnet: 6 out of 6 malformed.

If I had stopped at the summary, the conclusion would have been easy: both engines failed.

Then I opened the raw records.

Across the runs, the same word — malformed — was hiding three completely different events. One request never reached a model because the API credits were exhausted. One local response was corrupted by terminal control bytes leaking through a CLI capture path. One Sonnet response was valid JSON wrapped in markdown fences that my parser refused to strip.

The scoreboard had compressed “no model run,” “transport corruption,” and “valid answer rejected by the parser” into one red cell. Those are not three versions of the same failure. They live in different layers, require different repairs, and support different conclusions. None of them was evidence that a model had failed the task.

The scoreboard lied — not because it fabricated a number, but because it erased the cause behind it.

I had already told the first half of this story in Your Harness Will Lie to You Before Your Model Does. That title is already carrying the Smash Stories entry. This Clear the Lineup entry is the implementation sequel: the exact capture fixes, the scoring-integrity repair they exposed, and the Sentry instrumentation that now shows which layer broke before the summary gets to blame the model.

The moment the story changed

The eval had been frozen since July 1. By the time I could run both engines cleanly, I thought I was finally measuring model behavior. Instead, the first thing the experiment measured was my confidence in my own harness.

That confidence failed twice.

The local model was doing real work while the pipe mangled its answer. Sonnet was returning the structure I asked for while the parser rejected its wrapping. The important move was not a clever patch. It was refusing to defend the scoreboard once the raw evidence contradicted it.

That changed the question from “Which model failed?” to “Which layer produced this result?” The rest of the work followed from that correction.

Project Overview

memory-authority-auditor detects when one instruction in an AI agent's memory silently overrides another. A semantic proposer (LLM) reads memory items and proposes authority changes. A deterministic confirmer checks for a verbatim citation in the source text before any proposal counts as a finding.

The eval harness runs two engines (Anthropic Sonnet and local llama3.2 via Ollama) against a frozen fixture, records every raw output and scoring decision, and writes timestamped JSON artifacts. The fixture and scoring rules were committed and frozen before the eval code ran.

Bug Fix or Performance Improvement

Two independent capture-layer bugs caused the eval harness to report working model output as malformed. Both bugs lived in one file (agents/semantic_proposer.py). Both produced the same label on the scoreboard — malformed — for completely different reasons. Neither was a model failure.

Bug 1: CLI subprocess captured terminal UI bytes as data.

The local llama path called ollama run via subprocess.run(capture_output=True). The Ollama CLI emits ANSI terminal control sequences (spinner, cursor repositioning, line-erase) that are invisible in a terminal but corrupt captured stdout. The raw artifact contained sequences like \x1b[K (erase to end of line) and \x1b[7D (cursor back 7) embedded inside otherwise valid JSON strings.

Result: 5 out of 6 cases reported as malformed. The one case that parsed cleanly produced a confirmed finding — the model was answering correctly and the pipe was garbling the output.

Bug 2: Parser rejected valid JSON wrapped in markdown fences.

The Anthropic Sonnet path returned valid JSON, but the model wrapped every response in a `json code fence. The parser called json.loads() directly on the raw text, which starts with three backticks, not a curly brace.

Result: 6 out of 6 cases reported as malformed. The raw output contained complete, well-formed proposals for every case. This is not an exotic failure — most LLMs wrap JSON responses in markdown fences by default.

Code

Single commit: 49902b2 — 54 lines changed in one file.

Full diff: git show 49902b2 -- agents/semantic_proposer.py

Change 1 — Replace CLI subprocess with HTTP API:

Before:
`python
subprocess.run(
["ollama", "run", "llama3.2", prompt],
capture_output=True,
text=True,
timeout=60,
)
`

After:
`python
request = urllib.request.Request(
OLLAMA_API_URL,
data=json.dumps({
"model": OLLAMA_MODEL,
"prompt": prompt,
"stream": False,
}).encode("utf-8"),
headers={"content-type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=120) as response:
payload = json.loads(response.read().decode("utf-8"))
`

Explicit error handling for network failures, timeouts, and invalid JSON responses replaced the opaque subprocess failure mode.

Change 2 — Strip markdown code fences before parsing:

`python
def _strip_code_fence(text: str) -> str:
t = text.strip()
if t.startswith("`"):
t = t.split("\n", 1)[1] if "\n" in t else t[3:]
if t.rstrip().endswith("`"):
t = t.rstrip()[:-3]
return t.strip()
`

Applied before json.loads():
`python
parsed = json.loads(_strip_code_fence(text))
`

The frozen fixture, scoring rules, and confirmer gate were not modified. Only the capture layer changed.

My Improvements

Artifact	Engine	Malformed	State
20260701T225629Z	llama3.2	5/6	Before fix — ANSI control bytes
20260709T191930Z	Sonnet	6/6	Before fix — markdown fence
20260709T190750Z	llama3.2	0/6	After fix
20260709T192344Z	Sonnet	0/6	After fix
20260709T202859Z	Both	0/18	v1 full run, 18 cases, clean

All artifacts — including the broken pre-fix runs — are committed in path_a_eval_artifacts/ and publicly recomputable.

What changed after the fix:

• The pre-fix scoreboard said both models failed almost every case. The post-fix v1 run showed Sonnet catching 12/12 positive cases by direction and 7/12 by exact match, with 0/6 false fires. That result was invisible until the capture path was honest.
• Malformed became a diagnosable signal. Each case now records a malformed_reasons string (e.g., "http 400: Your credit balance is too low") instead of just a boolean. Different failure modes — credits, ANSI, fences — no longer collapse into one label.
• Terminal UI was removed from the data path. The transport layer is now HTTP JSON, eliminating an entire class of capture corruption.
• Reproducibility was preserved. The fix touched only capture/parsing code. The frozen fixture, scoring rules, and confirmer gate were not modified to improve results.

Integrity hole the fixed scoreboard still had:

Fixing capture made the transport honest — but it exposed a subtler problem one layer up, in the scoring itself. When a case's output was genuinely unparseable — malformed — the harness still scored it. A malformed positive counted as a miss, and a malformed negative could count as a clean pass as long as no forbidden finding fell out of the garbage. Unparseable output was being read as evidence: evidence of a miss on one side, evidence of a clean negative on the other. It is neither.

Commit e933894 closes that. Malformed cases are now excluded from every positive, negative, catch, pass, false-fire, and ablation total. They do not disappear — each one still carries its raw output, its malformed_reasons, its counts, and its Sentry diagnostics, and the scoreboard renders it explicitly as unscored_malformed. The harness keeps the case for diagnosis and refuses to score it. Well-formed cases run through the exact same scoring path as before.

This did not move the published numbers. The clean v1 run (20260709T202859Z) had zero malformed cases across both engines, so 12/12 by direction, 7/12 exact, and 6/6 negative traps stand unchanged. This is a going-forward integrity fix, not a re-scored result — the old runner would have let a future malformed negative slip through as a clean pass, and now it cannot. Two focused regressions lock it in: a malformed positive and a malformed negative both stay recorded and both stay unscored. Focused: 2 passed. Full suite: 40 passed, 1 expected xfail.

Verify in under 60 seconds:

`bash
git clone https://github.com/keniel13-ui/memory-authority-auditor.git
git show 49902b2 -- agents/semantic_proposer.py

Pre-fix artifacts:

path_a_eval_artifacts/path_a_eval_20260701T225629Z.json (search \x1b for ANSI bytes)

path_a_eval_artifacts/path_a_eval_20260709T191930Z.json (raw_output starts with `json)

`

Why observability became part of the fix

I found these failures because the harness preserved raw output and I was willing to open it. That was enough to reconstruct the incident after the fact. It was not enough to make the next incident fast to diagnose.

Before Sentry, every failure arrived at the same destination: SemanticProposerError, then malformed on the scoreboard. The summary preserved the fact that something broke while discarding the path it took to get there. I still had to replay artifacts by hand to answer basic questions:

• Did the provider reject the request before inference?
• Did the transport corrupt a real response?
• Did the parser reject valid content?
• Did the model return the wrong schema?
• Did a well-formed case reach scoring and fail there?

That is why Sentry is not decoration on top of the bug fix. The capture patches repaired two known failures. The instrumentation repairs my ability to tell the next failures apart.

I do not want another red cell that merely says malformed. I want the trace to show the provider, engine, case, raw-output preview, parser symptom, malformed reason, and final scoring decision in the order they happened. The point is not more telemetry. The point is preserving causality.

Best Use of Sentry

The two bugs described above — ANSI corruption and markdown fences — both threw the same exception class (SemanticProposerError) and both produced the same label on the scoreboard: malformed. The eval harness counted them, but it could not distinguish terminal corruption from a code fence from an HTTP 400 from expired credits. Three completely different failure modes collapsed into one boolean.

Sentry was integrated into the eval harness specifically to prevent that collapse from happening again. The integration uses Error Monitoring, Distributed Tracing, and AI Agent Tracing from the Sentry SDK.

What the integration does:

The sentry-sdk[anthropic] package provides auto-instrumentation for Anthropic API calls. For local Ollama calls, a manual span records model ID, provider, a preview of the input, and the response size — giving both engines the same tracing coverage.

Every eval run creates a root transaction (path_a_eval) with nested spans: one per engine, one per case inside each engine. Each case span records the case ID, class, engine, proposal count, confirmed finding count, and scoring result. The full eval waterfall is one trace.

When the proposer returns output that fails JSON parsing, a breadcrumb fires before the exception handler. The breadcrumb carries the raw output length, a preview of the first 200 characters, and a boolean for whether the output started with a markdown code fence — the exact diagnostic data that would have immediately identified Bug 2 without replaying artifacts manually. When a proposal is missing required fields, a separate breadcrumb fires with the list of present keys vs. expected keys.

If any case throws an exception (network timeout, invalid JSON, transport failure), sentry_sdk.capture_exception() sends the full traceback with the breadcrumb trail attached. At the engine level, if any cases were malformed, sentry_sdk.capture_message() fires a warning with the malformed count.

The audit CLI pipeline (audit_cli.py) wraps the full run in its own transaction, and the run_audit function is decorated with @sentry_sdk.trace for automatic span creation.

Why it matters for these bugs specifically:

Both bugs would have been diagnosable on the first run with Sentry active. Bug 1 (ANSI) would show up as a SemanticProposerError with a breadcrumb whose raw_preview field contained visible \x1b[K sequences — no artifact replay needed. Bug 2 (fence) would show starts_with_fence: true in the breadcrumb data on every Sonnet case, immediately separating it from other parse failures.

The malformed label collapsed three causes into one boolean. Sentry breadcrumbs carry the cause. That is the difference between knowing something broke and knowing what broke.

Integration is zero-cost when disabled. All Sentry calls are no-ops when SENTRY_DSN is not set. When the Sentry work landed (d331801), the existing 38-test suite passed without Sentry configured and no test was modified. The later malformed-scoring fix (e933894) added two focused regression tests for the exclusion behavior, so the current suite is 40 passed / 1 expected xfail.

Files changed:

File	Change
requirements.txt	Added sentry-sdk[anthropic]>=2.0
agents/semantic_proposer.py	Breadcrumbs on JSON parse failure (raw preview + fence detection) and missing proposal fields; manual AI span for Ollama
audit_pipeline.py	@sentry_sdk.trace on run_audit
audit_cli.py	Sentry init + root transaction wrapping audit pipeline
path_a_eval_runner.py	Sentry init + root transaction, per-engine spans, per-case spans with scoring data, breadcrumbs on malformed output, exception capture

Sentry tools used: Error Monitoring (exception capture with breadcrumb context), Distributed Tracing (transaction → engine span → case span hierarchy), AI Agent Tracing (sentry-sdk[anthropic] auto-instrumentation for Anthropic + manual spans for Ollama).

What this changed in how I trust an eval

The patches are small. The trust model changed more than the code did.

An eval result now has to survive five separate questions before I treat it as evidence:

1. Did a model actually run? A credit, authentication, timeout, or provider failure is not model behavior.
2. Did the transport preserve the response? Terminal UI output is not automatically a machine-readable data channel.
3. Did the parser reject content or meaning? Valid JSON inside common wrapping is a parser-contract issue, not a reasoning failure.
4. Did the scorer have interpretable evidence? An unparseable case cannot honestly count as a miss or a clean pass.
5. Can another person reconstruct the path? Raw output, frozen fixtures, commit history, traces, and failed artifacts have to remain available.

This is also why I kept the broken runs. The embarrassing artifact is not debris to clean up before publication. It is the receipt that proves the before-state existed. Without it, the clean run is just a better-looking chart asking to be trusted.

The malformed-score repair pushed that lesson one layer deeper. After fixing transport and parsing, I found that the scorer could still convert “we cannot interpret this output” into a judgment about the model. A malformed positive became a miss. A malformed negative could look clean. The harness was still trying to force uncertainty into a binary result.

Now it refuses. The case stays visible. The raw answer stays visible. The reason stays visible. The trace stays visible. Only the score is withheld.

That is the distinction I want this project to enforce: unknown is a real state, not an inconvenient value to round into pass or fail.

The real lineup I cleared

I started with what looked like two model failures. The actual lineup was longer:

• a provider request that never became inference;
• a terminal channel pretending to be a data API;
• a parser too brittle for common model wrapping;
• a single malformed label collapsing unrelated causes;
• a scorer willing to treat unparseable output as evidence;
• and a summary confident enough to hide all of it behind one number.

Each repair cleared one obstruction without erasing the previous receipt. The HTTP path fixed transport. Fence stripping fixed parsing. Sentry restored causal visibility. The malformed filter stopped the scoreboard from converting uncertainty into a verdict.

The models were never the only thing on trial. The provider path, capture layer, parser, scorer, renderer, and my own willingness to trust a clean table were on trial too.

That is the lesson I am carrying forward: when an AI eval gives me a dramatic result, I do not ask whether the number looks plausible. I ask whether every layer between the model and that number left enough evidence to deserve belief.

The scoreboard is the last witness in the chain. It should never be the first one I trust.

---

Repository: memory-authority-auditor
Bug fix commit: 49902b2 — Fix both proposer capture bugs and record first clean two-engine Path A eval
Sentry integration commit: d331801 — Add Sentry observability to audit and eval pipelines
Malformed-exclusion commit: e933894 — Exclude malformed cases from eval aggregates; malformed cases stay recorded and diagnostic, never scored

Comments

[Mike Czerwinski]

Excluding malformed from the aggregate is the right fix for the failure you found, and the new failure it can quietly create deserves a flag. Malformed becomes a category with its own incentive now: anything that gets dropped from the denominator stops being scored at all. If the hardest, most ambiguous cases are also the ones most likely to trip a parser, malformed stops being noise and starts being a place where difficulty goes to disappear. A 0-false-positive, 12/12 direction-correct number is only as clean as the guarantee that malformed isn't quietly absorbing exactly the cases that would have been wrong.

The three causes you separated, provider outage, transport corruption, parser mismatch, aren't just different diagnoses, they're different repair costs and different owners. A transport corruption is a bug in your harness, fully within your control to fix. A provider emitting ANSI into what's supposed to be JSON is a contract violation on their side, fixable by you working around it but not by you fixing it. Those two probably shouldn't collapse into one "malformed" bucket even in the diagnostic view, because one tells you to fix your code and the other tells you to build a defense against someone else's non-conformance that could reappear in a different shape next release.

The concrete check I'd add: track malformed rate per model per week as its own tracked metric, not just a denominator exclusion. If Ollama's ANSI-corruption rate or Sonnet's fence-wrapping rate starts climbing, that's signal about the provider drifting, not noise to filter out before the real scoring starts.

[Self-Correcting Systems]

This is the sharpest version of the concern and you are right. excluding malformed from the denominator fixes one hole and opens a quieter one. malformed becomes the place difficulty can go to disappear. if the hardest, most ambiguous cases are also the ones most likely to trip a parser, then a clean 12/12 is only clean if malformed is not quietly eating the cases that would have been wrong.

on this run it is not, because that v1 artifact had zero malformed on both engines, so the number is not hiding anything today. but that is luck, not a guarantee, and your point is about the general case, which is exactly right.

so here is what i am taking from this. malformed rate per model per week becomes its own tracked metric, not just a denominator exclusion. if ollama's ansi rate or sonnet's fence rate starts climbing, that is provider drift and it should be loud, not filtered out before scoring starts.

and the second thing you said might be the better catch. the three causes are not just different diagnoses, they are different owners and repair costs. transport corruption is my bug. a provider emitting ansi into json is their contract violation that i can only defend against, not fix. collapsing those even in the diagnostic view hides who has to move. i am splitting them.

genuinely appreciate this. this is the kind of thing that makes the number honest instead of just clean.

[Mike Czerwinski]

The owner-split is the sharper move of the two you're making, because it changes what the metric is for. A malformed-rate number that's just diagnostic tells you something's wrong. A malformed-rate number split by who owns the fix tells you who to page, and those are different products even though they're built from the same count.

Worth deciding now, before the split exists in code: what happens when transport corruption and provider ansi-emission both spike in the same window. If the two counters are independent, that's a coincidence you can shrug off. If they're not, a correlated spike probably means something upstream of both changed, your own transport or a shared dependency neither owner controls, and neither counter alone tells you that story. A join across the split, not just the split itself, is probably worth keeping around.

[Self-Correcting Systems]

yeah the join is the part id have missed, and youre right to force the decision before the split exists in code. the split answers who owns this failure so you page the right person. but a correlated spike across two owners answers a different question, did something upstream of both of them move, my own transport or a shared dep neither owner controls.

and thats the more dangerous question, because the split actively hides it. two counters spiking in the same window, independent, you shrug. two counters spiking because one thing under both of them changed, and the split has you paging the provider and the transport owner at once, both coming back with nothing wrong on their side, while the real cause sits one layer down laughing at both pages.

so the split is the per owner view and the join is the something common changed view, and you keep both or the split lies to you the first time a shared dependency drifts. worth building the correlation on top of the split the same day, not after it burns me once.

[Mike Czerwinski]

Both owners getting paged and both coming back clean is the worst version of the failure, because it doesn't just waste two people's time, it actively produces evidence against the correlation you should be looking for. Two clean investigations reads like nothing is wrong here, when the actual finding is neither of these two things is wrong, which is itself informative, and that reframe requires someone to notice the join was the point, after the fact, exactly the kind of thing that gets skipped under incident pressure.

Cheapest version of the join that still catches this: don't build correlation detection, just log both counters' spike timestamps to the same timeline and eyeball it during the postmortem, cross-owner correlation review as a standing item, not a live alert. That gets you the these two things moved together insight without building a live system before you've seen a real instance of the failure it's meant to catch.

[Self-Correcting Systems]

the both-clean case being WORSE because it manufactures evidence against the correlation is the part i hadnt seen. two clean investigations reads as nothing wrong here, when the actual finding is neither of these two is wrong, which is the informative thing, and it only gets read that way if someone remembers the join was the point after the fact, exactly what gets skipped under incident pressure.

and the cheap version is the right call, dont build correlation detection before youve seen the failure once. log both counters spike timestamps to one timeline and make cross-owner correlation a standing postmortem item, not a live alert. you get the these moved together insight without shipping a live system to catch a failure you havent met yet. the join is a review discipline first and a detector only after a real instance earns it.

[Mike Czerwinski]

Review discipline first, detector only after a real instance earns it is the right ordering, and it's a cheap insurance policy against building a live correlation system that never fires because the failure it's watching for hasn't happened yet in your specific stack. The postmortem timeline is basically free to maintain and it's the artifact that tells you whether the detector would ever be worth building at all.

[Self-Correcting Systems]

agreed, and the ordering matters more than it sounds. a live correlation detector
built before a real instance exists is a thing that never fires and then gets
trusted for never firing. thats the same silence problem one layer up.

the postmortem timeline is basically free and its the artifact that tells you
whether the detector is worth building at all. if six months of timelines never
show the both clean case, i saved myself a system. if they show it twice, i now
have the two examples to build the detector against instead of guessing at the
shape.

[Synfinity Dynamics Pvt Ltd]

Really insightful article. The scoreboard should never hide transport, parsing, and scoring failures behind one label.

[Self-Correcting Systems]

Exactly. collapsing those layers did more than make debugging slower it erased
causality. a transport failure, valid JSON rejected by the parser, and genuinely
unparseable output all demanded different repairs, but the scoreboard reduced them
to the same word: malformed.

once that happens, the dashboard can turn “we do not know” into “the model failed”
or worse, into a clean pass. preserving the raw output and separating transport,
parsing, and scoring state is what made the scoreboard evidence again.

appreciate you reading the mechanism closely.