DEV Community: AsiaOstrich

The Biggest Flaw in My AI Evaluation Wasn't the Models. It Was My Scorecard.

AsiaOstrich — Sat, 25 Jul 2026 11:05:32 +0000

I recently ran a small evaluation to compare three AI coding assistants. The task sounded straightforward: give each model the same engineering artifact, ask it to review the work, then score the results against a checklist I had prepared beforehand.

I expected to learn which model was the better reviewer. Instead, I learned something much more uncomfortable: my evaluation couldn't reward the behavior I actually wanted.

The scorecard had a blind spot

The checklist covered every category of issue I expected a competent reviewer to raise. Each finding earned points if it matched the checklist. That looked objective — until I realized what happened to findings that weren't on the list.

They automatically became false positives.

In other words, my evaluation couldn't possibly reward a model for discovering something I had missed. It only rewarded agreement with my own thinking.

I wasn't measuring "who reviews better." I was measuring "who thinks more like me." Those are not the same question.

The fix wasn't a better checklist

At first I thought I simply needed a more complete checklist. That wasn't enough. No matter how much I expanded it, there would always be things I hadn't anticipated.

So I separated the results into two categories: expected findings, and new findings outside the checklist.

The important part wasn't creating the second column. It was refusing to judge it myself. If the entire purpose of that column is to capture things I failed to notice, I'm the least qualified person to decide whether those findings are valuable. So someone else reviewed those discoveries independently.

That changed the evaluation completely.

The result surprised me

The new category produced two genuine issues that weren't on my original checklist. Both were things I had overlooked.

Even more interesting, each one came from a different model — and neither model surfaced the other's finding. If I had picked a single "winner," one of those two issues would never have shown up.

That made me rethink the question itself. Instead of asking "Which reviewer is the best?" I started asking "Which reviewers complement each other?" Those are very different evaluation goals.

Role mattered more than model

Another observation was just as interesting. When the task was "follow explicit instructions and revise the work," the models performed similarly. When the task became "find something nobody explicitly asked you to find," the differences became much larger.

The ranking wasn't universal. It depended on the role.

That sounds obvious in hindsight, but many comparisons quietly assume a model has a single global ranking across every engineering task. My experiment suggested otherwise. Choosing an editor and choosing a reviewer may be two entirely different optimization problems.

What changed for me

I still use checklists. They're valuable because they make evaluations repeatable. But now I treat them as only half of the measurement. The other half is explicitly looking for discoveries that the checklist couldn't have predicted.

Otherwise, an evaluation becomes surprisingly good at rewarding agreement — and surprisingly bad at finding the thing you actually wanted someone to notice.

Disclosure: I used AI assistance to improve the English wording of this article. The engineering experience, analysis, and conclusions are my own.

A token-saving repo went viral. We read the source, not the README.

AsiaOstrich — Wed, 08 Jul 2026 12:00:11 +0000

Our daily scout surfaces AI-tooling repos every morning. Most of them you should not adopt — and the engineering is in deciding which parts, if any, are actually load-bearing for your problem. Here's one worked example.

A context-compression project — Headroom (headroomlabs-ai/headroom, Apache-2.0) — made the rounds with a great pitch: cut 60–95% of your LLM context tokens with "zero degradation." For an agent that streams huge tool logs into a model, that's the dream. The reflex it triggers in most teams: vendor the whole thing.

We did the slower thing first. We opened the source.

The README names and the real names disagree

Headroom's README sells three components — CodeCompressor, SmartCrusher, CacheAligner. Open src/ and the actual classes are CodeStructureHandler, JSONStructureHandler, and AnthropicCacheOptimizer + PrefixCacheTracker. Not a scandal — just a reminder that the README is marketing copy and the source is the spec. The real design is a clean three-stage pipeline: detect the content type (Magika), run a type-specific handler to extract a structural skeleton, and only then lossy-compress whatever's left.

The parts worth understanding:

Code (handlers/code_handler.py): tree-sitter parses the file (regex fallback) and keeps imports, signatures, and class declarations while dropping function bodies, comments, and whitespace. "Keep the API surface, drop the implementation." Works for Python/JS/TS/Go/Rust/Java/Perl.
JSON (handlers/json_handler.py): a per-token heuristic — keep keys, syntax, booleans, short values, and high-entropy strings (entropy ≥ 0.85); compress long strings; keep the first N array items in full. No schema, no dedup.
Cache alignment (prefix_tracker.py): one genuinely sharp invariant — freeze the already-cached prefix, only compress new content, so your compression never rewrites the prefix and silently invalidates the provider's KV cache.

The benchmark, read adversarially

This is where reading beats retweeting. The headline is "60–95% savings, zero degradation," measured with the standard lm-eval harness on a real model. Credit where due: the repo even ships its own worst-case and adversarial benchmarks and lists its failure modes — an honest signal most projects skip.

But line up the claim with what was actually measured:

The zero-degradation numbers come from GSM8K / TruthfulQA / SQuAD — short-answer tasks with almost no context to compress. "Zero degradation" there is trivially true; it can't support the 60–95% claim.
The runs that do compress heavily — agent traces — are n=5, scored by loose keyword matching.
It's all self-reported, with no third-party reproduction. One external review of prompt-heavy chat measured 20–40% savings, not 60–95%.

None of that makes Headroom bad. It makes the headline number a claim, not a result — and the difference matters when you're deciding what to build on.

What we borrowed — and what we left on the shelf

We took three ideas, each re-implemented to fit our own stack, not the package:

tree-sitter "keep signatures, drop bodies" for code blobs — a real semantic upgrade over blunt truncation.
The JSON keep-keys / high-entropy heuristic — cheap, deterministic, no model required.
Cache-aware compression — the "will this rewrite a cached prefix?" check, which is the highest-ROI idea in the whole repo and the one nobody talks about.

We explicitly declined:

The reversible-retrieve loop (CCR) — it carries an extra store plus a retrieve round-trip to recover what compression dropped, costs hundreds of MB of RAM, and is already known to break under streaming (issue #1450). If your compression is allowed to be lossy, you don't need it.
The ML text model (Kompress) — small, English-biased, and when it's unavailable the whole text path silently falls back to first-2/3 + last-1/3 truncation. A dependency that decays into the thing it replaced.
The proxy / MCP server / cross-agent "learn loop" — orthogonal feature creep, not a compression upgrade.

The actual lesson

A viral repo is an input to a decision, not the decision. Read the source to find the two or three load-bearing ideas, check the benchmark against what it actually measured, and write down — on the record — what you didn't take and why. That discipline (read the code, not the README; borrow only what nets out positive) is exactly what we encode as a standard.

AI disclosure: I wrote this with AI assistance for English phrasing and structure. The source reading, the benchmark analysis, and the adopt/decline decisions are my own — the AI did not supply the technical judgement.

Picking a code-graph memory for your AI agent? Star count is the wrong axis.

AsiaOstrich — Mon, 06 Jul 2026 12:00:11 +0000

"Code memory for coding agents" is a hot lane right now. A 49k-star code-graph tool shows up, FalkorDB is pushing GraphRAG, and the reflex is the same one we apply to everything popular: lots of stars — let's wire it in. We ran one through our radar instead. Here's the part worth keeping: not the verdict, but the axis we used to reach it.

The lane has two very different shapes

Most "memory for your agent" is a vector database: chunk the repo, embed it, retrieve by similarity. That is genuinely good at one thing — finding code that looks like other code. Paste a function, get back functions shaped like it.

But the question I actually ask every day isn't "what looks similar to this." It's:

"If I change this function, what breaks?"

Similarity can't answer that, and not by accident. The callers of a function look nothing like the function. A logging wrapper, a retry decorator, a route handler three layers up — none of them are textually similar to the body you're editing. They're connected by structure, not by surface. Answering "what breaks" is a graph traversal — walk the call edges — not a nearest-neighbour lookup in embedding space.

That's the real divide in this lane: similarity retrieval vs structural (graph) retrieval. Almost everything in the hype cycle is the former. The thing you reach for during a risky refactor is the latter.

What we did with a 49k-star tool

CodeGraph (MIT, ~49k stars, 20+ languages, auto-syncing local index, MCP server for Cursor/Claude Code/Codex) is a serious piece of work on the structural side. We evaluated it properly — read the repo, mapped its capabilities against ours.

And we deliberately did not adopt it.

Not out of not-invented-here. Our own engine, EngramGraph, already covers the core: a tree-sitter code graph where callers <fn> is a graph walk, plus a deterministic, zero-LLM index (no embeddings to drift, cheap to rebuild, same answer every run). The decision to build rather than wire in an external daemon was already made; adopting CodeGraph wholesale would have added a dependency and an architecture conflict, not a capability.

So we filed it as Assess, and wrote down the three things it's genuinely useful for — without adopting it:

A language-coverage blueprint. CodeGraph parses 20+ languages and recognises 17+ framework routing patterns. We don't import its code; we use its parser list as a reference map for where to extend ours next.
A benchmark method. Its 7-codebase, four-axis measurement (cost / tool-calls / tokens / latency) is a reusable yardstick template for proving a code graph actually reduces agent exploration — not a set of numbers to copy. Their absolute figures are theirs; the method travels, the numbers don't.
An interop signal. It exposes its graph over MCP to several agents. That's market evidence that "cross-agent code-graph over MCP" is wanted — worth watching, not worth coupling to.

The point isn't the verdict

A "no" with reasons is more useful six months later than a "yes" you can't reconstruct. The lesson that outlives this particular tool:

Tool selection in a hyped lane is not a star-count comparison. It's one question — does this solve the thing I actually ask every day? For code memory, that question splits cleanly along similarity-vs-structure. Once you know which side of that line your real problem lives on, half the shiny options on the radar quietly stop being relevant.

AI disclosure: I wrote this with AI assistance for English phrasing and structure. The repo reading and the adopt/decline decision are my own — the AI did not supply the technical judgement.

Most AI-generated tests are written once and never questioned. That's the bug.

AsiaOstrich — Thu, 02 Jul 2026 12:00:25 +0000

Our daily scout surfaces AI-tooling papers every morning. Most of them don't change how we build. This one made us look hard at a habit we'd stopped noticing — so here's the worked example.

In January 2026, "The Rise of Agentic Testing" (arXiv:2601.02454) made the case that the way we generate tests with LLMs is quietly broken. Not the models — the shape of the loop.

The default loop is single-shot

Ask an LLM for tests and you usually get them in one pass: generate once, done. The problem is that a one-shot test is written blind — the model never sees the test actually run. So you get the three failure modes everyone has hit: tests that don't compile, tests that assert nothing meaningful, tests that duplicate each other. There's no execution-aware feedback closing the loop, so a test that passes for the wrong reason sails straight through.

The paper's proposal: make test generation a closed loop — generate, run in a sandbox, analyze the failure, then refine the test itself, and iterate until it converges. Standard agentic-loop thinking, pointed at the test instead of the code.

code-centric vs test-centric — the part worth internalizing

Here's the distinction that's actually load-bearing.

When a test fails, most pipelines (ours included, historically) are code-centric: the failure is a signal to change the code until the test goes green. The test is treated as ground truth.

But the test isn't always right. Sometimes the failing test is the one telling the truth and the "fix" is to weaken an assertion until it stops complaining. A test-centric loop asks a different question first: is this failure because the code is wrong, or because the test is imprecise / asserting the wrong thing? If it's the test, you fix the test — make it a sharper check — instead of bending the code to satisfy a bad check.

Why this matters beyond semantics: a test that passes for the wrong reason is worse than no test. It's a false green — it merges, it sits in CI looking like coverage, and it gives everyone permission to stop paying attention to exactly the thing it was supposed to guard. That's not coverage. That's debt wearing a coverage badge.

The honest caveat (this is the discipline, not a footnote)

We could only get the paper's abstract. No full text on arXiv, no HTML, and no public repo — so there was no implementation to read and no methodology to verify. The paper also headlines numbers like "−60% invalid tests / +30% coverage."

We are explicitly not adopting those numbers. They're a single-source claim, measured on one microservice app, unreproduced, and from a domain that isn't ours. Porting a benchmark figure you can't reproduce into your own targets is how you manufacture a result and call it evidence. Every threshold we'd ever set has to be a relative comparison against our own baseline — not "the paper got X, so we should too."

So what did survive the filter? One idea: when a test fails, first decide whether the test or the code is wrong — and if it's the test, refine the test. That's it. Not the multi-agent topology, not the numbers. The one load-bearing idea.

Why this lands for us

A test that's allowed to refine itself into a stronger check is just the natural extension of an anti-fake test gate — the kind of gate that flags a test which asserts nothing and quietly passes. Detection is step one; turning a weak test into a real check is step two. We treat a passing test as a check that has to keep being true, not a one-time green tick.

Half of staying current is reading the papers. The other half is the discipline to take only the one idea that's actually load-bearing for your problem — and to refuse the numbers you can't reproduce.

AI disclosure: I wrote this with AI assistance for English phrasing and structure. The paper reading and the judgement about which idea was load-bearing are my own — the AI did not supply the technical judgement.

Our graph database was abandoned upstream — here's the 6-line migration (EngramGraph 0.3.0)

AsiaOstrich — Wed, 24 Jun 2026 12:00:26 +0000

In October 2025, Kùzu Inc. archived KuzuDB — the embedded graph database — with a one-line note: "Kuzu is working on something new." The npm package was deprecated, and its transitive dependencies (tar@6.2.1, npmlog, gauge) carried 5 high-severity vulnerabilities with no fix coming.

EngramGraph runs on an embedded graph database. That database was Kuzu.

EngramGraph 0.3.0 is the migration release — and the migration turned out to be far cheaper than expected. Here's why, and what we learned.

The fix: ryugraph

The community responded to KuzuDB's abandonment with several forks. We evaluated them and picked ryugraph (Predictable Labs):

MIT licensed, actively maintained, published on npm
Designed as a drop-in Kuzu replacement — and in our experience, it actually is
Same storage format lineage: existing .engram/graph.db files keep working

The core API — prepare(), execute(), query(), getAll() — is signature-identical to kuzu. Our entire test suite (69 tests) passed on the first run after the swap.

Why the migration was 6 lines

One architectural decision from day one paid for itself here: every raw database call goes through a single thin wrapper (GraphConnection, ~80 lines). The other 20+ call sites in the codebase only ever see the wrapper.

So the migration was:

package.json: kuzu → ryugraph
connection.ts: one import line
Two type-only imports (KuzuValue → RyuValue)

That's it. If your project wraps its native dependencies behind one interface, an upstream abandonment becomes an afternoon, not a quarter.

The leftover CVEs (and the override trick)

Swapping kuzu killed the deprecated-toolchain CVEs, but ryugraph itself pins cmake-js@^7.3.0, whose tar@6.2.1 carries known path-traversal CVEs. The fix exists upstream (cmake-js@8 uses a patched tar) — ryugraph just hasn't bumped yet.

npm overrides to the rescue:

"overrides": { "cmake-js": "^8.0.0" }

npm audit: 5 high → 0.

One caveat worth knowing: npm overrides don't propagate to downstream consumers. If you depend on engramgraph (or anything that depends on ryugraph), add the same override to your own package.json until ryugraph bumps cmake-js upstream.

AI disclosure: I wrote this with AI assistance for English phrasing and structure. The migration, the CVE work, and the wrapper decision that made it six lines are my own — the AI did not supply the technical judgement.

Seven kinds of spec ↔ test ↔ code ↔ doc drift — and why naming them makes them checkable

AsiaOstrich — Wed, 24 Jun 2026 01:10:29 +0000

Spec-driven development has a quiet failure mode: your specs, tests, .feature files, source, and user docs all start aligned — and then drift apart. A test loses its spec. An acceptance criterion never gets a test. A spec is marked done but was never implemented. Nobody notices until it surfaces in review or production.

The problem with "are the artifacts still consistent?" is that it isn't one question. It's several, and each has a different mechanical answer. Naming them separately is what makes the check automatable instead of a review-time judgement call.

Seven kinds of cross-artifact drift

Orphan test — a test with no spec behind it. Either the spec was deleted, or the test is guarding something nobody agreed to.
Uncovered AC — an acceptance criterion with no test. The requirement exists on paper and nowhere else.
not_implemented — a spec marked done but absent in code. The dangerous one, because the status field asserts the opposite of the truth.
Cross-spec conflict — two specs that contradict each other. Whichever one you read last wins.
Orphan .feature — a feature file with no owning spec. BDD scenarios that outlived their requirement.
AC-without-scenario — an acceptance criterion that no BDD scenario exercises. Possibly covered by unit tests, but never described as behaviour.
user-guide ↔ E2E drift — docs and end-to-end tests out of sync. Two artifacts that both claim to describe what the user sees, disagreeing.

Why the enumeration is the point

Each of the seven is a link that should resolve and doesn't: spec→test, AC→test, spec→code, spec→spec, feature→spec, AC→scenario, doc→E2E. Once you state them that way, checking is a graph walk and a report of the edges that dangle — not a reading exercise.

That difference matters because a consistency check performed by reading only happens when someone remembers to read. A check expressed as dangling edges can run on every commit, and the version that runs on every commit is the only one that stays true.

The list is not exhaustive. It's the set that is cheap enough to check mechanically, which is exactly what makes it worth automating rather than intending.

AI disclosure: I wrote this with AI assistance for English phrasing and structure. The drift taxonomy and the decision about which links are worth checking are my own — the AI did not supply the technical judgement.

A hyped memory paper dropped. We didn't adopt it — and that was the point.

AsiaOstrich — Mon, 15 Jun 2026 12:00:25 +0000

Our daily scout surfaces AI-tooling papers every morning. Most weeks the disciplined answer to a shiny new architecture is "no" — and why you say no is where the engineering judgment lives. Here's one worked example.

In May 2026, SAGE: A Self-Evolving Agentic Graph-Memory Engine (arXiv:2605.12061) made the rounds. The pitch is genuinely good: turn graph memory from a passive retrieval middleware into a dynamic long-term substrate, via two coupled components — a Memory Writer that incrementally builds the graph from interaction history, and a Memory Reader that retrieves and feeds back to the writer. That reader→writer feedback loop is the self-evolution engine. The paper reports strong multi-hop QA results, best after two self-evolution rounds.

The reflex that paper triggers in most teams: new SOTA memory architecture — let's rebuild on it.

We did the opposite. We opened our own design doc first.

We already had it

EngramGraph (our open-source code + knowledge-graph memory engine) already specifies a SAGE Evolution Loop:

a Writer that updates node confidence in the graph from feedback events — test pass/fail, human corrections, spec-status changes;
a Reader that returns high-confidence nodes first on query, and feeds that signal back to the Writer.

Line that up against the paper:

SAGE paper	EngramGraph (existing design)
Memory Writer, incremental build	Writer: feedback event → update node confidence / edges
Memory Reader, retrieve + feed back	Reader: high-confidence-first → feeds back to Writer
Downstream feedback drives evolution	Signals: test pass/fail, human fix, spec-status change

Architecturally isomorphic. Adopting the paper's architecture wholesale would have added roughly nothing — except a rewrite, a migration, and the risk that comes with both.

So we borrowed the rigor, not the architecture

A paper that matches your design is still valuable — just not as a blueprint. It's valuable as backing and method. We extracted exactly the three things our design was actually missing:

Formalization — academic grounding for why a reader-writer feedback loop beats static-graph retrieval. Useful when you have to defend a design, not just ship it.
A validation method — the paper's benchmark thinking inspired a code-graph memory-quality yardstick, which moves our loop from "designed" to "measurably verified." This is the real prize: a way to prove the thing works, not just assert it.
Iteration-rounds design — the paper's "best after two self-evolution rounds" gave us a convergence-rounds definition our loop simply didn't have.

The honest caveat (this is the part that matters)

Here's the line we drew, in writing, before borrowing anything:

The paper's benchmarks are multi-hop QA, Natural Questions, LongMemEval, HaluMem — none of which is a code-graph scenario. The absolute numbers do not transfer.

So we explicitly refuse to port the paper's numbers into our domain. Every threshold in our borrow record is a relative comparison — confidence-weighted reader vs. our own static-MERGE baseline — not "the paper got 91.6, so we should too." Cargo-culting a benchmark from dialogue memory into a code graph is exactly how you manufacture fiction and call it a result.

We also wrote down what we deliberately did not take: the paper's agent-dialogue-memory scenario doesn't fit a code graph, so its scenario-specific mechanics stay on the shelf.

Why log a "no" at all

Because a tech radar isn't a list of things you adopted. It's a record of judgment. We filed SAGE as Trial, scoped to relative validation, with the non-adoptions named. Six months from now, that record tells us — and anyone reading — not just what we believed, but how carefully we believed it.

Half of staying current is reading everything. The other half is the discipline to take only the part that's actually load-bearing for your problem.

AI disclosure: I wrote this with AI assistance for English phrasing and structure. The paper reading and the decision not to adopt are my own — the AI did not supply the technical judgement.