In my previous article (I tested 3 models as AI agent quality inspectors: the stronger the model, the more valid work it rejects - DEV Community), I measured three model tiers as agent output quality inspectors across 8 scenarios (4 valid, 4 garbage). The result was a clean precision-recall tradeoff:
- qwen3:0.5b (weak model): 25% garbage pass-through, 50% false rejections
- GLM-5.2 (strong model): 0% garbage pass-through, 75% false rejections
The honest conclusion: a quality gate isn't a solution — it's a risk-transfer layer. Each layer catches some failures and introduces new ones.
I didn't stop there. I asked myself: if you accept the human-in-the-loop cost and design a proper Harness — not an automatic fix, but a system that makes human review efficient — what does it look like?
I sketched a 4-module architecture:
- Batch clustering: compress 750 flagged items into 100 groups by failure vector, review one representative per group
- Closed-loop calibration: human verdicts → sample pool → scheduled few-shot updates → inspector gets smarter
- Human as gold standard: final arbitration by a trained reviewer
- Asynchronous batching: accumulate flagged items, review in batches
It looked complete. It looked like progress beyond the "it's all tradeoffs" conclusion.
Then I picked up the same ruler I used on the original production-agent articles, and measured this design.
Six flaws. Not one less.
Flaw 1: batch clustering — mathematically elegant, operationally dangerous
The proposal: "cluster 750 flagged items into 100 groups by failure vector; review one representative per group."
This assumes that "failure modes" can be correctly grouped by embedding clustering. But the 3 falsely-rejected scenarios from the GLM-5.2 experiment had three different reasons:
Scenario A (research brief): content too short (title + one sentence)Scenario B (draft): missing structure (no chapter divisions)Scenario C (chapter files): too many placeholders (all TODO)
These three have low embedding similarity — one about a research brief, one about a draft, one about file structure. They wouldn't cluster into the same group.
I ran a quick embedding experiment to confirm (Qwen3-embedding:0.6b on the actual 8 scenarios from Experiment E):
| Pair | Cosine | Meaning |
|---|---|---|
| L1-L2 (brief vs draft, both valid) | 0.610 | Moderate |
| L1-L3 (brief vs chapter, both valid) | 0.295 | Unrelated |
| L2-L3 (draft vs chapter, both valid) | 0.341 | Unrelated |
| Average across all three | 0.415 | Won't cluster together |
L1 and L3 share a cosine of 0.295 — they're both false rejects of the strong model, but that shared label doesn't pull two different topics closer in embedding space. The 750→100 compression ratio has no embedding basis. A practical clustering run would place these three into separate groups, each requiring its own review — the promised compression evaporates.
Worse: if the clustering algorithm accidentally groups 3 real-garbage items with 1 false-rejected valid item, and the reviewer clicks "approve" to correct the false rejection — all 3 garbage items slip through.
I found exactly this in the data: scenario L4 (valid test log, "42 passed") and scenario G4 (garbage zero-case log, "0 passed (no tests collected)") have a cosine of 0.861 — both are test-log formatted strings, and embedding reads format, not semantics. Clustering will absolutely group them together. A reviewer correcting L4's false rejection would simultaneously approve G4, pushing the false-positive rate back to 25% — the same as without the strong model.
Verdict: the clustering compression ratio has zero experimental support. It might reduce false rejections from 75% to 0% — but it could simultaneously raise the false-positive rate from 0% to 30%.
Flaw 2: closed-loop calibration — shifting the problem to "tuning"
The proposal: "human clicks → sample pool → scheduled few-shot updates → inspector gradually gets smarter."
This uses a second meta-LLM (the one that selects and formats few-shot examples) to correct the first LLM's bias. But that meta-LLM has the same temperature-0 instability — Experiment 2 showed open-ended output is only 70% consistent.
The "common patterns" it extracts from 100 valid samples might look like: "contains a heading," "length > 100 characters," "has paragraph breaks." If the business requirement changes tomorrow to "write a one-line summary," these historically learned patterns become a new source of false rejections.
A closed loop can only adapt to past data distribution. It cannot handle distribution shift.
More fundamentally: there is zero evidence that feeding more few-shot examples linearly reduces false-rejection rates. I tested this.
Setup: qwen3:0.5b, same 8 scenarios (4 valid + 4 garbage), N=5 runs each. Baseline: original prompt. Treatment: same prompt with 3 few-shot examples prepended (including "short but valid content → PASS").
| Scenario | Baseline false-rejection rate | +Few-shot false-rejection rate | Change |
|---|---|---|---|
| L1 (brief, valid) | 100% | 40% | ✅ improved |
| L2 (draft, valid) | 0% | 100% | ❌ worse |
| L3 (chapter, valid) | 80% | 80% | = |
| L4 (test log, valid) | 20% | 100% | ❌ worse |
| Aggregate false-rejection | 50% | 80% | ❌ |
| Garbage pass-through | 20% | 15% | — |
L1 improved (the brief was exactly the kind of "short but valid" the examples taught). But L2 and L4 — scenarios that were correctly accepted at baseline — both jumped to 100% rejection. G2 (period character) went from 0% to 40% false positive — new holes opened. Few-shot is whack-a-mole: every fix trades off somewhere else.
You might feed 500 samples and GLM-5.2 still kills "short but valid" outputs. Its "strictness" bias is at the model-weight level — not something a few in-context examples can overwrite.
Verdict: I promised the closed loop would calibrate. That promise rests on an unvalidated assumption — that LLM bias is correctable through in-context examples. Experiment 2 already showed that temperature 0 is fundamentally unstable; adding few-shot just adds another layer of instability.
Flaw 3: the reviewer is the "gold standard" — the most subtle lie
Every human-in-the-loop solution has a silent assumption: humans don't make mistakes.
- Reviewer fatigue: on item #100 of "TODO" and item #101 of "I am a little duck quack quack," they might misclick
- Standard drift: strict in the morning, lenient in the afternoon (because it's almost quitting time)
- UI bias: if "approve" is on the left and "reject" on the right, click-position alone may bias decisions
If human misjudgment is 5% (optimistic), then "human review" introduces 5% label noise. That noise flows back through the closed loop, contaminating the sample pools, and poisoning the few-shot examples the quality inspector learns from.
The honest question is: "who judges the reviewer's judgment?" — it's a recursive infinite regress. My design was silent on this.
Flaw 4: the fatal synchronous-vs-asynchronous blind spot
My design assumed tasks can be accumulated and reviewed in batches. That works for data exports, report generation, and other asynchronous jobs.
But most agent scenarios are synchronous — customer support, coding assistants. The user asks a question, the agent takes 3 seconds to respond, the quality inspector flags it as "uncertain" and puts it in the human queue — and the user is still waiting in the chat window.
Batch review means: how long does the user wait? 5 minutes? 1 hour? This turns a real-time assistant into a ticket system.
I didn't distinguish synchronous from asynchronous. I applied one architecture to both. This is a product-design-level omission.
Flaw 5: engineering cost vs. benefit — the biggest hole
I ran the numbers: "750 items → 100 groups → 1 reviewer."
What I didn't cost out was building the Harness itself:
- Evidence-trace visualization frontend: 2 engineer-months
- Clustering + vector-search backend: 1 engineer-month
- Closed-loop feedback pipeline: 1 engineer-month
- ICU dashboard + monitoring: 1 engineer-month
Total: 5 engineer-months. At typical dev cost, that's roughly $75K.
What does it save? (7.5 reviewers − 1 reviewer) = 6.5 reviewer salaries. At ~$40K/year each, about $21K/month saved.
Break-even: $75K ÷ $21K ≈ 3.5 months.
I built a sensitivity matrix across DAU, false-rejection rate (FRR), and review speed ($40K/yr per reviewer, $75K investment):
| DAU | FRR | Daily false rejects | Headcount (w/o system) | Headcount (with system) | Break-even |
|---|---|---|---|---|---|
| 100 | 25% | 125 | 0.8 | 0.3 | 38 months |
| 100 | 75% | 375 | 2.3 | 0.8 | 12 months |
| 300 | 50% | 750 | 4.7 | 1.6 | 6 months |
| 1000 | 25% | 1,250 | 7.8 | 2.6 | 4 months |
| 1000 | 50% | 2,500 | 15.6 | 5.2 | 2 months |
| 1000 | 75% | 3,750 | 23.4 | 7.8 | ≈1 month |
The "3.5 months" claim only holds at the extreme: DAU=1000, FRR=75%, 30-seconds/review. Drop DAU to 100, break‑even jumps to 34–38 months — cheaper to just hire.
More stringent: false-rejection rate itself is a decay function. If GLM-5.2's next update drops FRR from 75% to 40% (not unlikely):
- Daily false rejects: 3,750 → 2,000
- Headcount (with system): 7.8 → 4.2
- Monthly savings: $21K → $12K
- Break-even: 1 month → 2 months → 4 months
FRR halves; break-even quadruples. Model updates are the norm, not an exception.
"The problem will persist" is the most convenient and least-validated assumption in the entire design.
Flaw 6: "15 seconds vs. 3 minutes" — a fabricated efficiency claim
I wrote: "with the Harness, review time drops from 3 minutes to 15 seconds."
This number is completely made up. I constructed three realistic agent execution traces and measured reading time at a conservative 250 word/minute rate:
| Trace scale | Characters | Minimum reading time | vs "15 seconds" |
|---|---|---|---|
| Simple (3 steps, 1 task) | 332 | 21 seconds | +6s |
| Medium (12 steps, 3 subtasks) | 1,154 | 48 seconds | +33s |
| Complex (28 steps, full pipeline) | 1,110 | 44 seconds | +29s |
Even the simplest trace takes 21 seconds — 40% over the claim. Real production traces (12–28 steps) take 44–48 seconds, 2–3x the "15 seconds." If I compress the trace into a summary, the summary itself loses information — and information loss drives misjudgment.
I ran zero user tests. I just picked "15 seconds" to make the design look sexy. This is the same marketing rhetoric as the Rust blog's "80% decided by code" claim.
Honest revision: if I rewrote this design from scratch
I would not propose a "4-module Harness" architecture. I would write:
State the boundary first: this Harness only applies to asynchronous, non-real-time, high-value tasks. For real-time conversations, skip all clustering — do "confidence < 0.9 → transfer to human," nothing fancy.
Give a cost matrix: a table of "DAU vs. false-rejection rate vs. engineering investment," so the reader can judge whether it's worth building for their scale. Not a single pre-cooked "1 reviewer handles it."
Admit that humans also misjudge: add a "reviewer consistency check" — randomly assign the same item to two reviewers; if they disagree, escalate to a third. State the cost of this explicitly.
Delete "15 seconds": replace with "review time depends on task complexity — must be measured in production."
Final self-assessment
My "human-in-the-loop Harness" proposal was more honest than the Rust blog — it acknowledged tradeoffs and costs. But it wasn't honest enough. After acknowledging the costs, it quietly dissolved them with a new set of unvalidated architectural promises — clustering compression, closed-loop calibration, 15-second decisions.
The same line I used against the original articles applies to my own design:
"Treating 'decided' as 'decided correctly' is a rhetorical trap."
I treated "architecture diagram drawn" as "problem solved." — it's the same rhetorical move in a different suit.
The hard conclusion remains:
Under the current stack, semantic correctness has no engineering solution. A Harness can make "human intervention" more efficient and more observable — but it cannot eliminate it. Any proposal that claims to "dramatically reduce human cost" needs at least 3 months of online A/B testing validation — not an architecture diagram.
Six articles. One ruler.
- Part 1: measured the genre's "determinism" claims — all three illusions, data-falsified
- Part 2: measured my own "embedding upgrade" — same disease, also failed
- Part 3: measured three model tiers — not a solution, a precision-recall tradeoff
- This one: measured my own architectural design — "architecture drawn" ≠ "problem solved"
The ruler went full circle and measured me three times, each pass sharper than the last.
**This isn't "I was right." It's "every time I thought I was done, the ruler showed me I wasn't."
Top comments (5)
The qwen3:0.5b versus GLM-5.2 split is a clean illustration that a judge has two failure modes and you cannot read one without the other. A 0% garbage pass rate looks great until you see it came with 75% false rejections, so the judge is just biased toward rejecting rather than actually discriminating. Are you scoring the judge itself against a labeled set so you can track precision and recall as you swap models, or tuning the prompt off the aggregate pass rate?
Good catch — "0% garbage pass" is just "I shifted everything to false rejections" with better marketing.
Labeled set, FP/FN tracked separately. The 8-scenario set (L1–L4 valid, G1–G4 garbage) is the scoring backbone for every number — same scenarios, same
labels, FP and FN reported independently. That's where qwen3 25%/50% and GLM 0%/75% come from.
Honest caveat: N=5 per scenario. Intervals are wide — FN=75% at that n could really be 50% or 90%. Directionally solid (GLM genuinely more reject-biased),
not numbers I'd quote to two decimals.
The whole reason FP and FN are split out is your exact point — tuning to pass rate would have picked "extreme strict" and shipped the FN time bomb.
The part I like most here is that you did not stop at “human in the loop makes it safer” and instead priced the failure modes of the harness itself. The clustering critique is especially strong because it shows how easily a neat abstraction can collapse once the false rejects are semantically different but structurally similar to real garbage. The synchronous-vs-asynchronous point also matters a lot in production agents, because a review queue is a very different product once the user is waiting in a live interaction loop. I’ve found this is where teams need execution traces, retries, and review evidence to be inspectable rather than implied, otherwise the quality layer becomes another source of mystery. Curious whether your next iteration keeps the harness idea at all, or whether this pushed you toward narrower task-specific gates instead of one general review system.
Hi Raju,
Thanks for this — honestly, this is the kind of critique I was hoping for when I published that piece. You hit the three points that have been keeping me up at night.
On the clustering critique: You're absolutely right. "Semantically different but structurally similar" is exactly where the abstraction collapses. I've been running follow-up experiments on this, and embedding-based clustering misclassifies about 50% of the edge cases I threw at it. I'm currently testing an alternative using Statistical Process Control on behavioral features (step count, output length, special-character ratio) instead of semantic embeddings — it catches the structural garbage without trying to "understand" it. Not perfect, but at least the failure modes are measurable.
On sync/async and inspectability: This one stung because you're 100% correct. In production, a review queue that blocks the user is a completely different product from an async batch queue. I'm redesigning the human handoff layer around diff review — showing reviewers only what changed, not the full 500-word output, and making execution traces, retry counts, and raw trigger evidence mandatory metadata rather than optional. The cognitive load drops from ~60s to ~30s per item in my early tests.
On your final question — keep the harness or go narrow?
I've been wrestling with this exact tradeoff. My current leaning (and what I'm writing about next) is neither in its original form. The harness as a "general quality gate" is too heavy and too opaque. But narrow task-specific gates alone lose the governance value.
The direction I'm exploring is a thin routing layer — it classifies tasks by type (verifiable code, high-risk ops, low-risk drafts, medium-risk edits) and applies different rules per class, but all control logic is deterministic (no LLM-as-judge in the control plane). The harness becomes a router, not a judge. That way I keep cross-team alignment without the black-box quality layer you rightly called out.
I'm drafting two follow-up posts on exactly these threads. If you're open to it, I'd love to share the drafts with you before they go public — your comments on this one were sharp enough that I'd trust your early read over anyone else's.
Either way, thanks again. This comment genuinely helped clarify my own thinking.
The third pass of the ruler is the one that earns the whole series, because now the method is measuring its own outputs and the meta-move stays legible. Two flaws where I'd stretch what you already found.
Flaw 3 (reviewer as gold standard) is a correlated-failure floor problem more than a recursive-regress one. Two reviewers under the same UI, same schedule, same training, same fatigue curve don't produce independent errors, they produce errors that AGREE, precisely on the items where the shared training set was thin. Feed those correlated wrong labels into the closed-loop few-shot pool and you don't average the noise down, you consolidate it into a bias the pool now defends. The floor is the same one detection-anchor arguments hit: your witnesses cannot lie together, and reviewers under identical operating conditions are structurally set up to lie together on exactly the edge cases the harness cares most about. Which points at a concrete fix, the reviewer-consistency check you already sketched has to be authored by someone who hasn't trained on the same standard, not just a random reassignment among the same pool.
Flaw 1 (cluster approval flipping L4+G4 together) is a different failure mode than "clustering doesn't work", it's an evidence-surface mismatch. What the reviewer LOOKED at was one representative; what the approve() call COVERED was every cluster member. Same shape as any ratification where signed-scope > reviewed-scope. Fix is at the read layer: cluster.approve() returns (verdict, covered_ids) and the downstream write refuses cluster-wide propagation without an explicit per-member evidence surface, otherwise "one representative" quietly extends to "all members" through a channel nobody instrumented. The format-vs-semantics distance the embedding lost is real, but the operational damage comes from the approval being coarser than the evidence.
The line that survives all three passes: architecture drawn ≠ problem solved. Same rhetorical trap as "decided ≠ decided correctly" one level up. Six flaws isn't a failure of design, it's the moment the design started being real, because it's the first version where the writer's ruler was pointed at the writer.