This is genuinely useful data. The "directional error" insight is the kind of thing you only find by actually running experiments, not by reading benchmarks.
Your finding that 16.7% of DeepSeek's failures are structurally perfect but semantically wrong is exactly why automated eval has a ceiling. The model writes a confident, well-formatted response in the opposite direction of what the user wanted. No amount of format checking catches that.
I'd love to see the 24 test samples open-sourced. If you put them up, I'd be happy to contribute more edge cases — I've been running into similar issues with agent evaluation loops where the "success" criteria themselves are ambiguous.
One thing I've noticed: the 5-10% random audit you suggested is underrated. Most teams skip it because it feels slow, but it's the only way to catch silent regressions. Would be curious to see what your audit catches over time.
Link to your article? Would like to read the full breakdown.
But here's the twist: right after I published that, I took your suggestion about the 5‑10% random audit and built a harness around it – to systematically test whether that audit would actually catch drift. The result is my third post, which just went live:
👉 [link_to_third_post]
That post found six flaws in my own harness design, and two of them are direct echoes of your comment:
The feedback loop only works if humans actually see the ambiguous cases – but your random audit slice, if not stratified, can miss the most dangerous corner cases. In the harness, I initially sampled uniformly; that missed the long‑tail directional errors because they're rare in the overall flow but catastrophic when they happen.
The “5‑10%” number is arbitrary – my logs showed that the drift pattern changes with load, so a fixed percentage either over‑audits (wasting reviewer time) or under‑audits (missing regressions). I'm now moving to an adaptive sampling rate driven by the model's confidence score, which I describe in the third post.
You also asked about open‑sourcing the 24 test samples – yes, I've packaged them, along with the harness script, in the same GitHub repo. I'll push a dedicated samples/ folder this week. I'd genuinely love to have your edge cases added; the kind of “ambiguous success criteria” you mention is exactly where the current test set is weakest.
Finally, your line about “no amount of format checking catches directional errors” – that's now the central conclusion of the third post, too. The harness moves the error, but doesn't eliminate it. So your audit instinct is the only real safeguard we have.
Thanks again for pushing this forward – your comments have shaped the trajectory of the whole series.
Wow, I'm honestly surprised my comment had that kind of ripple effect. You did the hard part — actually running experiments and building the framework. I just pointed at something I'd been burned by before.
The adaptive sampling based on model confidence is a smart direction. Fixed-percentage audits feel "fair" but they miss exactly the kind of long-tail directional failures you're describing. The model is most confident when it's wrong in a structured way.
Happy to share more edge cases. A few I've been collecting:
Ambiguous negation: "Don't ignore the warning" (double negative that models sometimes parse as "ignore")
Implicit context: "Continue where I left off" with no prior context in the session
Tone preservation under constraints: "Rewrite this casually but keep the technical terms"
Conflicting instructions: "Be thorough but keep it under 50 words"
Drop me the GitHub link when the samples/ folder is ready. I'll fork and add a PR with these cases + the eval criteria I use for them.
Also — if you're writing a 4th article, the "why fixed audits fail" angle seems like it deserves its own deep dive. You've got the data for it now.
Xiao Man, this comment genuinely made me pause – because you just described exactly what my harness logs showed, but in clearer language than I've managed in three posts.
The fact that you recognized the ripple effect means you've been burned by the same class of failures. That's not luck – it's pattern recognition from real production pain.
"The model is most confident when it's wrong in a structured way" – this line is going in the next article. In my third‑post logs, I saw the same pattern: the judge gave high‑confidence passes to outputs that were semantically reversed (delete → keep, stop → continue) but structurally pristine. The confidence score was inversely correlated with the danger. You've named the phenomenon.
Your edge cases – these are gold
Case My harness result (third post)
Ambiguous negation: "Don't ignore the warning" The judge passed it as "warning ignored" – exactly the directional failure I documented
Implicit context: "Continue where I left off" with no prior context The harness didn't even flag this – it assumed context was present (flaw #6 in my post)
Tone preservation under constraints The strong model rejected it outright (false rejection), the weak model passed garbage through – the tradeoff curve again
Conflicting instructions: "Be thorough but keep it under 50 words" Both models struggled – the judge flagged it as "impossible," which I would have counted as a false positive, but maybe it's the correct response (the task itself is contradictory)
Your last case – "Be thorough but keep it under 50 words" – raises a deeper question I hadn't addressed: what do we count as a "failure" when the instruction itself is impossible? The harness currently classifies "impossible" as a failure of the agent, not the task spec. That's a design flaw I need to fix.
GitHub link and PR invitation
The repo is live:
👉 github.com/zxpmail/blog → agent-determinism-illusions/samples/
I'll push the samples/ folder with all 24 cases from the second post, plus the 8 phase‑gate cases from the first post, by end of day. Fork it, submit a PR with your edge cases, and I'll merge them into the test suite – your eval criteria will be especially useful because you've already thought about what "success" means for each one.
The fourth article
You're right – "why fixed audits fail" is its own post. I have the logs, and they show that uniform sampling missed 3 out of 4 directional failures because those failures were rare but catastrophic. I'm already sketching the outline:
Fixed sampling: feels fair, fails at the long tail
Adaptive sampling by confidence: catches structured wrongness
Cost‑aware sampling: weight by potential impact, not by random draw
The residual: when confidence and impact diverge (the hardest case)
If I write it, I'll be citing your "most confident when wrong in a structured way" line. Let me know if you want early access to the draft when it's ready.
Thanks for pushing beyond "good comment" into actual contribution. You're making this series better than I could alone.
That means a lot, thanks. And yeah, I'd love to review the draft of article 4 — the "why fixed audits fail" angle is exactly what this data supports.
Forking the repo now. I'll get those edge cases into proper test format and submit a PR. Should have it ready in a day or two.
One thing I noticed mapping cases to your framework: the "conflicting instructions" category might need sub-categories. "Be thorough but under 50 words" fails differently than "be casual but keep technical terms" — one is length vs depth, the other is style vs content. Models handle style conflicts better than constraint conflicts.
Also happy to be quoted. Just use the handle xm_dev_2026, keeps it simple.
The style/constraint split is sharper than my single "conflicting instructions" bucket — taking it. Style conflicts (casual-but-technical) often have a
feasible middle the model can find; constraint conflicts (thorough-but-50-words) often don't, and the model flagging "impossible" is sometimes correct
signal rather than failure. Different evaluation logic for each, not one bucket.
Draft access confirmed — I'll send the outline + draft once samples/ is up. Looking at 1–2 weeks; I want the cases anchored in real data before writing
the conclusion.
Handle xm_dev_2026 noted. Looking forward to the PR.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
This is genuinely useful data. The "directional error" insight is the kind of thing you only find by actually running experiments, not by reading benchmarks.
Your finding that 16.7% of DeepSeek's failures are structurally perfect but semantically wrong is exactly why automated eval has a ceiling. The model writes a confident, well-formatted response in the opposite direction of what the user wanted. No amount of format checking catches that.
I'd love to see the 24 test samples open-sourced. If you put them up, I'd be happy to contribute more edge cases — I've been running into similar issues with agent evaluation loops where the "success" criteria themselves are ambiguous.
One thing I've noticed: the 5-10% random audit you suggested is underrated. Most teams skip it because it feels slow, but it's the only way to catch silent regressions. Would be curious to see what your audit catches over time.
Link to your article? Would like to read the full breakdown.
Xiao Man, thanks for this – you're asking exactly the questions that sent me back to the lab.
First, the link you asked for: this is the full second post – dev.to/zxpmail/i-tested-3-models-a...
But here's the twist: right after I published that, I took your suggestion about the 5‑10% random audit and built a harness around it – to systematically test whether that audit would actually catch drift. The result is my third post, which just went live:
👉 [link_to_third_post]
That post found six flaws in my own harness design, and two of them are direct echoes of your comment:
The feedback loop only works if humans actually see the ambiguous cases – but your random audit slice, if not stratified, can miss the most dangerous corner cases. In the harness, I initially sampled uniformly; that missed the long‑tail directional errors because they're rare in the overall flow but catastrophic when they happen.
The “5‑10%” number is arbitrary – my logs showed that the drift pattern changes with load, so a fixed percentage either over‑audits (wasting reviewer time) or under‑audits (missing regressions). I'm now moving to an adaptive sampling rate driven by the model's confidence score, which I describe in the third post.
You also asked about open‑sourcing the 24 test samples – yes, I've packaged them, along with the harness script, in the same GitHub repo. I'll push a dedicated samples/ folder this week. I'd genuinely love to have your edge cases added; the kind of “ambiguous success criteria” you mention is exactly where the current test set is weakest.
Finally, your line about “no amount of format checking catches directional errors” – that's now the central conclusion of the third post, too. The harness moves the error, but doesn't eliminate it. So your audit instinct is the only real safeguard we have.
Thanks again for pushing this forward – your comments have shaped the trajectory of the whole series.
Wow, I'm honestly surprised my comment had that kind of ripple effect. You did the hard part — actually running experiments and building the framework. I just pointed at something I'd been burned by before.
The adaptive sampling based on model confidence is a smart direction. Fixed-percentage audits feel "fair" but they miss exactly the kind of long-tail directional failures you're describing. The model is most confident when it's wrong in a structured way.
Happy to share more edge cases. A few I've been collecting:
Drop me the GitHub link when the samples/ folder is ready. I'll fork and add a PR with these cases + the eval criteria I use for them.
Also — if you're writing a 4th article, the "why fixed audits fail" angle seems like it deserves its own deep dive. You've got the data for it now.
Xiao Man, this comment genuinely made me pause – because you just described exactly what my harness logs showed, but in clearer language than I've managed in three posts.
The fact that you recognized the ripple effect means you've been burned by the same class of failures. That's not luck – it's pattern recognition from real production pain.
"The model is most confident when it's wrong in a structured way" – this line is going in the next article. In my third‑post logs, I saw the same pattern: the judge gave high‑confidence passes to outputs that were semantically reversed (delete → keep, stop → continue) but structurally pristine. The confidence score was inversely correlated with the danger. You've named the phenomenon.
Your edge cases – these are gold
Case My harness result (third post)
Ambiguous negation: "Don't ignore the warning" The judge passed it as "warning ignored" – exactly the directional failure I documented
Implicit context: "Continue where I left off" with no prior context The harness didn't even flag this – it assumed context was present (flaw #6 in my post)
Tone preservation under constraints The strong model rejected it outright (false rejection), the weak model passed garbage through – the tradeoff curve again
Conflicting instructions: "Be thorough but keep it under 50 words" Both models struggled – the judge flagged it as "impossible," which I would have counted as a false positive, but maybe it's the correct response (the task itself is contradictory)
Your last case – "Be thorough but keep it under 50 words" – raises a deeper question I hadn't addressed: what do we count as a "failure" when the instruction itself is impossible? The harness currently classifies "impossible" as a failure of the agent, not the task spec. That's a design flaw I need to fix.
GitHub link and PR invitation
The repo is live:
👉 github.com/zxpmail/blog → agent-determinism-illusions/samples/
I'll push the samples/ folder with all 24 cases from the second post, plus the 8 phase‑gate cases from the first post, by end of day. Fork it, submit a PR with your edge cases, and I'll merge them into the test suite – your eval criteria will be especially useful because you've already thought about what "success" means for each one.
The fourth article
You're right – "why fixed audits fail" is its own post. I have the logs, and they show that uniform sampling missed 3 out of 4 directional failures because those failures were rare but catastrophic. I'm already sketching the outline:
Fixed sampling: feels fair, fails at the long tail
Adaptive sampling by confidence: catches structured wrongness
Cost‑aware sampling: weight by potential impact, not by random draw
The residual: when confidence and impact diverge (the hardest case)
If I write it, I'll be citing your "most confident when wrong in a structured way" line. Let me know if you want early access to the draft when it's ready.
Thanks for pushing beyond "good comment" into actual contribution. You're making this series better than I could alone.
That means a lot, thanks. And yeah, I'd love to review the draft of article 4 — the "why fixed audits fail" angle is exactly what this data supports.
Forking the repo now. I'll get those edge cases into proper test format and submit a PR. Should have it ready in a day or two.
One thing I noticed mapping cases to your framework: the "conflicting instructions" category might need sub-categories. "Be thorough but under 50 words" fails differently than "be casual but keep technical terms" — one is length vs depth, the other is style vs content. Models handle style conflicts better than constraint conflicts.
Also happy to be quoted. Just use the handle xm_dev_2026, keeps it simple.
The style/constraint split is sharper than my single "conflicting instructions" bucket — taking it. Style conflicts (casual-but-technical) often have a
feasible middle the model can find; constraint conflicts (thorough-but-50-words) often don't, and the model flagging "impossible" is sometimes correct
signal rather than failure. Different evaluation logic for each, not one bucket.
Draft access confirmed — I'll send the outline + draft once samples/ is up. Looking at 1–2 weeks; I want the cases anchored in real data before writing
the conclusion.
Handle xm_dev_2026 noted. Looking forward to the PR.