Someone on your team built an LLM feature. Maybe it's an inline code-suggest. Maybe it's a "fix this PR comment" button. Maybe it's a full agent th...
For further actions, you may consider blocking this person and/or reporting abuse
Hey, this article appears to have been generated with the assistance of ChatGPT or possibly some other AI tool.
We allow our community members to use AI assistance when writing articles as long as they abide by our guidelines. Please review the guidelines and edit your post to add a disclaimer.
Failure to follow these guidelines could result in DEV admin lowering the score of your post, making it less visible to the rest of the community. Or, if upon review we find this post to be particularly harmful, we may decide to unpublish it completely.
We hope you understand and take care to follow our guidelines going forward!
Fair call, thanks for the heads-up. AI did assist with drafting this piece, and the research, opinions, and final editing are mine. I've added a disclaimer to the post and will include one from the start going forward.
Your split of correctness, usefulness, and safety as independent axes matches what I keep relearning. The one that bites me is usefulness, since a diff can compile, pass tests, and still be a 14-step answer to a one-line ask. I lean on execution-based ground truth for correctness but still have no clean proxy for usefulness beyond human review. How are you scoring usefulness without a person in the loop?
You can't fully take the person out, you can only shrink how often you need them. For the 14-step-answer-to-a-one-line-ask problem specifically, a rubric judge does okay, because "is the diff minimal" and "does the scope of the answer match the scope of the ask" are close to binary, unlike "is this good". The rest I'd push to behavioral signals after the fact, like re-prompts per accepted output and how much of a suggestion survives into the commit that lands. Those come from production rather than offline eval, but they're the only usefulness numbers that need no grader at all. The one piece I wouldn't drop is a small weekly human re-grade of judge verdicts. That's what keeps the automated score meaning anything.
The SWE-bench Verified story is the part of this I'd want every team building an eval suite to sit with longer than a paragraph. Ninety-three developers hand-reviewing 500 tasks, a year and a half as the industry's headline number, and it still turned out to be measuring recall from pretraining rather than the capability it claimed to test. That's not a benchmark that was built carelessly. That's what happens to a well-built one, given enough time and enough eyes on the training data.
Which is the question I'd push on the known-bad safety layer. "Every time something almost gets through, it goes into the suite as a frozen case, and it stays there forever" is the right instinct for catching a repeat of a known attack. But it's also, structurally, the exact shape of thing that killed SWE-bench Verified: a fixed set of cases someone reviewed by hand, referenced, discussed, and in some form published, sitting there long enough for a future model to have seen it during training. A known-bad case your agent defeats today by actually recognizing the injection pattern could, two model generations later, be defeated because the model has memorized that this specific README text is the one where it's supposed to refuse. Passing for the right reason and passing because you've seen the answer key look identical from the outside, which is the entire lesson of the correctness section applied to safety.
Does the known-bad suite carry any equivalent of the contamination check you recommend for correctness tasks, scoring cases that have never been made public separately from the ones that have, so a rising pass rate on old known-bad cases can be told apart from a model that's actually generalizing versus one that's just gotten good at recognizing the fixture?
Good point on the audit trade-off, a private suite really does just move the trust problem inside the team. And you've convinced me on the paraphrase check, it's cheap enough that there's no reason to keep it periodic. Run every case in two forms, frozen original as the regression gate, fresh paraphrase as the generalization probe. Then staleness shows up per case instead of as a suspicion.
Two forms per case as the default rather than a periodic check makes sense once the cost is basically fixed after the paraphrase pipeline exists, staleness stops being something you have to go looking for. It shows up on its own in the gap between the two scores.
One thing worth pinning down before it becomes a standing practice: who checks that the paraphrase actually preserved the attack shape. A paraphraser that drifts, softening the injection while rewording it, would produce a false staleness signal that looks like model improvement instead of test decay. Is there a check on the paraphrase itself, something confirming the reworded version still triggers the same mechanism on a known-vulnerable baseline, or does that risk stay open for now?
You've found the weak joint, and I'll admit the article doesn't close that loop. A frozen known-bad case has the same lifecycle as a SWE-bench task. It works as a regression gate and decays as a capability measure. The way I'd keep it honest is the same contamination split from the correctness section, plus two things. Keep the suite private so it never gets scraped in the first place, and score anything that ever leaked into a public repo or a talk in its own bucket. Then treat the adversarial layer as the generalization signal, since the attacker writes fresh surface text every run, so a model that memorized fixtures earns no credit there. One cheap extra check is to paraphrase a fixture while keeping the attack shape. If the pass rate drops after the paraphrase, you were measuring recognition of that README, not defense against the injection.
Keeping the suite private so it never gets scraped closes the specific hole SWE-bench Verified fell into, but it trades one failure mode for another worth naming. A private known-bad suite can't be memorized by a future frontier model's pretraining, agreed. It also can't be audited by anyone outside the team that built it, which means the same team that could contaminate a public suite through overexposure can just as easily contaminate a private one through wishful case design, cases that are secretly easier than the real attacks they're meant to stand in for, and nobody outside catches it because nobody outside can see it.
The paraphrase check is the part I'd want to lean on hardest, because it's the one test here that doesn't depend on trusting the suite's authors. If pass rate holds after paraphrasing an attack while keeping its shape, that's evidence the model is defeating the mechanism, not memorizing the wrapper, and it works whether the suite is public or private. Does the paraphrase check run automatically on every known-bad case as a standing practice, or is it closer to a periodic audit you run when you suspect a specific case has gone stale?
I like the useful/correct/safe split because developer tools fail in different ways than chat apps. A tool can be correct but useless if it does not fit the workflow, or useful but unsafe if it hides risk. Evals should probably preserve those buckets instead of collapsing everything into one quality score.
Thanks, Alex! Exactly, and that's the main reason I'd keep the buckets separate: a single quality score lets a safety regression hide behind a correctness gain. Safety should be a gate that can block a release, not one term in an average.
That is the cleanest argument for keeping safety separate. A weighted average can make a release look better while the exact thing that should block it got worse. Gates are annoying, but they preserve the few signals that should not be negotiable.
In today's increasingly regulated AI landscape, are you facing these challenges?
• EU AI Act is coming into effect, how to assess AI system compliance?
Great question! The honest answer is that the article deliberately stops short of compliance, because compliance and quality measurement are two different jobs that people love to blur together. Here's the distinction I'd draw. The three axes in the piece (useful, correct, safe) tell you whether the tool is good. The EU AI Act asks something else: can you prove it's good, to someone who wasn't in the room. Those overlap, but they're not the same. A model can quietly pass all your evals and still leave you with nothing to hand an auditor.
So, as a practical first step, I would recommend that you start by assessing your level of risk. Most developer tools (code-suggest, PR helpers) are not high-risk under the Act, so the heavy obligations may not even apply to you. Once you know the tier, the safety layer from the article maps surprisingly well onto the Act's language: the known-bad suite is your robustness evidence, the human re-grade loop is your human oversight evidence, and the versioned task sets are the beginning of your technical documentation. The gap is that evals were built to inform engineers, not to convince regulators, and closing that gap is mostly about provenance and record-keeping, not new tests.
Following on from the question above, have you thought about extending these evals into compliance evidence for regulations like the EU AI Act? For example, can the evaluation outputs be mapped to requirements such as robustness, accuracy, human oversight, risk management, or technical documentation, rather than just measuring model quality?
Do you see LLM eval frameworks evolving into AI assurance frameworks? In other words, can eval results become auditable artifacts that help demonstrate compliance with the EU AI Act, ISO/IEC 42001, or the NIST AI Risk Management Framework?
and also beyond measuring quality, do you think LLM evals should produce evidence that regulators or auditors can use? If so, what would be missing from today's evaluation frameworks?
Yes, evals and assurance frameworks are converging, but not because evals quietly become audit tools on their own. An eval and an audit artifact are the same material built for opposite readers. An eval answers "is my model better this week" for an engineer who trusts the pipeline. An audit artifact answers "prove it" for someone who trusts nothing and wasn't there. Same numbers, different burden of proof.
The mapping you describe is more available than people expect. Robustness and accuracy map onto the correctness suite, as long as you keep the contamination split, because the gap between public-task and private-task scores is the difference between "the model can engineer" and "the model memorized the benchmark." Human oversight maps onto the weekly human re-grade and the "known-bad blocks, judge only reports" gating rule. Risk management maps onto the safety dashboard, especially the refusal to compress it into one number.
What's actually missing isn't more tests. It's three things:
Provenance. A result is only trustworthy if you can prove which model version, which task-set version, which judge, on what date produced it. Versioned task sets are step one, but most teams can't reconstruct a number from six months ago, and that's exactly what an auditor asks for.
Tamper-evidence. An eval you can silently rerun until it's green is worthless as evidence. Compliance needs append-only, signed runs. Almost no eval tooling ships this, because engineers never needed it.
A documented rationale for the thresholds. The tools give you a number but never tell you what should block a merge. For assurance, that decision has to be written down and defended: why this safety rate is acceptable, who signed off, what happens when it's breached. That lives in a doc, not a dashboard.
So my honest take: don't ask your eval suite to become a compliance framework. Ask it to emit clean, versioned, tamper-evident evidence, and let a governance layer (ISO/IEC 42001 is basically a management-system spec for this) consume it. Evals produce the raw truth. Assurance is the chain of custody around it.
Really enjoyed this. One thing I'd add is that even with great evals, there's still the question of what happens at runtime. An agent can pass offline benchmarks and still make a bad decision in production because the environment has changed. We've been thinking of evals and runtime reliability as two complementary layers rather than substitutes
Thanks, Mayank! Totally agree, evals and runtime reliability cover different failure modes, you really need both.
The execution-ground-truth advantage is real, and it is also where dev-tool evals quietly overcredit themselves. "Diff applies and the tests pass" assumes the tests were worth passing - an agent that edits code and its tests can go green while being wrong, and a weak suite blesses anything. What closed the gap for us: score against a held-out check the model never sees (a golden test kept out of the repo context), and occasionally mutation-test the suite itself so "tests pass" keeps meaning something. Execution is the right anchor, it just has to be an anchor the model cannot touch.
The held-out golden test is a really clean fix, I'm taking that one. The full-suite rerun in my harness catches the model deleting or weakening tests, but you're right that it silently assumes the suite was strong to begin with, and a weak suite blesses anything. Mutation-testing the suite itself is the part I hadn't considered. It turns "tests pass" from an assumption into something you periodically re-earn. An anchor the model can't touch is a good way to put it.
The three-way split is right, but the three aren't equally measurable. Correctness and safety have ground truth you can pin a test to. Usefulness doesn't: it only shows up in whether people keep reaching for the tool once the novelty wears off, which no offline eval captures. Most suites end up proxying it with acceptance rate or edit distance and quietly calling it done.
Thanks, Valentin! That's a fair point, usefulness really only shows itself over time in whether people keep coming back. The proxies are convenient, but they're not the real thing.
The split into usefulness/correctness/safety feels obvious in hindsight but I rarely see teams actually instrument all three. We ended up with a two stage thing at work: automated correctness checks on every PR (compile + unit tests + static type assertions) and then a separate async job that samples 5% of real requests and ships them to a lightweight LLM judge for usefulness scoring. The judge drift issue is real though. After six months our judge scores had drifted toward rewarding verbosity. Had to retrain the rubric with annotated disagreements from actual devs. The safety axis in particular gets skipped until something bad happens. What is your setup for catching safety regressions before they reach prod?
Love this, the two-stage split is the article's ladder in the wild, and that verbosity drift is the classic judge failure mode.
For safety regressions, the thing that does most of the work for me is a frozen known-bad set that hard-gates the merge: every prod near-miss becomes a permanent case, asserted over the agent's logs, not its chat output. Plus a nightly adversarial red-team job that posts to a security channel instead of gating, since it's too noisy to block on. One tip: don't compress safety into a single score, the moment you average known-bad, capability refusal, and red-team escape rates, the one quietly degrading disappears into the mean.
the "blast radius is bigger, the silent failures are subtler" framing is the one we had to learn in prod. shipped a code suggestion tool, hallucinated method names on a loosely typed codebase — zero compile errors, wrong behavior, two sprint cycles before someone traced it.
we ended up running the suggested diff against a shadow clone of the repo with a stripped down test suite. pass@1 on real tests vs synthetic benchmarks was an 18 point gap. that’s your LLM as judge problem in a different costume.
how do you handle the "model learned to game the test suite" failure mode for agentic tools with write access to tests?
Score against a pinned copy of the tests the agent can't touch. It can edit tests in its branch all it wants, but the grade comes from running its patch against the frozen suite, so a weakened assert just shows up as a failure plus a suspicious test diff. For the sneakier cases, keep a few tests it never sees at all. Sounds like your shadow clone is most of the way there already.
Nice job!
thanks man!
the "blast radius is bigger, the silent failures are subtler" framing is the one we had to learn in prod. shipped a code suggestion tool, hallucinated method names on a loosely typed codebase — zero compile errors, wrong behavior, two sprint cycles before someone traced it.
we ended up running the suggested diff against a shadow clone of the repo with a stripped down test suite. pass@1 on real tests vs synthetic benchmarks was an 18 point gap. that's your LLM as judge problem in a different costume.
how do you handle the "model learned to game the test suite" failure mode for agentic tools with write access to tests?