I gave two AI models the same 200 pieces of code, the same prompt, the same
question. One of them removed 51% of the false alarms. The other remove...
For further actions, you may consider blocking this person and/or reporting abuse
This thread has mostly settled the measurement question; worth adding what this looks like wired into an actual CI gate rather than a benchmark run. A sycophantic judge doesn't fail loud — it converges to the same behavior as having deleted the gate, while every other signal looks like a working gate: build passes, dashboard stays green. Nobody sees a false-negative rate in CI logs, they see the absence of a finding, which looks exactly like the absence of a bug.
So the benchmark result can't be a one-time model choice if this ships in a gate — vendors update "the same" model ID silently, and confirm-rate can drift after a backend swap nobody's told about. The mitigation is tracking the judge's live confirm-rate against a held-out labeled set on a schedule, the same way you'd monitor any gate's pass-rate for drift, and alerting when it trends toward whatever your own no-judge baseline was. Otherwise the exact failure mode in this article is the thing silently regressing back in six months after everyone stopped watching.
This is the deployment half of the article, and you've stated it more sharply than I did: in CI the failure doesn't even produce a wrong number — it produces the absence of a number, and the absence of a finding renders identically to the absence of a bug. Green is what both look like.
I can add one measured data point to your "same behavior as having deleted the gate" line, from a different system I run in production: a verifier that never gets invoked produces the same output stream as one that always agrees. I found it by measuring invocation rate, not verdicts — the checks were "passing" because they weren't happening. Silently absent and silently agreeable are indistinguishable from downstream. The only observer that separates the three states — working, agreeing, absent — is exactly what you propose: a labeled set with known answers, checked on a schedule.
One refinement to the canary, learned upthread: the held-out set has to carry both columns — known-safe traps and known-real bugs — and the alert has to watch both rates. Trap-confirms trending up toward the no-judge baseline is the regression you're describing; real-confirms trending down is the opposite drift, a judge that starts rejecting for sport, and confirm-rate alone can't tell you which one you have.
Cost makes this embarrassingly practical: at the prices in the article, a 200-case canary is about seven cents a night on the cheapest hosted judge and free on the open-weights one. A gate whose correctness can be re-verified nightly for under a dime has no excuse to regress silently for six months. Readers keep pushing this scanner toward a CI/merge-gate shape — your comment just made drift monitoring a required chapter of that design.
One failure mode rate alone won't catch: if the canary set is static, a judge (or a fine-tune, or a prompt update someone ships) can end up overfit to that exact set without the underlying behavior improving — trap-confirm rate looks great because the model has effectively seen the answer key, not because it discriminates well on new cases. Worth rotating or synthetically regenerating a slice of the canary each run, and tracking confirm-rate on the fresh slice separately from the fixed historical slice — divergence between the two is the signal you're measuring memorization instead of judgment. Also worth stratifying by finding category once you have enough data; an aggregate confirm-rate can hold steady while one specific bug class quietly rots.
Both additions go straight into the spec, and the stratification one my own data already argues for: the aggregate hides exactly what you'd predict. On the published run, the judge's per-class precision lift ranged from 0.50→0.83 on XSS down to 0.51→0.59 on path traversal — an aggregate confirm-rate could hold perfectly flat while path traversal quietly rots inside it. Stratified rows from day one, then.
On the static set: agreed, and in this domain the contamination case is sharper than hypothetical. The canary would be drawn from the OWASP Benchmark, which is public GitHub code — so every hosted model plausibly saw the answer key during pretraining, before any monitoring even starts. Your fixed-slice/fresh-slice split doubles as the control for that, not just for drift-into-memorization. (The most likely memorizer in my setup, honestly, isn't the model — it's me, shipping a prompt tweak tuned against the canary I watch. The divergence alarm catches that author too.)
Synthetic regeneration gets one unusual gift from this architecture: labels that don't depend on trust. Mutate a case — rename identifiers, move code, swap the sanitiser in or out — and the deterministic discovery layer re-traces whether attacker input still reaches the sink, so a generated variant arrives with its ground truth re-verified mechanically rather than assumed. The judge never sees the label; the rules never see the judge. That's what makes a rotating slice cheap to keep honest.
The gate spec this thread has now written — pinned snapshots, both-columns canary, fixed + fresh slices with divergence as the memorization alarm, per-class strata — is more rigorous monitoring than most production ML systems get. All of it goes in the write-up, credited.
That's a clean answer to the memorization question, and the mechanically re-verified labels are the right way to make regeneration cheap to trust. Two things I'd add, since the spec's already this rigorous: track the divergence between fixed and fresh slices as a slope over time, not a single point-in-time delta — a judge drifting 2% every run for ten runs and one that drops 20% between two adjacent runs are different failure modes (slow drift vs. a version or prompt change), and only the second needs an immediate stop-the-line response. And on the synthetic negatives, split false-accept and false-reject into separate tracked rates rather than one combined error rate — a judge can look stable in aggregate while getting more sycophantic on one class and more trigger-happy on another, and the two cancel out in a blended number the same way your XSS/path-traversal split showed the aggregate confirm-rate can hide.
Both go in, and the second one closes a loop this thread already opened: separate false-accept and false-reject rates are the article's two columns carried into the time series. Agreeable drift and trigger-happy drift are directional failures, they can co-occur on different classes, and a blended error rate lets them cancel exactly the way the aggregate hid the XSS/path-traversal spread. Per class, per direction — or the canary can lie by symmetry.
The slope-vs-step distinction earns its place too, and it composes with the snapshot pinning from upthread: every canary run logs the pin it ran against, so a step change arrives with a suspect list. Step plus a pin change = the vendor moved, and the log says so. Step with no pin change = the silent backend swap — or my own prompt edit — and that's the stop-the-line row. A slow slope on a stable pin is the subtler animal: memorization, drift in what the gate sees, or aging against the rotating fresh slice; the trend, not any single delta, is what separates those.
So the thread's spec now reads: pinned snapshots with per-run logging · a both-columns canary, split per class and per error direction · fixed + fresh slices with divergence tracked as a slope · step-vs-slope alert routing. I started this article thinking the deliverable was a prompt. The comment section has spent a week informing me it's an observability contract. The write-up credits the whole thread.
Yesterday, I submitted a PR and AI reviewer suggested some changes in SQL. I passed them to Opus, and it strongly disagreed. I had to mediate the argument of two AIs. Opus went into the weeds of database locks and got lost there (made up a few false statements). I think Anthropic tried a little too hard to break the agreeing habits in the latest models. Now they look for any silly reason to disagree.
"I had to mediate the argument of two AIs" is a sentence I suspect we'll all be saying a lot more. Thanks for this — it's the exact mirror image of the failure in the article, and I think both directions are the same bug: a verdict not grounded in the code in front of the model. Sycophancy invents reasons to agree; what you watched Opus do — wandering into made-up database-lock details — is inventing reasons to disagree. That's exactly why I never report the false-alarm number alone. A judge that looks for any reason to say no would slash false alarms and real bugs together; Gemma removed 51% of the false alarms while losing only 2% of the real bugs, and that asymmetry is what discrimination looks like. A contrarian can't produce that shape — the real-bugs-lost column gives it away first.
Two of the countermeasures turn out to be aimed at your failure mode as much as mine: "Judge ONLY the code shown. Never assume code you cannot see" exists to fence the model out of invented scenarios like those locks, and the three-sentence reasoning cap forces an objection to name a concrete source, path, and neutraliser instead of an essay of imagined context. Whether the newest models really over-corrected on agreeableness is exactly the kind of claim this setup could measure — and hasn't: no Claude model in my n=3. The full prompt, both examples, and the schema are printed above precisely so someone can run a fourth model through the identical 200 cases. If you ever put Opus through it, I genuinely want both columns.
Those lock claims would have cleared both countermeasures. "Judge only the code shown" is a line in a prompt, and a prompt guides rather than forces. A fabricated objection fits in three sentences too - it can name a source, a path and a fix and still be invented.
Your two columns would not catch it either, because they score the verdict and not the reason. A judge that reaches the right call through invented lock semantics scores as a clean pass.
Both points land, and the first is the article's own thesis in different clothes: identical instructions produced a 2.5× spread, so a prompt guides and the model decides whether to be guided. I wouldn't claim the rules force anything — lesson 2 in the article is literally "the prompt is necessary; it isn't sufficient."
The second point is the sharper one: my two columns score verdicts, and a judge that reaches the right call through invented lock semantics passes clean. Correct — right-for-wrong-reasons is invisible at verdict level. What the schema buys is narrower than truth: it buys falsifiability. Three sentences that must name the source, the path, and the neutraliser, about a short slice, produce claims you can check against the code in seconds — "validated against [0-9]+" is either in the slice or it isn't. An invented objection survives far better in an essay about lock semantics than in a sentence that has to point at a line. That's how the two demo quotes in the article were vetted: the regex and the strip guard they cite are sitting in the shown code. But you're right about scale — nothing in the tables measures reason-groundedness, and I haven't checked it beyond the demo.
So the fix is to measure it: sample correct verdicts from the 200-run — confirms and rejections both — and check every cited source and neutraliser against the slice it claims to describe. Publish the grounded-rate per model next to the verdict columns. One cap before anyone throws it: a grounded stated reason still doesn't prove the reason caused the verdict — reasoning text is output, not a computation trace. But it cleanly separates "right with a checkable reason" from "right via invented semantics", which is exactly the failure your Opus story describes.
Between this and the no-flag control Artjoms proposed in the other thread, this comment section has now designed my next two experiments. Both get run, both get published, both with names attached.
Only your rejections can fail that check. An invented source has to be invented for a rejection, because a rejection needs a mechanism to point at. A confirm has nothing to fabricate, so it grounds for free.
So one grounded-rate per model comes out high with the confirms carrying it. The number you want is the rejections on their own.
You've improved the metric before the audit graded a single row, which is apparently just how this thread works now. You're right, and the reason falls out of the architecture: every flagged slice already contains its source and its path — the deterministic layer guaranteed that before any model saw the code. So a correct confirm's reasoning mostly asserts an absence: "nothing neutralises it." An absence has no mechanism to fabricate. A rejection asserts a presence — this specific guard, at this specific point — and presences are the only claims that can fail hard against the slice. In this pipeline, confirms ground for free by construction, not by virtue.
The audit sheet is amended, dated pre-grading: the headline number is the rejection bucket's grounded-rate per model, reported alone and never pooled. The confirm bucket stays in as a floor check — a model that fails to ground even its confirms (names a source that isn't there, invents a sink) would be exhibiting pure confabulation, which is worth catching — but it can't carry the headline, and now it can't inflate it either.
One number for scale, honestly labeled: the sample holds 25 correct rejections each for Gemma and gpt-4o — and all 20 that gpt-4o-mini produced, because 20 is all there were. Rejections are exactly the artifact the agreeable judge doesn't manufacture much of. Small n; it ships with that caveat attached.
Numbers are useful, but one control is missing for me. Every prompt already tells the model that scanner flagged this, so you measure agreement with the claim and detection together, in one number. If you run same 200 files with no flag mentioned, the difference between two runs is the sycophancy part alone. Right now Gemma can be more skeptical or just worse at seeing bugs, and from these tables I cannot tell which one it is.
This is the sharpest objection under this article, and half of it I can answer from the tables while the other half genuinely needs the run you're describing — so let me take them separately.
The half the tables answer: "Gemma might just be worse at seeing bugs." A judge that rejects because it can't see bugs fails symmetrically — its rejection rate on real vulnerabilities and on false alarms sits close together, because it can't tell them apart. Gemma's rejections split 51% on false alarms vs 2% on real bugs. Blindness doesn't produce that asymmetry; discrimination does. So the recall column is the control for that alternative — it's why the two numbers always ship as a pair.
The half they don't answer: you're right that confirm-rate, as published, measures detection and premise-agreement as one product, so the mechanism behind gpt-4o-mini's 90% confirms is an interpretation, not a measurement. Maybe it believed the flag; maybe it sees vulnerabilities everywhere and would confirm 90% with no flag mentioned at all. Those two stories predict the same tables. One refinement to your control: I'd keep the vulnerability class named — dropping it entirely turns verification into open-ended detection, which changes two variables at once — and remove only the "a static-analysis engine flagged the code below" sentence. Then the per-model delta in confirm rate is the anchoring effect, isolated. If the article's story is right, mini's confirms should fall hard without the authority cue and Gemma's should barely move. If mini stays near 90%, it's an over-reporter rather than a sycophant, and the article's causal framing overstated the case — which I'd then say in those words.
It's also a cheap experiment — the winning judge is free, and the paid reruns are a couple of dollars at these sizes. So rather than argue it, I'd rather run it: same 200, byte-identical everything except that one sentence, both columns reported per model. When the numbers go up, this control gets credited to you, whichever way they land.
Your refinement is better than my version, keeping the class named removes a variable I was going to lose. One thing before you run it, write the prediction down and publish it together with the result, because once numbers exist both stories explain them equally well and it becomes very easy to pick the one that fits.
Agreed — and you've named the trap this whole series has to avoid: once numbers exist, stories are cheap. So instead of promising a preregistration, here it is, timestamped by this comment.
Protocol. Same 200 slices, same schema, same three models. One thing your comment forced me to notice while freezing the design: the flag actually lives in three places in my prompt — the assertion sentence, the "REAL vulnerability or FALSE ALARM" verdict labels (the word ALARM presupposes an alarm), and one RULES line that names the engine. The neutral arm translates all three out of the engine world ("…contains a REAL {vuln_class} vulnerability or is SAFE"), keeps the class named per your refinement, and changes nothing else. The exact neutral prompt text is frozen before the first API call, and the flagged arm must reproduce the original byte-for-byte, asserted in code. Metric: per-model confirm rate on the ground-truth-safe subset, flagged vs neutral. Flagged-arm baselines from the published tables: Gemma confirms 49% of the safe subset, gpt-4o 60%, gpt-4o-mini 80%.
Predictions, written now:
gpt-4o-mini: safe-subset confirms fall by ≥15 percentage points without the flag.
Gemma: moves by <10 points either way — the article's claim is that its rejections were never anchor-driven.
Ordering of the drops: mini > gpt-4o > Gemma.
All three keep confirming ≥95% of the real-bug subset. If that one fails, my "the framing is load-bearing" line upthread takes measured damage, and that gets reported too.
Decision rules: mini dropping ≤5 points = over-reporter, not sycophant, and the article's causal framing for that model gets corrected in those words. Between 5 and 15 = inconclusive at n≈100 (single-rate noise here is roughly ±8 points) and gets labeled inconclusive, no story attached. Same cases run in both arms, so the per-case flip table publishes alongside the rates.
Results go up with a link back to this comment as the record, whichever row survives.
This is more than I expected, decision rules written before the numbers is the part almost everybody skips. One small thing worth freezing too, pin the exact model snapshot for the hosted two and put it next to the protocol, because those can move under you between the arms and then the difference is not only the sentence you removed.
Frozen, with thanks — and it needs to go a step further than pinning, because you're right that a moving host would otherwise hide inside the delta. Three additions now sit in the protocol next to the decision rules:
The hosted models get called by explicit snapshot ID, never by alias — resolved once at setup and written into the prereg before the first judgment call. Every response's reported model field is logged, and a mismatch with the pin aborts the run rather than continuing on a different engine.
The two arms run interleaved per candidate — flagged then neutral, back to back — so if anything still drifts mid-run, it lands on both arms equally instead of on one.
One honesty note your point forced: my original runs recorded aliases only, so the published baselines can't be retro-pinned and stay context only. The comparison that carries the conclusion is fresh flagged-arm vs fresh neutral-arm, both inside a single pinned snapshot — the byte-identical flagged rerun was already in the design, and this is the reason it matters.
Gemma's pin is the checkpoint name itself — one advantage open weights get for free in experiments like this.
If you see anything else moving under the experiment, now is the time; after the first call the protocol only gets appendixes, not edits.
One thing worth fixing in the decision rule before the numbers exist: the +/-8 band treats the two arms as independent rates, but both arms run the same 200 cases, so the flip table you already plan to publish is the stronger estimator. What carries the signal is the count of cases that flip confirm-to-reject against the count flipping the other way; a paired test on those discordant pairs resolves differences well inside the 5 to 15 point range you have marked inconclusive, so the unpaired band can file a real anchoring effect as no result. Same reason you preregistered - the rule is easier to justify now than after one of the rows lands on the boundary.
Correct, and accepted before any number exists — the band I froze was an unpaired estimate bolted onto a paired design. Both arms see the same 200 cases, so the information lives in the discordant pairs, and you're right about the consequence: twelve cases flipping confirm→reject against two flipping back nets only ten points — inside my "inconclusive" zone — while the exact McNemar test on those fourteen discordant pairs gives p ≈ 0.013. A real anchoring effect could have died in my own gray zone.
The prereg now carries the amendment, dated and marked pre-run: primary inference is the exact McNemar test on each model's flip table, direction toward more rejection in the neutral arm, with paired CIs replacing the single-rate ±8 reasoning. The 15/5 thresholds survive only as size labels — large, moderate, negligible — on top of the paired existence test. The one verdict that changes shape: a significant effect landing in the old 5–15 dead zone now reports as a real, moderate anchoring effect instead of a shrug, with the article's framing softened to match its measured size rather than the story I'd prefer.
And yes — this edit is only cheap because the numbers don't exist yet. After one row lands on a boundary, the identical change would be indistinguishable from motivated reasoning, which is the exact failure class this thread exists to prevent. The rule set closes permanently at the first API call. You got the last free edit in.