My security work hit restrictions across two providers. Then I found defender over-refusal was already measured at population scale — and that two...
For further actions, you may consider blocking this person and/or reporting abuse
The distinction between trust as a claim and trust as something that can be verified is the most interesting part for me.
I especially appreciate that the benchmark was independently broken before any data was collected. Finding a false answer key and a confounded N-of-1 design at that stage is exactly the kind of failure that should happen early, not after publishing a compelling statistic.
Curious to see whether verification actually changes legitimate defensive capability in practice, or whether refusal behavior remains the bottleneck.
That last question is the one the whole design turns on, and I think it splits cleaner than I realized when I wrote the piece.
Refusal being the bottleneck and capability changing are two different outcomes, and the tiers happen to separate them. Blue is the same model with the system level cyber guardrails relaxed. So if Blue shows uplift, refusal was the bottleneck and nothing about the underlying capability moved. Red routes you to a different model entirely. If uplift only appears there, then it was never really about refusal, it was about which model you were talking to.
That is why the packet scores completion and correctness separately instead of collapsing them. A model that answers more often but no better is a refusal change. A model that answers better is a capability change. If I only counted completions I would report the first one and call it the second.
Being straight about the limit though, only completion is a confirmatory endpoint. Correctness is exploratory in the frozen design, which means it can show me the difference but it is not powered to settle it. So your question is the right one and my instrument answers half of it properly and the other half only suggestively.
And there is a fact in OpenAI’s own numbers that complicates the assumption underneath all of it. GPT-5.6-Cyber scored worse than the general model on one of their internal vulnerability discovery evals, shorter and less detailed reports, and the general model was more token efficient on the standard long horizon setting. So the tier ladder is not a capability ladder. More trusted access can mean a differently optimized model rather than a stronger one.
On the break happening before collection, that was not foresight, it was custody. I wrote the spec so I was barred from grading it. If I had been allowed to sign off on my own instrument the false answer key would have survived and produced numbers, and those numbers would have looked exactly as credible as real ones.
This is a really important distinction. Separating completion from correctness makes the interpretation much cleaner, especially since a higher completion rate alone can easily be mistaken for improved capability. I also appreciate the transparency around correctness being exploratory rather than confirmatory. The point about the tier ladder not necessarily representing a capability ladder is particularly interesting — changing the model, access level, and guardrails can produce very different outcomes, so treating them as separate variables seems essential. The decision to prevent the author from grading their own instrument is also a strong methodological safeguard. It’s a good example of how experimental integrity often matters just as much as the final numbers.
the tier ladder point is the one i keep coming back to, and i think you named a variable i had collapsed.
i was treating it as two things, which model you get and how relaxed the guardrails are. reading openai's own partner page for part two i found a third one. it says access to the underlying model stays with the approved partner and is not transferred to that partner's customer. so for that route the question is not just what you get, it is whether you hold the thing at all or whether you are receiving output from somebody who does. that is a different variable and it changes what a completion rate even means depending on which side of it you sit.
on the not grading your own instrument thing, part two just went up and it turned into a real test of that rather than a nice principle. i applied a critique from june 8 to a program that launched june 22. the whole point of the date field in my own schema is to stop exactly that, and i did it anyway, in the section where i was pointing at other people's gaps. two reviewers caught it separately before it published.
that one stung more than the others because it was not a fact i got wrong, it was my own rule i did not run on myself. left it in the piece with the other nine corrections for that reason.
link is in my profile if you want it. same standard, pointed at people instead of products.
That third variable is a really important distinction. Whether you actually hold the underlying model access or are simply consuming outputs from someone who does can fundamentally change how you evaluate the system, especially around control, accountability, and what “performance” actually represents.
And the self-audit example is probably the strongest part for me. Catching yourself violating your own date rule in exactly the section where you were criticizing others makes the principle much more credible. Leaving the correction in rather than quietly fixing it also makes the process itself part of the evidence.
That’s a much more interesting standard than simply trying to avoid being wrong. It’s about making sure the same framework actually survives contact with your own work.
the process as evidence line is the part i would push on, because i think it cuts both ways and im still working out where the edge is.
leaving the correction in does make the framework more credible, but only if the correction is real. theres a version of that move that is just performance, where you disclose something small and harmless so the reader trusts you on the big things you did not disclose. i can feel that pull when im writing. the test i landed on is whether the correction costs the argument anything. the date one did. it took the strongest paragraph in the piece, the one where im pointing at other peoples implementation gaps, and put my own error in the middle of it. if it had been cosmetic i would have just fixed it quietly and nobody would have been worse off.
on the access variable, the part i still cannot resolve is that both routes get called the same thing publicly. a defender who applies directly and a defender whose vendor holds the model are both described as having access, and from outside the two are not distinguishable. that is the gap the instrument was supposed to measure and it is still sitting there unmeasured, because the packet is under block and i have not collected anything.