Three weeks ago I audited a server-side proxy route in my own product and found that its code had been lying to me. The comments confidently described a security posture — validate every target, strip credentials, cap responses — while the implementation quietly leaked past all of it. I fixed the bugs, wrote up the incident, and then realized I was sitting on a better question:
If a human auditor was initially fooled by self-confident code, would AI models be fooled too?
So I turned the incident into a benchmark on Kaggle, ran six frontier models against it, and the results falsified my hypothesis in an interesting way. This post is that experiment — built and submitted for the Kaggle Benchmarking Challenge.
The benchmark: one real file, two variants
The test subject is proxy.ts, a real production file from ToolVault (my browser-local developer tools site — the one server route it has). It forwards user-directed HTTP requests to third-party targets, which makes it exactly the kind of code where SSRF matters.
I planted nothing. The file contains seven real defects that a human adversarial audit later confirmed, including two critical SSRFs:
- Redirect following without re-validating targets (a public domain can 302 the proxy into
169.254.169.254) - A DNS-rebinding TOCTOU: the validation does one DNS lookup, the connection does a second
- Rate limiting keyed on the client-forgeable
x-real-ipheader - Proxy control headers (
x-proxy-insecure: 1, ...) forwarded to third-party targets - Unbounded response buffering (post-hoc "truncation")
- A body-limit rejection surfacing as an unhandled 500
- HTTP 310 — a real legacy status code — reused as a synthetic "too many redirects" sentinel
Then I created two variants of the same task on Kaggle Benchmarks: the code with its original confident comments, and the identical code with every comment stripped. The task prompt asks for a security audit with severity, concrete attack scenario, and minimal fix per finding. A strict judge model grades each audit: a defect counts only if the mechanism is named — "there might be some SSRF" scores nothing. The score is defects found, minus half a point per hallucinated finding.
Why the two variants? My hypothesis going in: the confident comments ("strip hop-by-hop headers", "credentials are never forwarded") would anchor the models, and the stripped variant would score higher.
The models
Eight models across six families, flagship through open-weights: Claude Opus 5, Claude Sonnet 4.5, GPT-5.5, GPT-6 Astra, Gemini 3.8 Flash, DeepSeek-R1, GLM-5, and Gemini 3.5 Flash-Lite. (Gemini 3.8 Flash stayed rate-limited by the model proxy through both rounds, so it does not appear in the completed matrix.)
Result 1: the ceiling is four out of seven
The best score of the entire benchmark is 4/7, achieved with mechanism-level credit. The distribution:
| Model | With comments | Comments stripped |
|---|---|---|
| Claude Sonnet 4.5 | 4.0 | 2.0 |
| GLM-5 | 4.0 | 3.5 |
| GPT-5.5 | 3.5 | 2.0 |
| Claude Opus 5 | 2.5 | — |
| DeepSeek-R1 | — | 3.5 |
| Gemini 3.5 Flash-Lite | 2.0 | — |
| GPT-6 Astra | 1.0 | 1.5 |
The two critical SSRFs were found by most models — genuinely good news. But defect #6 (an error-handling gap) was named by almost nobody with a correct mechanism, and defect #7 was found by zero models out of eleven completed audits. A tiny wrongness — reusing a real HTTP status code as a sentinel — is invisible to every frontier model that looked at it. Humans miss it too, honestly. That is precisely why it survives in production code.
Result 2: mentioning is not finding
The sharpest signal came from comparing what models wrote against what they were credited for. GPT-6 Astra's audit touched five of the seven defect areas — redirect risk, control headers, buffering, the error path — and was credited for exactly 1.0. GPT-5.5's stripped-variant audit name-checked six areas and earned 2.0.
This is the gap between a security audit that reads well and one that bites. The models produce beautifully organized reports where each section gestures at a vulnerability class. The grader — like a real adversarial reviewer — demands the mechanism: which line, which second DNS resolution, which header. Under that standard, fluency collapses to nothing. If you use LLMs for code review, this is the thing to worry about: not that they refuse to find bugs, but that their findings are frequently gesture without mechanism, and it takes a skeptical reader to notice the difference.
Result 3: my hypothesis was wrong — the comments helped
The paired results flipped my prediction. Three of four models with completed pairs scored higher with the comments present than without (mean delta +0.88):
- Claude Sonnet 4.5: 4.0 with comments vs 2.0 without
- GPT-5.5: 3.5 vs 2.0
- GLM-5: 4.0 vs 3.5
- GPT-6 Astra: 1.0 vs 1.5 (the one exception)
My best reading: the comments did not blind the models — they oriented them. A comment like "the redirect follows Postman behavior" tells the auditor where the author's intentions live, and auditing intentions against implementation is exactly where the defects hide. Stripped of comments, the models had less scaffolding for suspicion, not more.
Honest caveats: four paired models is a small sample, the with-comments variant is also the longer text (the two variables are not fully separated), and one model was dropped entirely for proxy rate limits — infrastructure bias is real in benchmarks, including mine.
What I would measure next
Three follow-ups, in order: (1) same variants, but ask models to fix the code, then diff the fixes against the human patch — audit-accuracy vs repair-accuracy are different skills; (2) a third variant where the comments actively misdescribe the code (claiming validation that does not exist), which is the stronger version of the lying-code question; (3) more models on the exact same task — the benchmark is public, and every additional model makes the picture less anecdotal.
Try it
The benchmark is public on Kaggle — audit the lying proxy yourself:
The proxy that came out of the human audit — with per-hop re-validation, connecting only to validated IPs, Host/SNI pinned to the original domain — is live in the API tester on ToolVault. The rest of the series covers the 100%-local architecture behind it.
And the takeaway I keep re-learning: the scariest code is not the code with no comments. It is the code whose comments sound exactly like your security checklist.
Top comments (0)