The interesting result isn't who won. It's that across four security domains, Claude and Gemini missed the same hardening steps — and if you've shi...
For further actions, you may consider blocking this person and/or reporting abuse
That 63% stat matches our internal testing — we run an AI test automation framework and consistently find that AI-generated code passes lint but fails integration tests on exactly the same edge cases. Both models missing the same hardening steps is the real story here.
Have you looked at complementing the ESLint plugin with runtime testing (fuzzing/invariant checks) vs just static analysis? I suspect the gap between "lint passes" and "production-safe" is even wider than the 63% suggests.
The lint-passes-but-integration-fails gap is real and imo understated. Static rules catch the vulnerability you already know about — the missing algorithm allowlist on JWT, the absent $where guard in Mongo. But they can't catch the vulnerability that only exists in context, like a timing-safe comparison that's correct in isolation but sits behind a cache layer that makes the timing difference observable anyway. Fuzzing gets closer because it exercises the actual runtime path, but even that misses deployment-specific stuff like reverse proxy configs that strip security headers before they reach the app. The 63% is probably a floor, not a ceiling.
That gap between "correct in isolation" and "safe in the deployed chain" is the part static analysis structurally can't reach — it has no model of what sits in front of or behind the code it's scanning. I've hit the header-stripping case directly: a
Strict-Transport-Securityheader set correctly in app code, verified by the linter, silently dropped by an nginxproxy_passblock that didn't forward it. Passed every static and even most dynamic checks because the app's response was correct; only a live curl against the actual edge caught it. Where I'd push back slightly: fuzzing plus config-as-code scanning (checking the proxy/infra config itself, not just the app) closes a good chunk of that specific gap. It doesn't touch your timing/cache example though — that's an emergent property of two correct components, and I don't think any static or fuzz tooling catches that class at all yet.The cache-layer timing example hits home — that's the exact class of bug that passes any isolated review because it is correct in isolation. The flaw only exists in the composition.
From the testing side we see the mirror image in test tooling itself. You mock the cache → the timing comparison passes. You mock the DB → the projection passes. Each layer validates fine on its own, until the deployment stitches them together and the assumptions collide in ways none of the individual checks could surface.
The 63% floor resonates. It's almost like "lint passes" and "production-safe" are measuring different things entirely. Has anything been more effective at catching these composition-level gaps than just deeper staging environments?
The technique change under mocking is the crux — when you mock the cache, you've frozen the exact variable whose real-world variance creates the bug, so the test can't fail even in principle. I've seen this most with connection-pool exhaustion: pool mocked in unit tests, DB mocked in integration tests, neither layer ever sees the other's backpressure, and the pool starves in production under retry storms that only exist because both layers are real. Deeper staging helps, but what actually caught it for me was contract tests asserting timing/backpressure behavior explicitly, not just output shape — forcing the composition assumption to be a first-class thing under test rather than an emergent property nobody wrote down.
That gap between "correct in isolation" and "safe in the deployed chain" is the part static analysis structurally can't reach — it has no model of what sits in front of or behind the code it's scanning. I've hit the header-stripping case directly: a 'Strict-Transport-Security' header set correctly in app code, verified by the linter, silently dropped by an nginx 'proxy_pass' block that didn't forward it. Passed every static and even most dynamic checks because the app's response was correct; only a live curl against the actual edge caught it. Where I'd push back slightly: fuzzing plus config-as-code scanning (checking the proxy/infra config itself, not just the app) closes a good chunk of that specific gap. It doesn't touch your timing/cache example though — that's an emergent property of two correct components, and I don't think any static or fuzz tooling catches that class at all yet.
The 63% hardening gap is the part that matters most. Model comparison is interesting, but the operational takeaway is that generated code needs a security checklist around it: auth boundaries, input validation, secret handling, logging, dependency risk, and failure behavior. The model is only one layer.
The 63% number tracks with what I've hit in practice: models default to happy-path auth checks and skip the token-refresh/session-expiry branch almost every time, because that failure mode rarely shows up in training examples labeled "correct." Same pattern with secret handling — code will read from env vars correctly but silently log the full request object on error, dumping the secret into stdout. Where I'd push back slightly: a static checklist catches maybe half of this. The other half is dynamic — you need to actually exercise the failure paths (expired tokens, malformed input, dependency timeouts) because the code often looks hardened until you trigger the branch nobody tested.
That is the gap I keep seeing too. Static checks are useful as a floor, but they do not prove the branch behaves under stress. Expired tokens, dependency timeouts, malformed payloads, and noisy logs are where the "secure enough" story usually collapses.
the '63% of AI code skips hardening' stat is the real headline, not which model won. that skipped-hardening gap is precisely what a harness has to close: in Moonshift the security/hardening checks are gates between steps, not optional passes, before agents build + deploy + market a SaaS overnight. excellent benchmark work. first run's free if you want to see hardening enforced rather than hoped for.
63% is close to a number I've hit directly: benchmarking a static-analysis rule set against known-vulnerable fixtures, the gap wasn't detection logic, it was that agents write code that passes the happy-path test and stop — no negative test, no boundary check, no rate limit, because nothing forced a second pass. Gates between steps is the right shape for that; the failure mode you're targeting is real and I've seen it survive code review too, since reviewers eyeball diffs, not omissions. Where I'd push back: gates only close the gap if the check set is opinionated enough to catch omission, not just regression — a generic linter gate would wave the same 63% through.
Insightful!
Appreciate your sharing about Moonshift's approach!
Very interesting. Thanks
You welcome!
It is very interesting to see the different between the security protocols of claud and gemini.
They land differently because of how the two labs weight refusal training against helpfulness metrics. Claude's constitutional AI approach tends to catch injection-style prompts (concatenated SQL, unsanitized shell calls) more consistently, because the training explicitly rewards flagging the pattern even when the surrounding ask looks legitimate. Gemini's safety layer felt more calibrated toward output filtering than input-time reasoning about intent, in my testing — it'll block an obviously malicious payload but is more likely to wave through code that's dangerous by omission, like missing bounds checks. Neither one reliably applies hardening unless you prompt for it explicitly, which is really the finding underneath the "63%" number in this piece: the gap between models is smaller than the gap between "asked for security" and "didn't."
It is very interesting! I hear the latest AI security breaches from OpenAI, Anthropic, and Meta since you made your post. We are in a dangerous place with the advancement of AI models. I hope that Google and Microsoft are implementing their own security protocols with their own experimental AI models at the moment. We don't more stories in the news lately.
I'm glad you found it valuable!