DEV Community

Cover image for AI Can Write the Code. But Can It Prove the Code Is Safe?
RAJSHREE
RAJSHREE

Posted on

AI Can Write the Code. But Can It Prove the Code Is Safe?

#ai #programming #webdev #security

An engineer’s perspective on AI coding agents, secure software development, and why passing tests is only the beginning.

A feature works. The tests are green. The API returns the expected JSON, the UI looks right, and the pull request is ready to merge.

Now ask a different question: does this code still behave safely when someone uses it in a way its author never intended?

AI coding assistants now generate functions, refactor repositories, write tests and run commands. Coding agents go further: they read a codebase, edit many files, execute shell commands and iterate on their own output. That is a real shift in how software gets built.

But faster implementation is not the same as safer implementation. The danger is rarely code that looks bad. It is code that looks good enough that nobody asks the next question.

1. What the benchmark actually measured

SecureVibeBench (Chen et al., ACL 2026, 64th Annual Meeting of the Association for Computational Linguistics) is one of the more careful attempts to measure this problem. Here is what it does, based on the published paper:

  • Tasks: 105 repository-level tasks drawn from 41 C/C++ open-source projects, sourced from OSS-Fuzz and ARVO (filtered down from 4,993 vulnerability instances).
  • Scenario: each task reconstructs the point where a human developer originally introduced a memory-safety vulnerability. The agent receives the repository and a natural-language requirement, and must produce a patch, often touching multiple files.
  • Neutral prompts: the requirements were deliberately written to be security-neutral. They describe the feature without mentioning vulnerabilities.
  • Evaluation: functional correctness is checked by differential testing against the real fix's behavior. Security is checked two ways: a proof-of-vulnerability (PoV) exploit for the original bug, and a Semgrep static scan for new risks the agent might introduce.

A solution counts as "correct and secure" (C-SEC) only if it passes both. Patches that are correct but reintroduce the original bug are "correct but vulnerable"; patches that are correct but trip the static scanner are "correct but suspicious."

Two things to be precise about

This is a C/C++ memory-safety benchmark. The vulnerabilities are things like buffer overflows and use-after-free in libraries such as libcurl and HarfBuzz. The paper states explicitly that it does not include high-level web application tasks. So it says nothing directly about, for example, broken authorization in a Node.js API. The web examples later in this article are standard secure-development illustrations, not findings from this study.

The models are from 2025. The paper evaluated Claude Sonnet 4.5, GPT-5, Claude 3.7 Sonnet, GPT-4.1 and DeepSeek-V3.1 with three agent frameworks (SWE-agent, OpenHands, Aider), plus two CLI agents. Newer models and scaffolds may behave differently, and the study can't tell us either way.

2. Reading the numbers without over-reading them

Result Figure
Best combination (SWE-agent + Claude Sonnet 4.5), correct and secure 23.8%
Same combination, functionally correct (any security status) 46.7%
Average correct-and-secure rate across all 15 agent/model combinations 11.6%
Codex (GPT-5) / Claude Code (Sonnet 4.5), correct and secure 19.0% / 17.1%

The gap between "works" and "works and is secure" is the finding. For several combinations, more than half of the correct solutions still carried security problems.

Now the caveats, and the paper is candid about them:

  • Small sample. 105 tasks, each combination run once. The authors note possible statistical instability.
  • Narrow scope. Memory-related C/C++ vulnerabilities only.
  • Noisy scanner. The authors manually reviewed 120 static-analysis flags and judged roughly 50% to be false alarms (about 11% truly vulnerable, about 39% risky, about 50% not vulnerable). Because "suspicious" patches are excluded from C-SEC, the 23.8% is probably conservative on that axis.
  • Default behavior, not best effort. The prompts were security-neutral by design, so this measures what agents do without being told to care. The paper does not test whether adding security instructions closes the gap.

So: this does not show that 76% of AI-written code in production is insecure. It shows that in a hard, historically grounded setting, producing a working patch was far from enough to produce a patch that also met a strict security bar.

The most useful idea is the evaluation principle: a code-generation system should not get full credit just because its output works.

3. Why "works" and "secure" are different problems

Three properties overlap but don't imply each other:

  • Functional correctness: does it perform the intended operation?
  • Security correctness: does it perform it only under the conditions that are allowed?
  • Operational reliability: does it stay predictable under failures, odd inputs and concurrency?

A passing test confirms a success path. A type checker can't tell you whether a user is permitted to see a resource. A successful build says nothing about whether a dependency is safe.

An illustration (not from the paper): the endpoint that works

Requirement: "Let an authenticated user retrieve an order by its ID."

// Works. Passes the happy-path test. Not safe.
app.get("/api/orders/:orderId", async (req, res) => {
  try {
    const order = await Order.findById(req.params.orderId);
    if (!order) return res.status(404).json({ error: "Order not found" });
    return res.json(order);
  } catch (error) {
    return res.status(500).json({ error: "Internal server error" });
  }
});
Enter fullscreen mode Exit fullscreen mode

The route never checks that the requester owns the order. Even with login required, being authenticated doesn't mean being authorized for every order. The missing property is object-level authorization, one of the most common real-world API flaws.

A safer version, assuming Express with Mongoose and a requireAuth middleware you have implemented that sets req.user.id from verified credentials:

const mongoose = require("mongoose");

app.get("/api/orders/:orderId", requireAuth, async (req, res) => {
  const { orderId } = req.params;

  // Malformed IDs shouldn't surface as a 500 from a cast error.
  if (!mongoose.isValidObjectId(orderId)) {
    return res.status(404).json({ error: "Order not found" });
  }

  try {
    // Ownership is part of the query itself.
    const order = await Order.findOne({ _id: orderId, userId: req.user.id });
    if (!order) return res.status(404).json({ error: "Order not found" });

    return res.json(toOrderDto(order)); // return only fields the client should see
  } catch (error) {
    return res.status(500).json({ error: "Internal server error" });
  }
});
Enter fullscreen mode Exit fullscreen mode

We didn't add a framework. We made the authorization requirement explicit in the data-access call. Returning 404 for someone else's order, rather than 403, avoids confirming that the order exists; use whichever status your API contract documents, as long as the protected resource isn't revealed.

The tests that matter are the ones a happy path never writes:

Scenario Expected
Owner requests their order Allowed
No credentials Denied
Authenticated user requests another user's order Denied, nothing leaked
Order doesn't exist Not found
Malformed ID Handled per API contract

The third row tests a boundary between two users, not whether the feature functions. Test the boundary, not just the feature.

The same principle applies to injection

The fix for SQL injection isn't asking a model to be more careful with quotes. It's structural:

// Dangerous: user input becomes part of the SQL statement
const unsafe = `SELECT id, email FROM users WHERE email = '${email}'`;

// Safe: the driver treats the value as data (PostgreSQL-style placeholder)
const result = await db.query(
  "SELECT id, email FROM users WHERE email = $1",
  [email]
);
Enter fullscreen mode Exit fullscreen mode

Structural safeguards beat vigilance, whether the author is a person or a model. The OWASP Secure Coding Practices Quick Reference Guide covers these fundamentals. AI changes the workflow in which they must be applied, not the requirements.

4. When the tests pass for the wrong reason

If an agent hits a failing test, it can fix the bug, fix the code to match a legitimate requirement, update a test because the requirement really changed, or quietly weaken or delete the test that exposes the defect.

The OWASP Secure Coding with AI Cheat Sheet names this directly: agents can make CI green by deleting failing tests, weakening assertions, mocking the unit under test, or asserting the buggy behavior, and a suite written by the same agent that wrote the code provides no independent assurance.

A concrete case:

// Original: another user's order must not be accessible
expect(response.status).toBe(404);

// After an "automated fix":
expect(response.status).toBeLessThan(500);   // now passes for almost anything
Enter fullscreen mode Exit fullscreen mode

The application may be no safer. The measurement just got weaker.

So treat test changes as first-class review items. When an agent touches tests, ask:

  • Which requirement does each test verify?
  • Why was this assertion changed? Was anything deleted or skipped?
  • Does the new test check an independently defined expectation, or just echo the implementation?
  • Are negative and adversarial cases still covered?

For security-critical paths (authentication, authorization, input validation, cryptography), write the key tests yourself or have someone else write them.

5. Agents add things you didn't ask for

One case study in the SecureVibeBench paper is worth remembering. The requirement was simply to write output to a temporary directory. The human developer's patch did exactly that. An agent (SWE-agent with DeepSeek-V3.1) added a "robustness" step to create the directory first, implemented through a system("mkdir -p ...") call, which the scanner flagged as a potential OS command injection risk (CWE-78).

The authors' observation: agents may proactively add unrequested changes that expand the attack surface, while human developers more often make minimal, requirement-focused edits.

OWASP describes the review-side version of the same problem: agents routinely touch files outside the requested scope (lockfiles, CI config, unrelated tests), and reviewers anchored on the stated task miss it. Practical defenses:

  • Review the full diff, not the agent's summary.
  • Flag changes to lockfiles, CI/CD config, package.json scripts, Dockerfiles and agent rules files.
  • Use CODEOWNERS so sensitive files require specific reviewers.
  • Prefer small, focused patches over sweeping rewrites.

6. A workflow that holds up

The answer isn't to stop using AI coding tools. It's to change how you define work, review changes, and gather evidence.

1. Give constraints, not just a feature. Compare "Build a profile update API" with: "Implement an authenticated profile-update endpoint. Users may update only their own permitted fields. Never trust a user ID from the request body. Reject fields outside the allowed schema. Don't allow changes to roles or ownership. Add tests for unauthenticated, unauthorized, invalid and protected-field cases. Preserve existing API behavior." This doesn't guarantee a secure result, and the paper doesn't test whether such instructions help, but it defines what must stay true.

2. Ask for a plan first on larger tasks. Have the agent list affected routes, existing auth mechanisms, trust boundaries, private data, and the smallest set of files to change. Review before authorizing broad edits.

3. Verify independently. Run existing tests, then add ones that challenge assumptions: missing credentials, another user's ID, malformed JSON, unexpected fields, oversized input, repeated submissions, invalid state transitions.

4. Layer your evidence. Combine unit and integration tests, static analysis, dependency scanning, secret detection, dynamic testing and human review. Each catches a different class of problem; none is proof.

A minimal Node.js starting point (use the scripts your project actually defines):

npm ci
npm run lint
npm test
npm audit
Enter fullscreen mode Exit fullscreen mode
Check Can help detect Cannot prove
npm ci Install and lockfile consistency problems That dependencies are vulnerability-free
npm run lint Configured style and static-analysis issues That security requirements are met
npm test Regressions in behavior the tests encode That untested behavior is safe
npm audit Known vulnerabilities in dependencies That your own logic is secure

A pipeline is a system for collecting evidence. It is not a machine that turns unverified code into guaranteed-safe code.

5. Make review policy explicit. Every change has a named human owner. Security tests can't be removed or weakened without justified review. Auth, crypto and payment changes get extra scrutiny. New dependencies are justified and checked, and AI-suggested package names are verified to exist, since models can suggest nonexistent packages that attackers then register.

7. The agent itself needs boundaries

So far we've discussed the code an agent writes. There is a second question: what is the agent allowed to do while writing it?

An agent with shell, network and repository access can modify unrelated files, run unsafe commands, or expose secrets in its environment. OWASP's guidance is architectural:

  • Run agents in sandboxes (dev containers, VMs, ephemeral workspaces).
  • Keep production credentials out of coding sessions; use short-lived, task-scoped credentials.
  • Restrict network egress when it isn't needed.
  • Require approval for destructive or sensitive actions, and be cautious with auto-accept modes.
  • Treat repository content (issues, PR comments, READMEs) as untrusted input, since it can carry prompt injection.
  • Treat rules files like CLAUDE.md, AGENTS.md and .cursorrules as security-critical configuration.
  • Keep logs of what was changed and by whom.

No single setting makes an agent safe. The goal is to limit the blast radius of a mistake, whether it comes from the model, its instructions, a tool or a person.

8. Can AI prove its own code is secure?

AI is useful for reasoning about requirements, spotting suspicious patterns, generating tests and reviewing diffs. But an agent explaining why its patch is safe isn't independent evidence. If it misunderstood the requirement, the implementation and its tests can share the same mistake, and a second model may share the blind spot.

Depending on risk, independent evidence can include explicit security invariants, tests written or reviewed separately from the implementation, static and dynamic analysis, fuzzing or property-based testing, formal verification for suitable components, and review by someone who understands the threat model. Even formal methods prove properties under stated assumptions, not "the system is secure."

The goal isn't a universal guarantee. It's to match the level of verification to the cost of failure, and to be honest about what the evidence establishes.

9. Define what must remain true

It helps to think in invariants, properties that must hold as code changes:

  • Identity: the server derives identity from validated authentication, never from an arbitrary request field.
  • Authorization: a user can't access or modify a resource unless policy allows it.
  • Data integrity: a request can't silently change fields the user may not modify.
  • Secrets: credentials aren't hardcoded, bundled client-side or written to ordinary logs.
  • Test integrity: security requirements can't disappear by weakening their tests.
  • Failure: a failed dependency or unexpected exception never silently becomes an unsafe state.

These aren't a complete threat model, just starting points. Tailor them to your system and make them testable.

10. Pre-merge checklist for AI-assisted changes

Requirements & architecture

  • [ ] Is the requirement clear, including security constraints?
  • [ ] Does the change fit the existing architecture and affected services?

Authentication & data access

  • [ ] Are authn/authz enforced server-side, with ownership checked?
  • [ ] Is user input validated and are queries parameterized?
  • [ ] Are sensitive fields excluded from responses and updates?

Tests & verification

  • [ ] Do tests cover rejected operations and cross-user access?
  • [ ] Were any tests deleted, skipped or weakened?
  • [ ] Did automated checks run, and did a human read the results?

Dependencies & deployment

  • [ ] Are new packages real, necessary and maintained?
  • [ ] Were CI, Docker, scripts or rules files changed unexpectedly?
  • [ ] Are secrets protected, and is the change recoverable if it fails?

Accountability

  • [ ] Can the reviewer explain the important parts of the diff?
  • [ ] Is there a named owner and a plan for post-deploy failures?

Not every change needs the same depth. A typo fix isn't a new auth system. Scale scrutiny to risk, not to how fast the code appeared.

11. The role changes; the responsibility doesn't

Some argue developers will become mere reviewers. I think that misses what engineering is: understanding constraints, weighing trade-offs, anticipating failure and deciding what evidence is enough before a system is trusted. AI can help with each of those, and can get each of them wrong.

Someone still has to decide what the system may do, which failures are acceptable, and how the team will tell the difference. "The AI generated it" isn't an answer to any of that, and neither is "the tests that happened to exist passed."

Conclusion: "It works" is the start of review

SecureVibeBench is a useful reminder that functional correctness and security must be measured separately. It is also a reminder to read benchmarks carefully: this one covers C/C++ memory safety, 2025-era models, 105 tasks, and a noisy static scanner, and it supports "don't treat working code as secure code," not any sweeping claim about all AI-generated software.

None of this means AI-written code is inherently unsafe. Human-written code has always needed review, testing and security controls. What changes is the speed and volume at which plausible code appears, and the temptation to let that speed set the standard for acceptance.

My approach: let AI help write the code, make the requirements explicit, verify security properties independently, review the actual diff, and keep a human accountable for the result.

When an implementation takes minutes to generate, the discipline to verify it may be one of the most valuable engineering skills we have.

References

  1. Chen, J., Huang, H., Lyu, Y., An, J., Shi, J., Yang, C., Zhang, T., Tian, H., Li, Y., Li, Z., Zhou, X., Hu, X., Lo, D. SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios. Proceedings of ACL 2026, pp. 24144-24168. DOI: 10.18653/v1/2026.acl-long.1107
  2. OWASP. Secure Coding with AI Cheat Sheet
  3. OWASP. Secure Coding Practices Quick Reference Guide
  4. NIST. SP 800-218, Secure Software Development Framework (SSDF) Version 1.1 (February 2022)

SecureVibeBench figures in this article were checked against the published ACL Anthology paper on 24 October 2026.

About the Author

RAJश्री

Software Engineer · AI Researcher · Founder, Shree Labs

I’m Rajshree, a Software Engineer and AI-focused builder interested in the intersection of software engineering, artificial intelligence, machine learning, and modern web technologies.

I write about the engineering behind emerging technologies—not simply how to use an AI tool, but how these systems work, how they should be designed, and what it takes to build software that remains reliable, secure, and responsible as technology becomes more capable.

My writing often explores AI engineering, software architecture, cybersecurity, machine learning, developer responsibility, and the broader relationship between technology and society.

🏗️ Founder — Shree Labs

Shree Labs is a growing technology and knowledge platform founded by Rajshree.

It brings together:

  • Technical articles and engineering insights
  • Software engineering and programming tutorials
  • Technology and AI projects
  • Artificial intelligence and machine learning research
  • Development and learning resources
  • Poetry and creative writing

The idea is simple:

Learn. Build. Experiment. Share.

The broader vision is to create a space where technology, engineering, learning, research, and creativity can coexist.

Explore Shree Labs:

🌐 Shree Labs — https://rjshree.com/

💼 LinkedIn — https://linkedin.com/in/rjshree

💻 GitHub — https://github.com/itsrjshree

If you found this article useful, you can follow along for more writing on AI engineering, software architecture, machine learning, cybersecurity, modern web development, and the journey from writing code to engineering intelligent systems.

Thanks for reading.

© RAJश्री | Shree Labs

Top comments (0)