Your agent shipped the upload endpoint. The feature works, the demo works, the pull request is tidy. Somewhere inside it, the filename a user uploads is also a path, and nobody tested that.
A benchmark published on October 10, 2026 puts a number on how often that happens. The Last Secure Code Benchmark (LSCBench, arXiv:2610.14766) took 150 real vulnerabilities, expanded them into 450 tasks across seven languages, and handed them to six frontier coding agents. The prompts asked only for functionality and never mentioned security, which is how the writeups describe ordinary agent briefing. Each task was graded by two hidden suites: functional tests, and security tests that replay the original attack. No partial credit, no judge model, one attempt per task.
The agents passed 73.6% to 91.3% of tasks functionally. The best of them, Claude Fable 5, produced code that was both working and secure on 25.1% of tasks. The other five landed between 16.4% and 24.7%.
The row that should worry you
| Model | Harness | Works | Works and secure |
|---|---|---|---|
| Claude Fable 5 | Claude Code | 91.1% | 25.1% |
| GPT-5.6-Sol | Codex | 89.1% | 24.7% |
| Qwen3.8-Max | Claude Code | 89.3% | 22.0% |
| GLM-5.3 | Claude Code | 87.1% | 21.3% |
| DeepSeek-V4-Pro | Claude Code | 73.6% | 18.0% |
| Kimi-K3 | Claude Code | 91.3% | 16.4% |
Read the last row twice. Kimi-K3 has the highest functional pass rate in the group and the lowest secure rate. A green test suite told you nothing about which agent produced safer code.
The aggregates are worse than any single row. Across all 2,700 runs, 66% ended with code that works and is still exploitable. Of the 150 scenarios, 57 were never secured by any of the six models, not once. The paper's own summary is blunter: the best model passes only 12% of the vulnerability scenarios across all vibe-coding settings.
The safe API was one argument away
The failure mode is omission, not error. In the benchmark's Java case study, Kimi-K3 rebuilt a configuration importer, passed all 14 hidden functional tests, and failed all 4 security tests, because it reached for SnakeYAML's default constructor, which honors every class tag in untrusted YAML. Two other models picked SafeConstructor and passed both suites. The safe API was in the same library.
The breakdown by weakness family shows the same shape at scale. Memory-safety fixes came out best, 44% of working code also secure, because those fixes are local: bounds-check this, free that. Injection followed at 28%. Then a cliff: server-side request forgery at 3%, information exposure at 1%. Those two require the agent to model the system around the code. Where does this URL reach? Who else reads this config dump? Agents write the feature and leave the door open.
That is also why the reminder experiment matters. The authors reran four models with one sentence appended to every task: an implementation meeting the functional requirements "while also being secure". GPT-5.6-Sol went from 24.7% to 46.2% secure. GLM-5.3 went from 21.3% to 38.2%. Qwen3.8-Max went from 22.0% to 37.6%. DeepSeek-V4-Pro barely moved, 18.0% to 19.1%. One sentence nearly doubled secure output for three of four models, and the two system-level categories stayed at the bottom. Security, for agents, is a retrieval problem: the knowledge is in the weights, and the trigger is missing.
The second suite
LSCBench's whole trick is replaying the original exploit. You can do the same for agent-written code with three test patterns, one per case study in the paper. Each asserts the exploit did not happen, which is more robust than asserting a specific exception: refusing is fine, executing is not.
Pattern 1, the parser the agent reached for:
# The agent wrote this. Functional tests pass: configs load fine.
import yaml
def load_config(path):
with open(path) as f:
return yaml.load(f, Loader=yaml.Loader) # honors !!python/object tags
# The second suite plays the hostile user.
def test_load_config_does_not_execute_payloads(tmp_path):
marker = tmp_path / "pwned.txt"
hostile = tmp_path / "evil.yaml"
hostile.write_text("!!python/object/apply:os.system ['touch %s']" % marker)
try:
load_config(hostile)
except Exception:
pass
assert not marker.exists(), "config loader executed an embedded command"
The fix is yaml.safe_load(f), which refuses object tags. The test passes however the safe version refuses, and fails only if the payload ran.
Pattern 2, the shell string:
# The agent wrote this. Functional tests pass: the file gets converted.
import subprocess
def convert(src, dst):
subprocess.run("convert %s %s" % (src, dst), shell=True)
# The hostile user names the file: x; touch pwned.txt; echo .png
def test_convert_does_not_chain_commands(tmp_path):
marker = tmp_path / "pwned.txt"
src = tmp_path / ("x; touch %s; echo .png" % marker)
src.write_text("data")
try:
convert(str(src), str(tmp_path / "out.png"))
except Exception:
pass
assert not marker.exists(), "filename was interpreted as shell syntax"
The fix is an argument list with no shell: subprocess.run(["convert", src, dst]). Two of the six models in the paper's Java command-injection case built a shell string instead of an argument list.
Pattern 3, the path that escapes:
# The agent wrote this. Functional tests pass: named files are served.
import os
def read_named_file(base, name):
return open(os.path.join(base, name)).read()
# The hostile user asks for ../secret.txt, or hands over an absolute path.
def test_read_named_file_refuses_escaping_paths(tmp_path):
base = tmp_path / "files"
base.mkdir()
leaked = tmp_path / "secret.txt"
leaked.write_text("top secret")
for hostile in ["../secret.txt", str(leaked), "a/../../secret.txt"]:
try:
content = read_named_file(str(base), hostile)
except Exception:
continue # refusing is fine
assert "top secret" not in content, "read escaped the base: %r" % hostile
The fix resolves the real path and checks containment:
def read_named_file(base, name):
root = os.path.realpath(base)
target = os.path.realpath(os.path.join(root, name))
if target != root and not target.startswith(root + os.sep):
raise ValueError("path escapes the base directory")
return open(target).read()
Name the file test_attack_suite.py and run it on every diff an agent writes. The three tests are cheap, they fail loudly, and they are the only examiner in your pipeline that plays the hostile user.
The one sentence, and what it cannot do
The cheapest intervention in the paper is also the most copyable. Append one sentence to the task: an implementation meeting the functional requirements "while also being secure". The everyday equivalent is a line in the file your harness already reads at session start:
Implement the requested functionality while also being secure. Prefer the
safe API when a library offers one: safe parsers, argument lists instead
of shell strings, confined paths. If a requirement conflicts with
security, stop and ask instead of guessing.
Do not mistake it for a fix. The paper's verdict is explicit: most tasks stay insecure even with the reminder, and SSRF and information exposure stay near the bottom. The sentence raises the probability that the agent reaches for the safe API. It does not give the agent the system-level context those two categories require, and the experiment appended it to the task prompt, not to a standing instruction file.
What this does not fix
- The benchmark measures default behavior. The models were not given security instructions, a secure-coding skill, or a review loop. An agent with an explicit security requirement and an automated review pass will do better than these numbers. Read them as a floor for the default workflow, not a ceiling for a governed one.
- One attempt per task. Everything is pass@1. Retry loops, self-review, and verifier agents change the picture, and none of that is in these numbers.
- It is a preprint. One benchmark, one snapshot. Agent harnesses change monthly. Treat 25.1% as a signal about a mechanism, not an effect size to quote in a roadmap.
- A pass means one weakness was avoided. The security suite replays the specific attack each task was built from. It does not certify every other line.
- The second suite only covers weaknesses you thought to test. The three patterns above catch the classes LSCBench found failing. They will not catch the class nobody wrote a test for. That gap is what review is for.
Start here
Pick the last module your agent wrote that takes untrusted input: a config parser, a filename, a URL, a shell command. Write the three tests above against it and run them before the feature tests. Expect at least one to fail. Then fix the code, not the test.
If you do only one thing today, add the security sentence to your AGENTS.md or CLAUDE.md. It costs one line, and the data says it moves the number for most models. Then book an afternoon for the second suite, because the sentence is a retrieval aid and the suite is the control.
Sources:
- Tianyi Wu et al., "Last Secure Code Benchmark: Is the Code You Vibe Secure?", arXiv:2610.14766, submitted October 10, 2026: https://arxiv.org/abs/2610.14766
- Gabriel Anhaia, "Coding Agents Pass Up to 91% of Tasks in a New Benchmark. At Most 25% Are Also Secure", dev.to, October 15, 2026: https://dev.to/gabrielanhaia/coding-agents-pass-up-to-91-of-tasks-in-a-new-benchmark-at-most-25-are-also-secure-1oib
- Daniel Sam Pete Thiyagu, "91% Passed, 25% Safe: Why Your Coding Agent Ships Code That Works and Gets You Hacked", dev.to, October 16, 2026: https://dev.to/danielsamfdo/91-passed-25-safe-why-your-coding-agent-ships-code-that-works-and-gets-you-hacked-5111
The headline figures (450 tasks, 150 vulnerabilities, 73.6-91.3% functional, 25.1% best secure rate, 12% scenario rate) are from the arXiv abstract. The per-model leaderboard, the 66% and 57-scenario aggregates, the weakness-family breakdown, and the reminder rerun numbers come from the project page and paper as relayed through the two writeups above; the project page itself was not opened directly. The reminder was tested appended to the task prompt, not as a standing instruction file.
This post was written with AI assistance. The author is responsible for its content.
Top comments (0)