DeepSeek V4.1 Flash achieved verified server-side command execution in all 11 vulnerable runs of Enclave's AI Hacking Race, while all four patched controls stayed unverified. The result matters because it measures a complete exploit loop rather than a model merely describing a vulnerability, but Enclave's own trace audit shows why a perfect score is not the same as a general claim about real-world hacking ability.
Key facts
- Enclave published the result on September 16, 2026.
- V4.1 Flash completed 11 of 11 vulnerable runs across modified Grafana, Jenkins and Nextcloud targets.
- Four patched controls remained unverified.
- The primary source is Enclave's benchmark audit.
The race gave each agent source code, a low-privilege account, Bash, high reasoning and the same time rules. A pass required the target to execute a command and submit a fresh run-specific value to a separate verification service. This distinction is central. A security report can be plausible and still not prove that an agent navigated authentication, application behavior and exploitation. Enclave required the equivalent of a stamped receipt from the target.
The four task families were a Grafana plugin import, a Jenkins command-line parser, a Jenkins upload race and Nextcloud shared-file access control. Enclave's leaderboard and methodology records 11 verified vulnerable outcomes and four patched controls. V4.1 Flash led the listed systems in this particular race, ahead of GPT 5.6 Sol's 9 of 11 and GLM 5.3's 8 of 11. Enclave explicitly warns that the small race does not measure overall model quality or stock-product security.
The more interesting part arrived after scoring. Enclave inspected the successful traces and found six intended paths—three Jenkins parser attacks, one Jenkins upload race and two Nextcloud access-control routes. Five successes used alternate paths: the three Grafana runs used a shorter plugin-folder route and two Jenkins-upload runs used shorter file-link routes. Enclave says these were properties of its private challenge variants, not newly discovered upstream vulnerabilities. It repaired the benchmark and says future comparisons must use matched versions.
That audit turns a catchy result into a practical security lesson. An attacker wants any working route; a benchmark author may want to measure a particular route. If the evaluator records only the final flag, it can over-credit the intended reasoning. The model's behavior is still capability-relevant—it searched for a shortcut—but a defender needs a path-level account to understand which control actually failed.
DeepSeek's official model card says V4.1 Flash is MIT-licensed and describes a 552B-parameter multimodal mixture-of-experts model with a million-token context window. The model card is a release source, not a consumer hardware guide: no verified disk-size or runtime-VRAM figure should be inferred from the parameter count. DeepSeek's release page describes a cache-first design intended for long-context workloads, but that architecture alone does not explain the benchmark result.
Enclave calls it its “best hacking model,” a claim whose scope is exactly the AI Hacking Race. The strongest counterargument is that four modified targets, 11 vulnerable runs and one agent harness cannot represent the distribution of real enterprise systems. Enclave is also a commercial autonomous-security company, so its benchmark is useful evidence rather than neutral certification.
The so-what is still direct. AI-cyber evaluations should require verified effects, patched controls, trace review and versioned target environments. For defenders, the result raises the priority of testing whether tool-using models can chain source reading, hypothesis generation and live actions. For evaluators, Enclave's self-correction is the story to emulate: a score is only trustworthy if the route behind it is inspectable.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)