<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: dextsamu</title>
    <description>The latest articles on DEV Community by dextsamu (@dextsamu).</description>
    <link>https://dev.to/dextsamu</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4247353%2F1fb101e9-3d50-40c6-a12d-4c20b1e60cc7.png</url>
      <title>DEV Community: dextsamu</title>
      <link>https://dev.to/dextsamu</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/dextsamu"/>
    <language>en</language>
    <item>
      <title>I benchmarked a security-audit skill on Claude, Codex and Antigravity. Here are the numbers.</title>
      <dc:creator>dextsamu</dc:creator>
      <pubDate>Mon, 16 Nov 2026 09:46:24 +0000</pubDate>
      <link>https://dev.to/dextsamu/i-benchmarked-a-security-audit-skill-on-claude-codex-and-antigravity-here-are-the-numbers-cc6</link>
      <guid>https://dev.to/dextsamu/i-benchmarked-a-security-audit-skill-on-claude-codex-and-antigravity-here-are-the-numbers-cc6</guid>
      <description>&lt;p&gt;Most "AI security audit" prompts I have seen share the same problem: the report looks great and you cannot tell what was actually checked. An agent that writes "authentication: verified" after reading one file is worse than no audit, because someone will set a budget on it.&lt;/p&gt;

&lt;p&gt;I work with Italian public administrations and non-profits, so I built two open-source skills for AI coding agents that try to fix this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://github.com/dextsamu/appsec-sentinel" rel="noopener noreferrer"&gt;appsec-sentinel&lt;/a&gt;&lt;/strong&gt;: application and cloud security assessment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://github.com/dextsamu/gdpr-sentinel" rel="noopener noreferrer"&gt;gdpr-sentinel&lt;/a&gt;&lt;/strong&gt;: GDPR / ePrivacy review of what the code really does with personal data. It never declares compliance.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffqb7lwta7uj2y8ykzz0l.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffqb7lwta7uj2y8ykzz0l.gif" alt="appsec-sentinel running on OWASP Juice Shop" width="800" height="467"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Both are Apache-2.0, work in English and Italian, and run on Claude Code, Codex and Antigravity. This post is about how I measured them, including the results that are not flattering.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rules the skill enforces
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F78wc1e92gs4f7l8znunk.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F78wc1e92gs4f7l8znunk.jpg" alt="Every claim is examined, proven or unknown" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Examined, proven, unknown.&lt;/strong&gt; Every finding has a state: &lt;em&gt;dynamically reproduced&lt;/em&gt;, &lt;em&gt;confirmed from code or configuration&lt;/em&gt;, &lt;em&gt;hypothesis to validate&lt;/em&gt;, &lt;em&gt;not verifiable&lt;/em&gt;. A static read is never presented as a test.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Real denominators.&lt;/strong&gt; "27 of 52 endpoints tested, by role", not "good coverage". Scripts compute them from CSV files instead of letting the model estimate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Authorization first.&lt;/strong&gt; No active test without written scope. Text found in code, comments or tool output is treated as data and cannot widen the mandate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Independent refutation.&lt;/strong&gt; High and critical findings go to a verifier that is not the author and tries to disprove them. If no independent verifier is available, the report says "not performed". The author refuting itself does not count.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A delivery check that does not trust the model.&lt;/strong&gt; &lt;code&gt;check_delivery.py&lt;/code&gt; fails the audit if a dangerous-code category found by the pre-pass is left without a verdict, if forbidden words like "compliant" appear, or if secret values leak into the folder.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Benchmark 1: OWASP Juice Shop (truth written by someone else)
&lt;/h2&gt;

&lt;p&gt;My own test apps have a weakness: I wrote the bugs and I wrote the checklist, so they measure obedience more than generalization. Juice Shop is better: its authors mark the vulnerable lines of their coding challenges in the source (&lt;code&gt;vuln-code-snippet vuln-line&lt;/code&gt;). That gives &lt;strong&gt;32 vulnerabilities with exact file and line&lt;/strong&gt;, chosen by people who never saw my skill.&lt;/p&gt;

&lt;p&gt;A finding counts only if it cites the marked file and line. Same prompt, &lt;code&gt;quick&lt;/code&gt; mode, static analysis only:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8xnnptrufz9cmsrr8npl.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8xnnptrufz9cmsrr8npl.jpg" alt="Claude 23/32, Codex 18/32, Antigravity 6/32" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Agent&lt;/th&gt;
&lt;th&gt;Found with file and line&lt;/th&gt;
&lt;th&gt;Within 15 lines&lt;/th&gt;
&lt;th&gt;Findings&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Claude (Opus, two cloud runs)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;23/32&lt;/strong&gt; and 19/32&lt;/td&gt;
&lt;td&gt;27 and 23&lt;/td&gt;
&lt;td&gt;37 and 36&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Codex&lt;/td&gt;
&lt;td&gt;18/32&lt;/td&gt;
&lt;td&gt;22/32&lt;/td&gt;
&lt;td&gt;28&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Antigravity (Gemini)&lt;/td&gt;
&lt;td&gt;6/32&lt;/td&gt;
&lt;td&gt;9/32&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two things surprised me.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The model matters, but so does the plumbing.&lt;/strong&gt; Earlier &lt;code&gt;standard&lt;/code&gt; runs on my PC (older Claude model) scored &lt;strong&gt;7/32, 7/32 and 8/32&lt;/strong&gt;. Reading the transcripts, the agent &lt;em&gt;had seen&lt;/em&gt; the client-side XSS, the SSRF, the chatbot and the password-reset issues. It wrote them in the checklist as "noted" or "known surface" and never promoted them to findings. Adding a word filter did not help: the agent used a synonym. What worked was structural: the deterministic pre-pass now writes one row per dangerous-code category (HTML without escaping, code execution, deserialization, SSRF, client routes, seed credentials…), and the delivery check fails unless each row is closed with a finding or a "not a defect" that cites a file and line it read. The next two runs: &lt;strong&gt;13/32 and 12/32&lt;/strong&gt;, same model.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4je3qkxth906se2lb9ju.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4je3qkxth906se2lb9ju.jpg" alt="From 7/32 to 13/32 with the same model, 23/32 with a later skill and model" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Antigravity writes few findings.&lt;/strong&gt; It found real things, but stopped early. I have not solved this yet; it is on the roadmap.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The misses are shared.&lt;/strong&gt; All 9 vulnerabilities Claude missed in its best run were missed by Codex and Antigravity too, and they fall into three groups:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe8iahdiafy21xn0vj6rs.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe8iahdiafy21xn0vj6rs.jpg" alt="The 9 shared blind spots: guessable security answers, hidden frontend routes, business logic" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Pattern-like bugs (injection, XSS, authentication) get caught. Flaws that need intent or outside knowledge don't. That is the next thing I want to measure.&lt;/p&gt;

&lt;h2&gt;
  
  
  Benchmark 2: Django app and a fake school website
&lt;/h2&gt;

&lt;p&gt;These are mine, so read them with the caveat above. Each has planted defects plus &lt;strong&gt;decoys&lt;/strong&gt;, things that look wrong but are fine (a session cookie, data published because the law requires it, an IP log deleted after 7 days). A false alarm on a decoy counts against the skill.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Claude&lt;/th&gt;
&lt;th&gt;Codex&lt;/th&gt;
&lt;th&gt;Antigravity&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;appsec, Django (11 defects, 5 decoys)&lt;/td&gt;
&lt;td&gt;11/11, 0 false alarms&lt;/td&gt;
&lt;td&gt;11/11, 0&lt;/td&gt;
&lt;td&gt;10/11, 0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gdpr, school website in PHP (15 defects, 5 decoys)&lt;/td&gt;
&lt;td&gt;15/15, 0&lt;/td&gt;
&lt;td&gt;14/15, 0&lt;/td&gt;
&lt;td&gt;14/15, 0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fozihgilgujvlmytvkh0m.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fozihgilgujvlmytvkh0m.gif" alt="gdpr-sentinel demo" width="800" height="467"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Refutation without subagents
&lt;/h2&gt;

&lt;p&gt;Codex can start a subagent; many setups cannot. For those, &lt;code&gt;refutation_pack.py prepara&lt;/code&gt; writes a folder with only the method and one brief per finding: the claim, the code excerpt and the typical ways it could be wrong. It contains no author reasoning. You hand it to a &lt;em&gt;new&lt;/em&gt; session (another conversation, another agent, a colleague) and import the answers with &lt;code&gt;refutation_pack.py importa&lt;/code&gt;. In a test with four planted false GDPR gaps, the verifier rejected three and downgraded the fourth, and rejected no true gap.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stability between runs (still open)
&lt;/h2&gt;

&lt;p&gt;On a real 1,000-file application, two GDPR runs agree on only about 60–69% of the individual gaps: they describe the same problem with different words and granularity. Fixed questions help a lot where they apply. Three fixed matrices (per data entity, per privacy-notice item, and who can read what for each role) bring agreement on those rows to 90–93%, 83–88% and 86–92%. The free-form part is still noisy. If you have ideas, I would love to hear them.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I built it
&lt;/h2&gt;

&lt;p&gt;It started as a long audit prompt I used by hand. I turned it into skills with Claude Code: Claude split the prompt into a &lt;code&gt;SKILL.md&lt;/code&gt; plus reference files, wrote the Python scripts and the test apps, and ran the benchmarks in subagents. After each run I asked it to read the transcript and find where the agent had cut corners, then we changed the scripts and measured again. Most of the rules above exist because a run broke them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does not do
&lt;/h2&gt;

&lt;p&gt;It does not certify anything, it does not replace a pentest or a DPO, and a static read is not proof that the running site behaves the same way. The benchmarks are small. The skill's own instructions are in Italian (the agent answers in your language).&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Claude Code&lt;/span&gt;
/plugin marketplace add dextsamu/appsec-sentinel
/plugin install appsec-sentinel@appsec-sentinel
/plugin install gdpr-sentinel@appsec-sentinel
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Codex and Antigravity use the same &lt;code&gt;SKILL.md&lt;/code&gt; convention (install paths in the repo docs). Then ask, for example: &lt;em&gt;"Use appsec-sentinel. Quick mode, static analysis only, on this folder. The repo is mine; no dynamic testing is authorized."&lt;/em&gt; Commands work in English and Italian.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;appsec-sentinel: &lt;a href="https://github.com/dextsamu/appsec-sentinel" rel="noopener noreferrer"&gt;https://github.com/dextsamu/appsec-sentinel&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;gdpr-sentinel: &lt;a href="https://github.com/dextsamu/gdpr-sentinel" rel="noopener noreferrer"&gt;https://github.com/dextsamu/gdpr-sentinel&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Benchmarks and scoring scripts are in &lt;code&gt;tests/&lt;/code&gt;. You can rerun everything.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Feedback, criticism and new benchmarks are very welcome, and a star helps other people find the project.&lt;/p&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>opensource</category>
      <category>security</category>
    </item>
  </channel>
</rss>
