<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Sunny Chokshi</title>
    <description>The latest articles on DEV Community by Sunny Chokshi (@sunnychokshi).</description>
    <link>https://dev.to/sunnychokshi</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4213445%2F1d1e097a-1d0d-4e9d-9239-5fc0f59c3e59.png</url>
      <title>DEV Community: Sunny Chokshi</title>
      <link>https://dev.to/sunnychokshi</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/sunnychokshi"/>
    <language>en</language>
    <item>
      <title>LLM benchmark stimulus audit: 33 of 80 pairs not clean</title>
      <dc:creator>Sunny Chokshi</dc:creator>
      <pubDate>Fri, 13 Nov 2026 15:15:59 +0000</pubDate>
      <link>https://dev.to/sunnychokshi/llm-benchmark-stimulus-audit-33-of-80-pairs-not-clean-1i5k</link>
      <guid>https://dev.to/sunnychokshi/llm-benchmark-stimulus-audit-33-of-80-pairs-not-clean-1i5k</guid>
      <description>&lt;p&gt;Sunny Chokshi. November 6, 2026.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0g7wi2y2ilpvvgo2dk9l.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0g7wi2y2ilpvvgo2dk9l.png" alt="LLM benchmark stimulus audit: 33 of 80 snippet pairs were not clean matches. On the 47 clean pairs, seven strongest models: balanced accuracy down 5.3 points, false positives up 9.1 points. 14,592 model calls, 16 open models." width="800" height="419"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;After my runs finished I did an LLM benchmark stimulus audit on my own matched pairs. Of the 80 pairs that have both a plain and an indirect form, 33 differ in more than the route to the misuse, and 47 are clean matches. This post shows what the audit file lists, what happens to the results when I keep only the clean pairs, and how to recompute the numbers in a couple of minutes. You can download the data below.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23747614" rel="noopener noreferrer"&gt;Download the data on Zenodo&lt;/a&gt; | &lt;a href="https://doi.org/10.5281/zenodo.23351769" rel="noopener noreferrer"&gt;Get the preprint on Zenodo&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a matched pair benchmark needs an audit
&lt;/h2&gt;

&lt;p&gt;Each pair holds a vulnerable snippet and a secure twin, written plainly and indirectly. The point is that only the way the misuse is written changes. If anything else changes between the forms, a score difference may not be due to the indirection alone. The audit came after the runs, and that order is a limit I return to below.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the LLM benchmark stimulus audit found
&lt;/h2&gt;

&lt;p&gt;The audit file records one primary reason per pair. By pair count: the API changes on the vulnerable side between forms (6), the plain form keeps an unused leftover of the indirect form (5), the indirect form states the safe value while the plain form omits it (5), the misuse depends on an environment variable (4), importlib is added in the indirect form (4), the secure twin changes API between forms (4), the secure twin contains the bad token (2), and other (3). That sums to 33. The unmatched group has five pairs in the audit file (cv01, cv02, cv04, cv05 and ns01). Two pairs also list a second reason, so the full list is in &lt;code&gt;audit/audit_pairs.csv&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Do the results survive the audit?
&lt;/h2&gt;

&lt;p&gt;Over all 80 pairs, balanced accuracy of the seven strongest models falls 4.5 points when the misuse is written indirectly (95% CI 1.4 to 7.6 in the release script, 1.3 to 7.6 in the paper, because the bootstrap seed differs). On the 47 clean pairs it falls 5.3 points (CI 2.1 to 8.5).&lt;/p&gt;

&lt;p&gt;The audit changes where I think the drop comes from. On the clean pairs the false positive rate rises 9.1 points (CI 4.2 to 14.6), while detection falls 1.5 points and its interval includes zero (CI -1.6 to 4.8). Over all 80 pairs detection falls 3.6 points (CI 1.0 to 6.5), and the false positive rise is 5.4 points with an interval that includes zero (CI -0.1 to 10.8). Leaving out only the four unmatched certificate pairs (cv01, cv02, cv04 and cv05, which keeps ns01) raises the false positive rise from 5.4 to 8.0 points (CI 3.1 to 13.2), and the clean set gives 9.1 points with an interval that excludes zero.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rerun it yourself
&lt;/h2&gt;

&lt;p&gt;The record holds 14,592 model calls (11,232 in the main log and 3,360 in the paired replication) from 16 open models with 3 repeats each, run locally at temperature 0. You need Python 3 and nothing else. Unzip the release and run &lt;code&gt;python3 analysis/v6.py .&lt;/code&gt; from the unzipped folder. It prints the numbers above, except the 1.3 to 7.6 interval, which comes from the paper's own script with a different seed. Every script reproduces its saved output byte for byte on the files in the record. Data is CC BY 4.0 and code is MIT. In the logs the &lt;code&gt;variant&lt;/code&gt; field says obscured where the paper says indirect.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;The audit came after the runs. The two exploratory splits in the paper were chosen after I saw the data, and the seven models in the paired study were chosen after the first study, so the confidence intervals treat them as fixed. Detection counts a vulnerable verdict given for any reason, and I did not check whether each detection named the right misuse. The snippets and labels were written with AI assistance and checked by me, so labeling errors are possible. The Ollama version was not recorded, and reruns of the model calls can differ by a few points even at temperature 0. The paper is a preprint, it is not peer reviewed, and it carries the notice that it has been submitted to the IEEE for possible publication.&lt;/p&gt;

&lt;p&gt;Open question: would the audit have kept a different set of pairs if I had done it before the runs? The data cannot answer that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read and cite
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23747614" rel="noopener noreferrer"&gt;Data package on Zenodo (doi:10.5281/zenodo.23747614)&lt;/a&gt; · &lt;a href="https://doi.org/10.5281/zenodo.23351769" rel="noopener noreferrer"&gt;Preprint on Zenodo (doi:10.5281/zenodo.23351769)&lt;/a&gt; · &lt;a href="https://sunny-chokshi.github.io/papers/does-indirection-hurt.html" rel="noopener noreferrer"&gt;Paper page and PDF&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Earlier post from the same study: &lt;a href="https://sunny-chokshi.github.io/blog/hex-key-size-false-alarms.html" rel="noopener noreferrer"&gt;A hex literal made LLM code reviewers raise false alarms&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Cite as: Chokshi, S. (2026). Does Indirection Hurt? Stimuli, per-call results, audit list and analysis scripts (v1.0.0) [Data set]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23747614" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23747614&lt;/a&gt;&lt;br&gt;
Preprint: Chokshi, S. (2026). Does Indirection Hurt? Paired Evidence on LLM Review of Indirectly Written Cryptographic API Misuse [Preprint]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23351769" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23351769&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. &lt;a href="https://orcid.org/0009-0003-4738-7759" rel="noopener noreferrer"&gt;ORCID&lt;/a&gt; · &lt;a href="https://github.com/sunny-chokshi" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>llm</category>
      <category>python</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>Adversarial code comments dataset for LLM reviewers</title>
      <dc:creator>Sunny Chokshi</dc:creator>
      <pubDate>Fri, 13 Nov 2026 15:15:27 +0000</pubDate>
      <link>https://dev.to/sunnychokshi/adversarial-code-comments-dataset-for-llm-reviewers-4o0h</link>
      <guid>https://dev.to/sunnychokshi/adversarial-code-comments-dataset-for-llm-reviewers-4o0h</guid>
      <description>&lt;p&gt;Sunny Chokshi. November 4, 2026.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpfx7ig98m5oxs9aag5b8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpfx7ig98m5oxs9aag5b8.png" alt="One comment changed what LLM code reviewers said about the same vulnerable code. CommentInject: 30 vulnerable functions, 14 CWE categories, 12 comment strategies, 3,360 recorded trials from four studies, on two to four local models." width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I wanted to know whether a comment can change what an LLM code reviewer says about the same code. So I built a small adversarial code comments dataset, CommentInject. The vulnerable function stays identical. Only a comment is added, and the comment claims the code is safe.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23088528" rel="noopener noreferrer"&gt;Get the dataset on Zenodo&lt;/a&gt; | &lt;a href="https://huggingface.co/datasets/sunny-chokshi/commentinject" rel="noopener noreferrer"&gt;Download the data on Hugging Face&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What is in the dataset
&lt;/h2&gt;

&lt;p&gt;There are 30 vulnerable Python functions across 14 CWE categories. CWE-89 (SQL injection) has five functions, and the other thirteen categories have one to three each. Every function is vulnerable. There are 12 comment strategies in four families of three: verification premise, context manipulation, credibility vocabulary, and pressure and distraction. One example is a comment saying the function passed penetration testing. All credentials in the samples are fake.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four studies, 3,360 trials
&lt;/h2&gt;

&lt;p&gt;The trials come from four studies run locally through Ollama at temperature 0. Studies 1 and 2 used qwen2.5-coder 7B and deepseek-coder 6.7B. Studies 3 and 4 added codellama 7B and codegemma 7B.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Attack ranking:            780 trials
Comment-stripping defense: 1,020 trials
Cross-model comparison:    600 trials
Transferability:           960 trials
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What the data shows
&lt;/h2&gt;

&lt;p&gt;In Study 2, clean code was flagged in 90.0% of reviews. With a misleading comment it fell to 75.2%, and with the comments stripped before review it returned to 90.0%. That study used two models.&lt;/p&gt;

&lt;p&gt;In Study 3, four models reviewed code under four comment strategies. Detection ranged from 17.5% for codegemma 7B (21 of 120 reviews) to 93.3% for qwen2.5-coder 7B (112 of 120). codellama and deepseek both reached 62.5% (75 of 120). A low number does not prove a model was fooled. The clean baseline of codegemma was already the lowest, 23 of 30 (76.7%), and every sample is vulnerable, so I have no safe controls to tell a fooled model from one that says "safe" often.&lt;/p&gt;

&lt;p&gt;The recorded data shows one verdict flip. deepseek-coder 6.7B finds the path-traversal function (V04) when it is clean, misses it with the authority-claim comment, and finds it again once the comments are stripped. That is one function and one model, so treat it as an illustration, not a result.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reuse the comment injection data
&lt;/h2&gt;

&lt;p&gt;The data loads from Hugging Face with the following call. The other configs are the strategies and one file per study, so you can test your own model and compare against the recorded verdicts. The harnesses are on &lt;a href="https://github.com/sunny-chokshi/commentinject" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datasets&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;
&lt;span class="n"&gt;samples&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;load_dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"sunny-chokshi/commentinject"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"samples"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"train"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;The models are 6.7B to 7B open models. All 30 samples are vulnerable, so the data cannot show whether a comment also changes verdicts on safe code. Each function, model and condition ran once, with no repeats. A VULNERABLE verdict counts as a detection even if the model names a different flaw. Comment stripping was tested with two models.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read and cite
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23088528" rel="noopener noreferrer"&gt;CommentInject dataset on Zenodo (doi:10.5281/zenodo.23088528)&lt;/a&gt; · &lt;a href="https://sunny-chokshi.github.io/papers/not-all-adversarial-comments.html" rel="noopener noreferrer"&gt;Paper page: Not All Adversarial Comments Are Equal&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Earlier post in the same line: &lt;a href="https://sunny-chokshi.github.io/blog/cryptobench-what-is-inside.html" rel="noopener noreferrer"&gt;CryptoBench: 54 snippets to test LLM crypto misuse detection&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Cite as: Chokshi, S. (2026). CommentInject: A benchmark of adversarial code-comment injection against LLM vulnerability detectors [Dataset]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23088528" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23088528&lt;/a&gt; (v1.0.0; all versions: &lt;a href="https://doi.org/10.5281/zenodo.23088527" rel="noopener noreferrer"&gt;10.5281/zenodo.23088527&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. &lt;a href="https://orcid.org/0009-0003-4738-7759" rel="noopener noreferrer"&gt;ORCID&lt;/a&gt; · &lt;a href="https://github.com/sunny-chokshi" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>llm</category>
      <category>python</category>
      <category>datasets</category>
    </item>
    <item>
      <title>LLM weak random number generator detection test</title>
      <dc:creator>Sunny Chokshi</dc:creator>
      <pubDate>Mon, 02 Nov 2026 16:26:45 +0000</pubDate>
      <link>https://dev.to/sunnychokshi/llm-weak-random-number-generator-detection-test-4mn4</link>
      <guid>https://dev.to/sunnychokshi/llm-weak-random-number-generator-detection-test-4mn4</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2nobruq9wyxl1nhma6wi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2nobruq9wyxl1nhma6wi.png" alt="15, 7, 1, 0: how often three local LLMs flagged four Python snippets with the same weak random number generator bug, out of 15 verdicts each." width="800" height="418"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://sunny-chokshi.github.io/blog/llm-weak-random-number-generator-detection.html" rel="noopener noreferrer"&gt;sunny-chokshi.github.io&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I ran an LLM weak random number generator detection test on four Python snippets that make the same mistake. Each one builds a value an attacker must not be able to guess, using the random module instead of a cryptographic source. Three local code models read each snippet 15 times between them. One snippet was flagged 15 times, one 7 times, one once and one never. This post shows the four snippets and what the spread tells me about how these models review code.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23113861" rel="noopener noreferrer"&gt;Read the preprint on Zenodo&lt;/a&gt; | &lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;Download the data (CryptoBench, Zenodo)&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Download the data: the dataset holds every snippet and every recorded verdict, and its snippets, harness and results are byte-identical to the v1.0.0 the preprint cites, so you can recount the numbers below.&lt;/p&gt;

&lt;h2&gt;
  
  
  The four snippets
&lt;/h2&gt;

&lt;p&gt;All four are in the weak PRNG class of CryptoBench, which I mapped to CWE-338. Each does a different job with the random module. These are excerpts, without the imports and function wrappers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;pr_seed&lt;/span&gt;   &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;seed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getrandbits&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;pr_otp&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;randint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;999999&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;pr_reset&lt;/span&gt;  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;string&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ascii_letters&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;pr_token&lt;/span&gt;  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;choice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;0123456789abcdef&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The class also has two secure controls that do the same kinds of jobs with the secrets module, a token and a reset string. In Python, random vs secrets is the choice the controls get right.&lt;/p&gt;

&lt;h2&gt;
  
  
  LLM weak random number generator detection, snippet by snippet
&lt;/h2&gt;

&lt;p&gt;I pooled the three models in my study that separate safe code from unsafe code: qwen2.5-coder 7B, qwen2.5-coder 14B and deepseek-coder 6.7B. Each model saw each snippet five times, so every snippet has 15 verdicts. The five repeats of one snippet are not five independent examples.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pr_seed    15 of 15 flagged
pr_otp      7 of 15
pr_reset    1 of 15
pr_token    0 of 15
controls    0 of 30
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pooled, that is 23 of 60, or 38%. The two secure controls were never flagged. So this class has a low detection rate and no false alarms.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I think the spread means
&lt;/h2&gt;

&lt;p&gt;Only one of the four, random.seed(time.time()), calls a seed function with the clock. The other three use randint, choices and choice, which look like ordinary calls to me. The OTP, the reset string and the hex token all need an unpredictable value, and the models flagged them 7, 1 and 0 times out of 15. I cannot say why the OTP did better than the reset string.&lt;/p&gt;

&lt;p&gt;My reading is that the models respond to a name they have seen labelled as dangerous more than to the job the value does. That is a hypothesis. I have not tested it, and the preprint says so too.&lt;/p&gt;

&lt;p&gt;Size did not fix it in my data. Across the four snippets, qwen 7B flagged 6 of 20 trials and qwen 14B flagged 11 of 20. That difference is not statistically significant (Fisher exact test, p = 0.20), and 20 trials per model cannot rule out a modest gain.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;The class has four vulnerable snippets, I wrote them, and they are short single-file Python. The models are local, quantized and run at Ollama defaults with one fixed prompt, and the score is the verdict only. The paper expects that a short linter rule for the random module near token code would do better than the model here, but I have not tested that. The paper is a preprint and it is not peer reviewed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read and cite
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23113861" rel="noopener noreferrer"&gt;Preprint on Zenodo (doi:10.5281/zenodo.23113861)&lt;/a&gt; | &lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;CryptoBench dataset (doi:10.5281/zenodo.23067051)&lt;/a&gt; | &lt;a href="https://sunny-chokshi.github.io/index.html#papers" rel="noopener noreferrer"&gt;Paper listing on my site&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Earlier post in the same line: &lt;a href="https://sunny-chokshi.github.io/blog/cryptobench-what-is-inside.html" rel="noopener noreferrer"&gt;CryptoBench: 54 snippets to test LLM crypto misuse detection&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Cite as: Chokshi, S. (2026). Known-Bad Names, Unknown-Bad Uses: What Local Code Models Detect When They Review Cryptographic API Misuse [Preprint]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23113861" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23113861&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Dataset: Chokshi, S. (2026). CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers [Dataset]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23067051&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. &lt;a href="https://orcid.org/0009-0003-4738-7759" rel="noopener noreferrer"&gt;ORCID&lt;/a&gt; | &lt;a href="https://github.com/sunny-chokshi" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>python</category>
      <category>llm</category>
      <category>cryptography</category>
    </item>
    <item>
      <title>CryptoBench: 54 snippets to test LLM crypto misuse detection</title>
      <dc:creator>Sunny Chokshi</dc:creator>
      <pubDate>Fri, 30 Oct 2026 15:58:02 +0000</pubDate>
      <link>https://dev.to/sunnychokshi/cryptobench-54-snippets-to-test-llm-crypto-misuse-detection-4542</link>
      <guid>https://dev.to/sunnychokshi/cryptobench-54-snippets-to-test-llm-crypto-misuse-detection-4542</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftzblk2kjflg6mo29htop.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftzblk2kjflg6mo29htop.png" alt="CryptoBench: 54 snippets, 9 misuse classes, 1,890 recorded trials, 7 models. Weak PRNG was the lowest class at 38.3% detected." width="800" height="418"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;CryptoBench is a small dataset I built to test LLM code review of cryptographic API misuse. This post lists what is inside, what the first results looked like, and how to rerun it on your own model.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;Get the dataset on Zenodo&lt;/a&gt; | &lt;a href="https://doi.org/10.5281/zenodo.23113861" rel="noopener noreferrer"&gt;Read the paper on Zenodo&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What is inside
&lt;/h2&gt;

&lt;p&gt;CryptoBench holds 54 short Python snippets across nine misuse classes, each mapped to a CWE: weak password hash, ECB mode, hardcoded key or secret, static or reused IV, weak PRNG for security values, short asymmetric key, obsolete cipher, unsalted or fast KDF, and disabled certificate verification. Each class has four vulnerable snippets and two secure controls that do the same job correctly, so 36 vulnerable and 18 secure in total.&lt;/p&gt;

&lt;p&gt;The controls are the point. A model that flags everything looks good on detection alone, and the secure snippets are what expose it. The snippets use PyCryptodome, cryptography, hashlib, requests, ssl and PyJWT. All of them are synthetic: no real keys, no exploit code, nothing that touches a live system.&lt;/p&gt;

&lt;h2&gt;
  
  
  The protocol
&lt;/h2&gt;

&lt;p&gt;The model sees one snippet and a fixed instruction to answer VERDICT: VULNERABLE or VERDICT: SAFE, plus a one-sentence reason. Only the verdict is scored. I ran seven open models through Ollama at its default sampling settings: qwen2.5-coder at 0.5B, 1.5B, 3B, 7B and 14B, deepseek-coder 6.7B and codellama 7B. Each snippet ran five times per model, which gives 54 x 5 x 7 = 1,890 recorded trials.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the first run showed
&lt;/h2&gt;

&lt;p&gt;Four of the seven models flagged 68 to 93 percent of the secure snippets. Their detection rates mean little. Three models separate safe from unsafe code: qwen2.5-coder 7B, qwen2.5-coder 14B and deepseek-coder 6.7B.&lt;/p&gt;

&lt;p&gt;Pooling those three, detection ranged from 96.7% for disabled certificate verification down to 38.3% for weak PRNG. The top number needs its control. The same three models flagged 86.7% of the secure certificate snippets as vulnerable, so the class shows detection without discrimination. Weak password hash is the cleaner case, with 81.7% detected and 3.3% false alarms. Even these three flag 30.0 to 42.2% of secure snippets, and obsolete cipher (91.7% detected) carries 63.3% false alarms. Weak PRNG had no false alarms and the lowest detection. Going from the 7B to the 14B qwen model raised weak PRNG from 6 to 11 of 20, a difference that is not statistically significant.&lt;/p&gt;

&lt;p&gt;My reading is that models catch names they have seen flagged, such as verify=False, DES, MD5 and MODE_ECB, and miss an ordinary API used in the wrong place, such as random.choice for a token. I have not tested that directly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rerun it
&lt;/h2&gt;

&lt;p&gt;You need Python 3 and a running Ollama server. No pip installs.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ollama pull qwen2.5-coder:7b
python3 crypto_bench.py &lt;span class="nt"&gt;--model&lt;/span&gt; qwen2.5-coder:7b &lt;span class="nt"&gt;--repeats&lt;/span&gt; 5
python3 scripts/analyze.py crypto_results.csv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pass your own results file to the analysis script, as above, or it reads the shipped data. It writes the per-model and per-class tables with Wilson 95% intervals. The per-class table pools three fixed models, so it only reflects a rerun that includes them. The data also loads from Hugging Face as &lt;a href="https://huggingface.co/datasets/sunny-chokshi/cryptobench" rel="noopener noreferrer"&gt;sunny-chokshi/cryptobench&lt;/a&gt;, and the harness is on &lt;a href="https://github.com/sunny-chokshi/cryptobench" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;Each class has four vulnerable and two secure snippets, and five repeats of one snippet are not five independent examples, so lean on the class ordering and not on a single percentage. Everything is short, synthetic Python reviewed one snippet at a time. The models are seven local models up to 14B parameters, at default sampling and with one fixed prompt. The protocol scores the verdict only, so a VULNERABLE verdict counts even if the reason names a different flaw. The paper is a preprint and is not peer reviewed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read and cite
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;CryptoBench dataset on Zenodo (doi:10.5281/zenodo.23067051)&lt;/a&gt; | &lt;a href="https://doi.org/10.5281/zenodo.23113861" rel="noopener noreferrer"&gt;Paper on Zenodo (doi:10.5281/zenodo.23113861)&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Related: &lt;a href="https://sunny-chokshi.github.io/papers/does-indirection-hurt.html" rel="noopener noreferrer"&gt;Does Indirection Hurt?&lt;/a&gt; and the post &lt;a href="https://sunny-chokshi.github.io/blog/hex-key-size-false-alarms.html" rel="noopener noreferrer"&gt;A hex literal made LLM code reviewers raise false alarms&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Cite as: Chokshi, S. (2026). CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers [Dataset]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23067051&lt;/a&gt; (all versions)&lt;/p&gt;

&lt;p&gt;Paper: Chokshi, S. (2026). Known-Bad Names, Unknown-Bad Uses: What Local Code Models Detect When They Review Cryptographic API Misuse [Preprint]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23113861" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23113861&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. &lt;a href="https://orcid.org/0009-0003-4738-7759" rel="noopener noreferrer"&gt;ORCID&lt;/a&gt; | &lt;a href="https://github.com/sunny-chokshi" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>llm</category>
      <category>cryptography</category>
      <category>python</category>
    </item>
    <item>
      <title>Detection rate alone ranks the worst LLM code reviewers first</title>
      <dc:creator>Sunny Chokshi</dc:creator>
      <pubDate>Wed, 28 Oct 2026 19:31:01 +0000</pubDate>
      <link>https://dev.to/sunnychokshi/detection-rate-alone-ranks-the-worst-llm-code-reviewers-first-5825</link>
      <guid>https://dev.to/sunnychokshi/detection-rate-alone-ranks-the-worst-llm-code-reviewers-first-5825</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3owbpasnvmtupyysmqpg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3owbpasnvmtupyysmqpg.png" alt="100% detection rate can tell you little. qwen2.5-coder 0.5B flagged 540 of 540 snippets, balanced accuracy 0.500. gpt-oss 20B reached 0.952." width="800" height="418"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;When I tested 16 open LLM code reviewers, the first table in my paper showed which of them separate secure code from vulnerable code at all. It is worth reading on its own.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;I used 180 Python snippets across nine cryptographic misuse classes, ten pairs per class. Each pair has a vulnerable snippet and a secure twin that does the same task correctly. I ran 16 open models, from 0.5B to 120B parameters, with three repeats per snippet, which gives 8,640 scored calls. For each model I report the true positive rate (TPR, vulnerable snippets flagged), the false positive rate (FPR, secure twins flagged), and balanced accuracy, the average of TPR and one minus FPR. A model that flags everything scores 0.5.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why detection rate misleads
&lt;/h2&gt;

&lt;p&gt;Judged on TPR alone, codellama 7B, devstral 24B, and qwen2.5-coder 0.5B would rank near the top, each at or near 100%. Their FPRs tell a different story. codellama 7B and qwen2.5-coder 0.5B flagged 539 and 540 of 540 scored calls. devstral 24B flagged 82.6% of the secure twins. Three more models (codellama 34B, qwen2.5-coder 1.5B and 3B) also flagged more than 70% of secure code. The 0.5B model also left out the requested verdict line in all but 37 of its 540 replies.&lt;/p&gt;

&lt;p&gt;With the secure twins in the table, the order changes. Of the 16 models, gpt-oss 20B had the highest balanced accuracy, 0.952, with a TPR of 93.7% and an FPR of 3.3%. Ranked by TPR alone, these models would sit at or near the top.&lt;/p&gt;

&lt;h2&gt;
  
  
  Does model size help?
&lt;/h2&gt;

&lt;p&gt;Size and score go together to some degree, but size does not settle the ranking. Over all 16 models, parameter count and balanced accuracy are related (Spearman 0.76). Across families it did not order the results: codellama 34B (0.609) and devstral 24B (0.587) were among the weakest of the larger models. Even within gpt-oss, the 120B model (0.931) scored below the 20B (0.952).&lt;/p&gt;

&lt;p&gt;What I take from this test: a detection rate on its own rewarded models that said "vulnerable" almost every time. I would report the false positive rate next to it and use secure twins that match the vulnerable code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;The snippets are short Python functions, reviewed one at a time, so the result speaks to a single-snippet review pass. The models are open models run with temperature 0. I tested only open-weight models and only Python, so I make no claim about closed models or other languages. TPR counts a vulnerable verdict given for any reason, and I did not check whether each detection named the right misuse. This is a preprint, it is not peer reviewed, and it carries the notice that it has been submitted to the IEEE for possible publication.&lt;/p&gt;

&lt;p&gt;The same paper goes on to test what happens when the same misuse is written indirectly. I covered one result from that part in my &lt;a href="https://sunny-chokshi.github.io/blog/hex-key-size-false-alarms.html" rel="noopener noreferrer"&gt;earlier post on a hex key size&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read and cite
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23351769" rel="noopener noreferrer"&gt;Preprint on Zenodo (doi:10.5281/zenodo.23351769)&lt;/a&gt; · &lt;a href="https://sunny-chokshi.github.io/papers/does-indirection-hurt.html" rel="noopener noreferrer"&gt;Paper page and PDF&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Related work in the same line, an earlier benchmark: &lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;CryptoBench (doi:10.5281/zenodo.23067051)&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Cite as: Chokshi, S. (2026). Does Indirection Hurt? Paired Evidence on LLM Review of Indirectly Written Cryptographic API Misuse [Preprint]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23351769" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23351769&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Dataset: Chokshi, S. (2026). CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers [Dataset]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23067051&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. &lt;a href="https://orcid.org/0009-0003-4738-7759" rel="noopener noreferrer"&gt;ORCID&lt;/a&gt; · &lt;a href="https://github.com/sunny-chokshi" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Originally published at &lt;a href="https://sunny-chokshi.github.io/blog/reviewer-that-flags-everything.html" rel="noopener noreferrer"&gt;https://sunny-chokshi.github.io/blog/reviewer-that-flags-everything.html&lt;/a&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>codereview</category>
    </item>
    <item>
      <title>A hex literal made LLM code reviewers raise false alarms</title>
      <dc:creator>Sunny Chokshi</dc:creator>
      <pubDate>Wed, 28 Oct 2026 19:30:33 +0000</pubDate>
      <link>https://dev.to/sunnychokshi/a-hex-literal-made-llm-code-reviewers-raise-false-alarms-8ai</link>
      <guid>https://dev.to/sunnychokshi/a-hex-literal-made-llm-code-reviewers-raise-false-alarms-8ai</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F93nsk6i5mk1e17c9srif.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F93nsk6i5mk1e17c9srif.png" alt="0xC00 equals a 3072-bit key. Secure, yet it drew false alarms. Minus 4.5 points balanced accuracy, plus 9.1 points false positives, 16 models, 90 snippet pairs." width="800" height="418"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I wanted to know whether an LLM code reviewer follows what the code does or what the code looks like. Cryptographic API misuse is a clean place to ask, because the same insecure choice can be written in many ways.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;I used 90 pairs of Python snippets across nine misuse classes. Each pair holds a vulnerable snippet and a secure twin. For 80 pairs I wrote a plain form, with the misuse at the call site under its familiar name, and an indirect form that reaches the same choice through an alias, constant, default, or configuration value. I ran 16 open models.&lt;/p&gt;

&lt;p&gt;The secure twins already showed a problem. Two models flag nearly every snippet, and four more flag over 70% of secure code. A reviewer that flags nearly everything tells me little about the code.&lt;/p&gt;

&lt;h2&gt;
  
  
  What indirection did to the scores
&lt;/h2&gt;

&lt;p&gt;Indirection lowered the balanced accuracy of the seven strongest models by 4.5 points over the 80 pairs (pair-cluster 95% CI 1.3 to 7.6). On the 47 pairs that survive a post hoc stimulus audit, the drop was 5.3 points (CI 2.1 to 8.5).&lt;/p&gt;

&lt;p&gt;Where the drop comes from needs care. After the audit it is mainly extra flags on correct code. False positives rose 9.1 points (CI 4.2 to 14.6), mostly in three models. Detection of real misuse fell 3.6 points over all 80 pairs (CI 1.0 to 6.5), but only 1.5 points on the audited pairs, and that interval (CI -1.6 to 4.8) includes zero.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the false alarms sit
&lt;/h2&gt;

&lt;p&gt;Exploratory splits point at where the false alarms come from. Most of the net false-alarm rise sits in pairs where the safe value has left its usual spelling or place at the call site. A 3072-bit key size written as 0xC00 is the example I keep coming back to. My reading is that the models react to an unfamiliar spelling instead of the value. The paper does not test that directly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;The snippets are Python and I wrote them. The models are open models. The audit was post hoc and the splits are exploratory. This is a preprint, it is not peer reviewed, and it carries the notice that it has been submitted to the IEEE for possible publication.&lt;/p&gt;

&lt;p&gt;Open question for the next study: do the extra flags go away if the reviewer is told to normalize values before judging? I have not tested that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read and cite
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://doi.org/10.5281/zenodo.23351769" rel="noopener noreferrer"&gt;Preprint on Zenodo (doi:10.5281/zenodo.23351769)&lt;/a&gt; · &lt;a href="https://sunny-chokshi.github.io/papers/does-indirection-hurt.html" rel="noopener noreferrer"&gt;Paper page and PDF&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Related work in the same line, an earlier benchmark: &lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;CryptoBench (doi:10.5281/zenodo.23067051)&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Cite as: Chokshi, S. (2026). Does Indirection Hurt? Paired Evidence on LLM Review of Indirectly Written Cryptographic API Misuse [Preprint]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23351769" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23351769&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Dataset: Chokshi, S. (2026). CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers [Dataset]. Zenodo. &lt;a href="https://doi.org/10.5281/zenodo.23067051" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23067051&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. &lt;a href="https://orcid.org/0009-0003-4738-7759" rel="noopener noreferrer"&gt;ORCID&lt;/a&gt; · &lt;a href="https://github.com/sunny-chokshi" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Originally published at &lt;a href="https://sunny-chokshi.github.io/blog/hex-key-size-false-alarms.html" rel="noopener noreferrer"&gt;https://sunny-chokshi.github.io/blog/hex-key-size-false-alarms.html&lt;/a&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>llm</category>
      <category>cryptography</category>
      <category>codereview</category>
    </item>
  </channel>
</rss>
