<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Sérgio Azevedo</title>
    <description>The latest articles on DEV Community by Sérgio Azevedo (@malapeiro).</description>
    <link>https://dev.to/malapeiro</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4247419%2F6371a78e-14a7-49e2-8d45-e089d06ce133.jpg</url>
      <title>DEV Community: Sérgio Azevedo</title>
      <link>https://dev.to/malapeiro</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/malapeiro"/>
    <language>en</language>
    <item>
      <title>I asked an LLM to harden a real SigmaHQ rule. Its self-test said 19/19 PASS. The real compiler said otherwise.</title>
      <dc:creator>Sérgio Azevedo</dc:creator>
      <pubDate>Mon, 16 Nov 2026 10:19:31 +0000</pubDate>
      <link>https://dev.to/malapeiro/i-asked-an-llm-to-harden-a-real-sigmahq-rule-its-self-test-said-1919-pass-the-real-compiler-said-3gga</link>
      <guid>https://dev.to/malapeiro/i-asked-an-llm-to-harden-a-real-sigmahq-rule-its-self-test-said-1919-pass-the-real-compiler-said-3gga</guid>
      <description>&lt;p&gt;I asked an LLM to harden a published SigmaHQ detection rule. It delivered a new version with its own test report: &lt;strong&gt;19/19 PASS, both false positives closed.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;When I ran the same rule through the real Sigma compiler against real Windows event logs, nothing had changed. Both false positives still fired. One line of the rule had quietly turned its own filter into a condition that is always true.&lt;/p&gt;

&lt;p&gt;This post is about that one line, why the model's self-test could not see it, and what happened after I reported it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rule:&lt;/strong&gt; SigmaHQ's &lt;code&gt;proc_access_win_lsass_memdump.yml&lt;/code&gt; (LSASS credential dumping, MITRE ATT&amp;amp;CK T1003.001).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ground truth:&lt;/strong&gt; 725 real &lt;code&gt;.evtx&lt;/code&gt; files (37,841 events) from &lt;a href="https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES" rel="noopener noreferrer"&gt;EVTX-ATTACK-SAMPLES&lt;/a&gt; and SigmaHQ's own regression data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Engine:&lt;/strong&gt; pySigma with the SQLite backend. Every rule version was compiled and executed, not just read.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The original rule detects 7 of the 11 documented LSASS dump samples in that corpus. I asked the model to recover the 4 it misses without losing the 7 and without opening false positives.&lt;/p&gt;

&lt;p&gt;Round 1 recovered all 4, but opened 2 false positives on real files. The model's own test (17 cases, some invented) had reported 0.&lt;/p&gt;

&lt;h2&gt;
  
  
  The line
&lt;/h2&gt;

&lt;p&gt;In round 2 I gave the model the two false-positive events and asked it to close them. It narrowed one branch of the rule to suspicious &lt;code&gt;CallTrace&lt;/code&gt; paths:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;CallTrace|contains&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s1"&gt;'\Users\'
&lt;/span&gt;  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s1"&gt;'\Temp\'
&lt;/span&gt;  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s1"&gt;'\Downloads\'
&lt;/span&gt;  &lt;span class="c1"&gt;# (three more paths)
&lt;/span&gt;  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s1"&gt;'\\'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The last value was meant to match UNC paths (&lt;code&gt;\\server\share&lt;/code&gt;). The model ran &lt;code&gt;sigma check&lt;/code&gt; (0 errors), ran its own Python test harness, and reported &lt;strong&gt;19/19 PASS&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Here is what pySigma actually compiles that to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="nc"&gt;CallTrace&lt;/span&gt; &lt;span class="k"&gt;LIKE&lt;/span&gt; &lt;span class="s1"&gt;'%\\Users\\%'&lt;/span&gt; &lt;span class="no"&gt;ESCAPE&lt;/span&gt; &lt;span class="s1"&gt;'\'
  OR CallTrace LIKE '&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;\\&lt;span class="k"&gt;Temp&lt;/span&gt;\\&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="s1"&gt;' ESCAPE '&lt;/span&gt;\&lt;span class="s1"&gt;'
  OR CallTrace LIKE '&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;\\&lt;span class="nc"&gt;Downloads&lt;/span&gt;\\&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="s1"&gt;' ESCAPE '&lt;/span&gt;\&lt;span class="s1"&gt;'
  -- (three more paths)
  OR CallTrace LIKE '&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;\\&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="s1"&gt;' ESCAPE '&lt;/span&gt;\&lt;span class="s1"&gt;'
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The last branch means "contains one backslash". Every Windows path contains a backslash, so the whole list is always true and the narrowing does nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the self-test passed
&lt;/h2&gt;

&lt;p&gt;In Sigma, backslash is the escape character, so &lt;code&gt;'\\'&lt;/code&gt; is &lt;strong&gt;one&lt;/strong&gt; literal backslash. That is intended and documented behavior, not a pySigma bug.&lt;/p&gt;

&lt;p&gt;The model believed &lt;code&gt;'\\'&lt;/code&gt; meant two backslashes. Its test harness did not call pySigma. It loaded the YAML and did plain Python substring checks, which also treat &lt;code&gt;'\\'&lt;/code&gt; as two backslashes. The test shared the author's wrong belief, so it passed.&lt;/p&gt;

&lt;p&gt;That is the finding: &lt;strong&gt;the model validated its rule against its own idea of what the rule means, not against the engine that runs it.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;When I showed the model the compiled SQL, with no further hint, it found the line, removed it, and switched its tests to the real compiler. The final version detects 11/11 with 0 false positives on this corpus.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproduce it in two minutes
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install pysigma pysigma-backend-sqlite
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sigma&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;collection&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;SigmaCollection&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sigma&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;backends&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sqlite&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sqliteBackend&lt;/span&gt;

&lt;span class="n"&gt;rule&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"""
title: repro
logsource: {category: process_access, product: windows}
detection:
  sel:
    CallTrace|contains:
      - '\Users\'
      - '\\'
  condition: sel
"""&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;sqliteBackend&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;convert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;SigmaCollection&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_yaml&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rule&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output (pySigma 2.1.0, pysigma-backend-sqlite 2.1.1):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;logs&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="nc"&gt;CallTrace&lt;/span&gt; &lt;span class="k"&gt;LIKE&lt;/span&gt; &lt;span class="s1"&gt;'%\\Users\\%'&lt;/span&gt; &lt;span class="no"&gt;ESCAPE&lt;/span&gt; &lt;span class="s1"&gt;'\' OR CallTrace LIKE '&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;\\&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="s1"&gt;' ESCAPE '&lt;/span&gt;\&lt;span class="s1"&gt;'
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What happened next
&lt;/h2&gt;

&lt;p&gt;I opened an issue on pySigma suggesting a validator for values like this, framed as an idea rather than a bug: &lt;a href="https://github.com/SigmaHQ/pySigma/issues/615" rel="noopener noreferrer"&gt;SigmaHQ/pySigma#615&lt;/a&gt;.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Within 12 hours another contributor implemented it: &lt;a href="https://github.com/SigmaHQ/pySigma/pull/617" rel="noopener noreferrer"&gt;pySigma#617&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;He ran it over the 3,305 official SigmaHQ rules. It found a published rule where an empty value turns a filter into a catch-all, now reported, with a fix proposed, in &lt;a href="https://github.com/SigmaHQ/sigma/issues/6596" rel="noopener noreferrer"&gt;sigma#6596&lt;/a&gt; / &lt;a href="https://github.com/SigmaHQ/sigma/pull/6597" rel="noopener noreferrer"&gt;sigma#6597&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As of this writing, both pull requests are open and awaiting maintainer review. Nothing is merged.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does not show
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;One rule, one technique, one backend. The mechanism is probably general; what is demonstrated is one instance.&lt;/li&gt;
&lt;li&gt;0 false positives on this corpus is not 0 false positives in production. The corpus has no clean enterprise telemetry.&lt;/li&gt;
&lt;li&gt;This is not "LLMs can't write detections". Shown the real compiler output, the model fixed its own bug.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The practical rule I took from it: &lt;strong&gt;never accept an LLM's self-test for a detection rule. Compile it with the real engine and run it against data the model did not choose.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Full study, rules, compiled SQL and evaluation script: &lt;a href="https://github.com/malapeiro/llm-defensive-disclosures/tree/main/sigma_t1003_study" rel="noopener noreferrer"&gt;sigma_t1003_study&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This work was done with LLM assistance throughout: one model built the rules, a second audited them, a third independently re-ran every measurement. I ran the experiment and decided what counts as evidence.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>detection</category>
      <category>sigma</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
