A SOC analyst spends an afternoon building a DNS tunneling detection rule: long subdomain labels, high character entropy, a spike in NXDOMAIN responses, too many unique subdomains under one registered domain in a short window. Standard signal set, nothing exotic. To validate it, they replay a handful of tunneling samples generated with a common lab tool and watch the rule fire cleanly on every one. Confident, they ship it to production that same afternoon.
By evening the rule has paged the on-call rotation three times. None of the three is tunneling. One is a corporate VPN client's routine health-check traffic, which legitimately generates long, mixed-character subdomains as part of how it phones home. Another is a logging agent whose telemetry hostnames are structurally close to what a slow DNS exfil channel produces: high entropy, unique per request, gone in seconds. By the second day, the rotation has stopped trusting the alert, and someone disables the rule rather than keep chasing it. The rule that could have caught a real slow-drip exfiltration is now off, and it stays that way because nobody wants to get paged for a VPN client again.
The individual signals in that rule are the right ones. Real DNS tunneling detection genuinely does rest on subdomain length distribution, per-label character entropy, the ratio of NXDOMAIN responses to total queries against a given parent domain, and the count of unique subdomains hit under one registered domain inside a rolling window, because a tunneling tool has to encode payload into subdomain labels, and that encoding looks statistically different from ordinary DNS traffic. None of that is wrong. What's missing is testing the rule against anything other than attack traffic.
Attack samples tell you a rule catches attacks. They tell you nothing about what else in your specific environment produces DNS traffic that happens to share the same statistical shape: CDNs, cloud logging and telemetry agents, VPN clients doing periodic health checks, any SaaS product generating per-session or per-tenant subdomains. The only way to find that out before your pager does is running the rule, before it ships, against a real corpus of your own environment's benign DNS logs, not a synthetic clean sample, your actual production noise. That's what "detection as code" is supposed to mean in practice: a rule gets a test suite, known-bad traffic it should fire on and known-good traffic pulled from your own logs that it should stay silent on, and it only ships once both pass. Skipping that step is how a technically correct rule becomes an alert nobody trusts by day two.
The other half of the discipline is that validating the logic once isn't the end of the work if the rule has to run somewhere specific. A rule tuned and tested as Sigma still has to get translated correctly into whatever your SIEM actually speaks, Splunk SPL, Sentinel KQL, Elastic EQL, and a translation that quietly drops a condition or changes an operator's semantics isn't "the same rule" anymore even though it carries the same name and the same intent. Detection engineering is writing the rule once, proving it against real traffic, and porting that proof across every syntax your environment actually runs, not writing four half-tested versions and hoping they agree.
The version of this story with a validation step built in doesn't end in three pages and a disabled rule. It ends in something that runs quietly in the background until the day it's actually needed, and nobody remembers it's there because it never asked for attention it didn't earn. Codelivly's Detection Engineering Book for SOC Analysts (427 pages) works through exactly that discipline: the same rule written and tested in Sigma, Splunk SPL, Sentinel KQL and Elastic EQL, so the reasoning gets done once instead of relearned every time a rule needs to run somewhere new.
If you want the hands-on version of this before anything else, the free Wazuh Rule Tuning lab on codelivly.com puts you in exactly this seat: a noisy rule, real logs, and the job of cutting the false positives without losing the true ones.
Top comments (0)