After event viewer crashed on a 400mb evtx, i wrote my own log triage cli

#blueteam #threathunting #python #cybersecurity

last week i was poking through event logs from a home lab vm i suspected had been scanned hard. dropped the evtx into event viewer. it took 90 seconds to load, then crashed the moment i tried to filter by event id 4624.

splunk is overkill for one machine. wazuh wants infra i didn't want to set up just to look at one file. pysigma converts sigma rules to backend queries, but i didn't have a backend. so i wrote threatlens.

it's a cli. point it at a log file or directory, get alerts mapped to mitre att&ck.

threatlens scan logs/ --min-severity high

that's the whole interface for the common case.

what i actually wanted

three things, roughly in priority order.

works on a single laptop with no infra. no daemon. no agent. no message queue. only runtime dep is pyyaml.

reads the formats i actually have. evtx (windows native), json/ndjson (modern stuff), syslog (linux), cef (network gear).

speaks sigma. the community has thousands of detection rules already written. i didn't want to invent another rule format.

what i tried first

python-evtx for parsing, which worked fine. then plyara for sigma (turns out plyara is yara, not sigma, oops). then pysigma, which converts sigma to backend queries. i didn't want a query string, i wanted in-memory matching against parsed events.

the precedence stuff bit me hard. my first parser handled a or b and c left-to-right and got the wrong answer half the time. rewrote it three times before it matched sigma's reference behavior.

the part i'm most proud of

the elasticsearch output. i wanted to push alerts to ES so they'd show up alongside other security data. the official elasticsearch python client is 40mb installed and pulls in dozens of transitive deps i didn't want to audit.

then i remembered: the bulk api is just newline-delimited json over http.

import json, urllib.request

def push_alerts(alerts, url, index, api_key=None):
    lines = []
    for a in alerts:
        lines.append(json.dumps({"index": {"_index": index}}))
        lines.append(json.dumps(a.to_dict()))
    body = ("\n".join(lines) + "\n").encode("utf-8")

    headers = {"Content-Type": "application/x-ndjson"}
    if api_key:
        headers["Authorization"] = f"ApiKey {api_key}"

    req = urllib.request.Request(
        f"{url.rstrip('/')}/_bulk",
        data=body,
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as resp:
        return json.loads(resp.read())

stdlib only. works against real ES clusters. saves about 40mb of install size and removes a whole category of supply chain risk.

attack chain correlation

single alerts are noisy. "failed logon" by itself means nothing. "failed logon burst, then privilege escalation, then lateral movement on the same account inside a 10 minute window" is a story.

the chain detector groups alerts by username and timestamp, then walks them through kill chain order. credential access then priv esc then lateral movement then execution. if the order matches and the events sit inside a tunable time window, it fires a single high severity chain alert that links back to the constituent events.

against a 52 event mixed-noise dataset i wrote, it pulls out two distinct chains and produces zero false positives on the benign activity. focused 26 event simulation lights up correctly too.

the 12 detectors

each one is a separate python module subclassing a DetectionRule base. brute force, lateral movement, privilege escalation, suspicious process, defense evasion, persistence, discovery, exfiltration, kerberos attacks (kerberoasting and as-rep roasting), credential access (lsass, sam, dcsync), initial access (external rdp, after-hours logons), and the chain correlator.

custom yaml rules and sigma rules get loaded on top of those. you can also drop a .py file into --plugin-dir and the loader picks it up at scan time.

what's still rough

the html report works but the css is ugly. svg donut chart for severity, expandable evidence per alert, but the typography needs polish.

i haven't tested against a real enterprise dataset. sample data is hand-crafted to exercise specific detectors. there's a synthetic generator for 1000 event datasets, but synthetic isn't real.

sigma loader doesn't handle count() by aggregations yet. doesn't handle cross-rule correlations either. those are the next two things on the list.

native evtx parsing requires python-evtx as an optional extra. without it you have to export to json first. i'd rather auto-detect at runtime and fall back gracefully.

what i'd do differently

write the sigma test corpus before the parser. every fix would have been faster with real test cases in place.

design the alert model around elasticsearch field naming rules from day one. had to rename three fields late because they weren't valid es field names.

decide upfront whether it's a tool or a framework. threatlens is mostly a tool, but the plugin system pushed it toward framework, and the ambiguity cost some design clarity.