i normalized four log formats into one shape so my detection rules didn't have to care

triagelens is a little soc triage tool i've been building. you paste in raw logs, it runs rule-based detections, maps them to mitre att&ck, scores the risk 0-100, and writes an analyst-style report. the part that almost sank the whole thing wasn't the detections. it was the logs.

the problem

it takes four formats: windows security event 4688, sysmon event id 1, linux ssh auth.log, and generic json. sounds fine until you actually look at them.

a "process started" event exists in three of those four. in windows security 4688 it's a structured event with a new process name field and a creator process. in sysmon event 1 it's also structured but the fields sit in different places with different names, plus you get a command line and hashes windows security doesn't give you. in auth.log there's no such thing as event 1, it's a plain sentence like Failed password for root from 203.0.113.9 port 41822 ssh2. no event id, no fields, just text.

my first pass had the detection rules reach straight into the parsed log. so a rule for suspicious process spawns had a branch for "if it came from sysmon look here, if it came from windows security look there." every new rule inherited that mess. add a fourth format and you're editing every rule again. that doesn't scale and it's miserable to test.

what worked

one shape. every parser has to produce the same thing, and the rules only ever see that thing.

export interface NormalizedEvent {
  id: string;
  timestamp?: string;
  source: LogSource;         // 'windows-security' | 'sysmon' | 'ssh-auth' | 'json'
  eventId?: string | number;
  host?: string;
  user?: string;
  process?: string;
  commandLine?: string;
  sourceIp?: string;
  destIp?: string;
  message: string;
  raw: Record<string, unknown>;
}

four parsers (windowsSecurity.ts, sysmon.ts, auth.ts, genericJson.ts) each do one job: take their format and fill in this interface. sysmon's command line goes into commandLine. windows 4688's new process name goes into process. auth.log's regex-scraped ip goes into sourceIp. after that, nothing downstream knows or cares where an event came from. the detection code reads commandLine and user and process, full stop.

raw keeps the original parsed object around so a report can still show the untouched line, but no rule is allowed to read from it. that was a deliberate rule i set for myself. the moment a detection peeks at raw, the whole abstraction leaks and you're back to per-format branches.

the payoff shows up in the rules

a detection is just an object with some metadata and a detect function that filters normalized events. here's the real one for encoded powershell, mapped to att&ck t1059.001 and t1027:

{
  id: 'encoded-powershell',
  title: 'Obfuscated or encoded PowerShell',
  severity: 'high',
  techniques: ['T1059.001', 'T1027'],
  detect: (events) =>
    events
      .filter(e => /(-enc(odedcommand)?\b|-e\s+[A-Za-z0-9+/=]{20,}|frombase64string|-nop\b.*-w\s+hidden|-windowstyle\s+hidden)/i
        .test(e.commandLine ?? ''))
      .map(e => evidenceLine(e, e.commandLine)),  // "[host] user: <commandLine>"
}

look at what it doesn't do. it never asks which format the event came from. it reads e.commandLine and that's the whole story. the same rule fires whether that command line arrived in a sysmon event 1 or a windows 4688 record, because by the time a rule runs, that difference is already gone.

testing got easier the same way. parser tests take a raw sample line and assert the resulting NormalizedEvent fields. rule tests build a synthetic normalized event by hand and never load a real log at all. two kinds of tests, split cleanly, and neither one has to care about the other's mess. that alone is worth the abstraction.

the annoying part

auth.log. the other three formats hand you structured data. auth.log hands you a sentence, so auth.ts is a pile of regexes pulling out the user, the source ip, and whether the login failed or succeeded.

worse, syslog timestamps carry no year. a line starts with Jul 12 08:22:01 and that's it. so timestamp on an ssh event is a best guess, and if you ever feed it logs that straddle a new year the ordering is a coin flip. i left it as a known limitation rather than pretend a parser can invent the year. it's the kind of thing that's obvious once you hit it and invisible until you do.

what's still rough

• auth.ts is regex-driven, so a distro that formats auth.log slightly differently will slip past it. it's tuned to the common openssh format and not much else.
• genericJson.ts is a catch-all. it trusts that a field called user means a user. garbage in, confidently-parsed garbage out.
• the no-year timestamp thing. i'd rather carry an explicit "year unknown" flag than a silently-wrong date, i just haven't done it yet.

none of this is production siem territory and i'm not pretending it is. it's an analyst aid and a learning project. but the normalize-first design is the one decision i'd keep if i rewrote it tomorrow. write the parsers once, and every rule you add after that stays format-blind. it works. not perfect, but it works.

code's here if you want to pick it apart: https://github.com/TiltedLunar123/triagelens