Stop Piping Raw Alerts into an LLM: Building a SOC Triage Agent That Correlates First and Escalates Last

Adam Lewandowski — Tue, 28 Jul 2026 01:01:36 +0000

Every few weeks another demo lands on my feed: someone pipes a security alert into a large language model, gets back a paragraph that sounds like a SOC analyst, and declares Tier-1 automated. Having spent years on the receiving end of enterprise alert queues — Cisco FTD intrusion events, Defender detections, sign-in risk alerts — I can tell you why those demos fall apart in production:

Alerts in isolation are noise. The signal is the story across alerts: the same actor showing up in a risky Entra ID sign-in, then AWS recon, then a persistence action. An LLM triaging one alert at a time can't see the campaign.
Cost scales with alert volume, and alert volume is enormous. If every event costs an API call, your triage bill grows linearly with your noisiest sensor.
Free-text verdicts aren't actionable. "This looks suspicious" doesn't page anyone. You need structured output a downstream system can route on.

If you haven't worked a queue like this, the ratios are hard to believe. A single FTD intrusion policy tuned one notch too aggressive will emit thousands of IPS events a day, most of them the same few signatures firing on scanner traffic that never reached anything vulnerable. A MECM patch wave lights up endpoint AV with detections on installer behavior that looks exactly like the droppers the heuristics were trained on. In most shops the alert-to-incident ratio is measured in orders of magnitude, not percentages. So the interesting engineering problem was never "can a model write a plausible paragraph about one alert." It's what you do with everything else.

So I built Aegis, a working AI SOC analyst, to demonstrate what I think the right shape is: normalize → correlate → then decide whether an LLM is even needed → structured verdict → gated containment. It ingests synthetic telemetry from four sources (AWS GuardDuty, CloudTrail, Entra ID sign-in logs, Microsoft Defender), stitches 17 raw events into incidents, auto-suppresses the false positives without spending a token, and hands only the genuinely ambiguous or dangerous cases to a Claude tool-use agent that must investigate before ruling.

The whole thing runs offline with zero dependencies (a deterministic rule engine stands in for the LLM), and upgrades to the live agent the moment an ANTHROPIC_API_KEY is present. In the demo run, 3 of 4 incidents cost $0.00. Only the real attack touches the API.

This article walks through the architecture and the design decisions, with real code from the repo.

The pipeline at a glance

   ingest  ─▶  correlate  ─▶  AI triage  ─▶  response playbook  ─▶  report
 (4 sources)  (mini-SIEM)   (Claude agent)  (Graph / PS / AWS)   (HTML + console)

ai-soc-analyst/
├─ run_demo.py                 # one-command entry point
├─ data/                       # synthetic telemetry (one file per source)
├─ src/aegis/
│   ├─ schema.py               # normalized SecurityEvent / Incident model
│   ├─ ingest.py               # source-specific normalizers → common schema
│   ├─ correlate.py            # union-find correlation engine (the mini-SIEM)
│   ├─ enrich.py               # threat-intel + identity enrichment (pluggable)
│   ├─ agent.py                # Claude tool-use agent + heuristic fallback
│   ├─ playbooks.py            # incident → containment actions
│   └─ report.py               # console + self-contained HTML report
└─ powershell/
    ├─ Invoke-EntraContainment.ps1   # Graph API identity containment
    └─ Get-SecureScoreDelta.ps1      # Secure Score posture job

Step 1: Normalize everything into one schema

GuardDuty findings, CloudTrail records, Entra sign-in logs, and Defender alerts all have wildly different JSON shapes, and worse, different severity vocabularies. GuardDuty gives you a 0–10 float; Microsoft gives you strings like "Informational." If you don't collapse those early, every downstream component grows source-specific branches.

Aegis flattens everything into one SecurityEvent dataclass, and severity coercion happens in exactly one place:

class Severity(IntEnum):
    INFO = 0
    LOW = 1
    MEDIUM = 2
    HIGH = 3
    CRITICAL = 4

    @classmethod
    def from_any(cls, value: Any) -> "Severity":
        if isinstance(value, Severity):
            return value
        if isinstance(value, (int, float)) and not isinstance(value, bool):
            # GuardDuty numeric severity: 0-3.9 low, 4-6.9 medium, 7-8.9 high, 9+ critical.
            if value >= 9:
                return cls.CRITICAL
            if value >= 7:
                return cls.HIGH
            if value >= 4:
                return cls.MEDIUM
            if value >= 1:
                return cls.LOW
            return cls.INFO
        text = str(value).strip().lower()
        return {
            "critical": cls.CRITICAL,
            "high": cls.HIGH,
            "medium": cls.MEDIUM,
            "moderate": cls.MEDIUM,
            "low": cls.LOW,
            "informational": cls.INFO,
            "info": cls.INFO,
            "none": cls.INFO,
        }.get(text, cls.MEDIUM)

One deliberate choice: unknown severity strings fall back to MEDIUM, not INFO. If a source starts emitting a vocabulary you didn't anticipate, you want those events above your triage floor, not silently dropped below it.

Adding a source is one normalizer function. The CloudTrail normalizer is a good example of encoding operational knowledge at ingest time. It scores write and defense-evasion API calls higher than read-only recon, so the correlation layer naturally weights persistence and tamper actions:

high_impact = {
    "CreateAccessKey", "StopLogging", "DeleteTrail", "PutUserPolicy",
    "AttachUserPolicy", "CreateUser", "UpdateAssumeRolePolicy",
}
recon = {"GetCallerIdentity", "ListUsers", "ListBuckets", "ListRoles", "DescribeInstances"}

Step 2: Correlate by actor, not just alert

This is the part most "LLM SOC" demos skip, and it's the part that actually matters. Aegis clusters events with a union-find over shared identity or shared source IP within a sliding time window, then unions overlapping clusters transitively — because attackers pivot:

for i in range(len(ordered)):
    for j in range(i + 1, len(ordered)):
        if ordered[j].timestamp - ordered[i].timestamp > window:
            break  # ordered by time; nothing further is in-window
        if _linked(ordered[i], ordered[j]):
            union(i, j)

The subtle bug most first attempts hit: the same human shows up under different identity strings in different control planes. A user's Entra UPN and their AWS IAM CLI user are, to the correlation engine, different principals — unless you teach it otherwise:

def _same_identity(p1: str, p2: str) -> bool:
    # 'j.okafor@corp.example', 'j.okafor-cli', and 'j.okafor'
    # all refer to the same human — a real attacker pivots across these.
    if not p1 or not p2:
        return False
    a = p1.split("@")[0].replace("-cli", "").replace("-", ".")
    b = p2.split("@")[0].replace("-cli", "").replace("-", ".")
    return a == b

Is this heuristic naive? Absolutely. It encodes one org's naming convention, and in production you'd back it with an identity graph (Entra ID ↔ AWS SSO mappings). But the architectural point stands: cross-control-plane identity resolution is a first-class correlation feature, not an enrichment afterthought. In the demo, this is exactly what stitches the Tor sign-in → MFA fatigue → AWS recon → CreateAccessKey persistence → StopLogging tamper → endpoint loader → mailbox-rule exfil chain into one P1 incident instead of seven mediums scattered across four consoles.

And this isn't a cloud-only problem. In a typical Cisco-and-Microsoft enterprise, one administrator exists as j.okafor@corp.com in Entra sign-in logs, CORP\jokafor in Windows Security events, a bare jokafor in the ISE live-session table (or only an endpoint MAC address, if the session authenticated via MAB), a local account name in FMC audit records, and CORP.LOCAL\jokafor to vCenter SSO. Five control planes, five spellings, zero agreement on a canonical key. A correlation engine that treats those strings as five different people will hand you five medium alerts where there is one story. SIEM vendors sell identity resolution as an enrichment add-on; I'd argue it belongs in the correlation key itself.

Step 3: Decide whether the LLM is even needed

Here's the cost-control lever that separates a demo from something deployable. Aegis runs tiered triage: a deterministic pre-filter auto-resolves the obvious cases for free, and only escalates incidents that carry a malicious indicator, a high-signal ATT&CK technique, or genuine ambiguity:

def _should_escalate(incident: Incident) -> bool:
    reputations = [enrich.lookup_ip(ip) for ip in incident.src_ips]
    has_bad_ip = any(r.reputation in {"malicious", "suspicious"} for r in reputations)
    high_signal = bool(set(incident.mitre) & HIGH_SIGNAL_TECHNIQUES)
    if has_bad_ip or high_signal:
        return True
    # No malicious indicators: auto-resolve service accounts and low-severity noise.
    if enrich.user_context(incident.principal)["is_service_account"]:
        return False
    if incident.max_severity <= Severity.LOW:
        return False
    # Ambiguous medium+ with no clear signal — worth a judgment call.
    return incident.max_severity >= Severity.MEDIUM

HIGH_SIGNAL_TECHNIQUES is a short, opinionated set — persistence (T1098.001), cloud-log tampering (T1562.008), PowerShell execution (T1059.001), mailbox-rule exfil (T1114.003), MFA fatigue (T1621), Tor proxying (T1090.003). These are the techniques where a wrong auto-suppress is expensive, so they always get a judgment call.

The suppressions every shop ends up writing are the same ones this pre-filter encodes: the internal vulnerability scanner tripping IPS signatures on schedule, the EICAR file the AV team drops every quarter, backup and CI service accounts authenticating from datacenter egress at 3 a.m. doing exactly what they're supposed to do. The failure mode is also the same everywhere. Suppression rules written broadly ("ignore this signature," "ignore this subnet") keep matching long after the environment changes, and one day the real thing rides in under an old exception. That's why _should_escalate is structured the way it is: suppression requires the absence of malicious indicators plus a positive reason to close (known service account, low severity), while the high-signal technique list is an always-escalate override no suppression can beat. Suppress by evidence, escalate by category — the inverse of how most alert-tuning debt accumulates.

In the demo run, three incidents (the research scanner, the EICAR test, the CI/CD service account) are closed by the free engine, and the API is invoked only for the actual intrusion.

Step 4: A tool-use agent that must investigate before ruling

When an incident does escalate, it isn't a single prompt. The Claude agent gets a SOC-analyst system prompt and four tools:

Tool	Purpose
`lookup_ip`	Threat-intel reputation for a source IP
`get_user_context`	Is this principal privileged? A service account?
`get_mitre_technique`	Resolve an ATT&CK ID to name + tactic
`submit_verdict`	Strict-schema final ruling — the only way to finish

The trick that makes the output dependable is submit_verdict. It's a tool with "strict": True and a closed schema; the model can't finish the loop without committing to typed fields:

{
    "name": "submit_verdict",
    "description": "Submit the final triage verdict. Call exactly once, after investigating.",
    "strict": True,
    "input_schema": {
        "type": "object",
        "properties": {
            "is_true_positive": {"type": "boolean"},
            "severity": {"type": "string", "enum": ["Critical", "High", "Medium", "Low", "Info"]},
            "priority": {"type": "string", "enum": ["P1", "P2", "P3", "P4"]},
            "confidence": {"type": "integer", "minimum": 0, "maximum": 100},
            "attack_narrative": {"type": "string", ...},
            "reasoning": {"type": "string", ...},
        },
        "required": ["is_true_positive", "severity", "priority", "confidence",
                     "attack_narrative", "reasoning"],
        "additionalProperties": False,
    },
}

No parsing prose, no regexing severity out of a paragraph. The verdict routes straight into playbook selection and the report. The agent loop also records every tool call into a human-readable trace, so the analyst reviewing the verdict can see how the machine investigated:

AI agent investigation (via claude-haiku-4-5):
    lookup_ip(185.220.101.47) -> malicious (tor-exit-node, anonymizer, c2-adjacent)
    get_user_context(j.okafor@...) -> standard user
    get_mitre_technique(T1562.008) -> T1562.008 — Impair Defenses: Disable Cloud Logs
    ✓ submit_verdict -> Critical/P1 (TRUE positive)

That trace is not a gimmick. If you're going to let a model influence incident priority, "show your work" is the minimum bar for analyst trust, and for the post-incident review when it gets one wrong.

Two more production-shaped details in the agent path:

Fail safe, always. The public triage() entry point wraps the live agent in a try/except; any API error or step-limit overrun falls back to the deterministic engine and annotates the verdict with what happened. One API hiccup never stalls the alert queue mid-shift.
Cost is visible, not a mystery. Every run tracks token usage per incident and prints an estimated dollar cost, with a cheap default model (Haiku-class) and an env-var override (AEGIS_MODEL) for deployments that want deeper reasoning on escalated incidents only.

Step 5: Containment that names its control plane

A triage verdict that ends in "investigate further" is a demo. Aegis maps verdicts to concrete containment actions, and every action names the real automation that would carry it out:

actions.append(Action(
    name="Disable compromised account & revoke sessions",
    target="Microsoft Graph (Entra ID)",
    automation=(
        f"pwsh ./powershell/Invoke-EntraContainment.ps1 "
        f"-UserPrincipalName {upn} -DisableAccount -RevokeSessions"
    ),
    rationale=(
        "Risky/failed MFA sign-in from a malicious IP indicates account "
        "takeover; disabling the account and revoking refresh tokens cuts "
        "active attacker sessions immediately."
    ),
    destructive=True,
))

The destructive=True flag is load-bearing: every account-disable, session-revoke, MFA-reset, and endpoint-isolation action is gated behind human approval, and the companion PowerShell (Invoke-EntraContainment.ps1) runs in simulation unless -Execute is explicitly passed.

Auto-containment earns trust slowly, and it should. The classic failure is an action keyed on the wrong attribute. Quarantine-by-identity in a NAC deployment like ISE will happily kill every session the account holds, including the endpoints nobody remembered were authenticating with it. Auto-disabling a "compromised" account that turns out to be a service identity takes down every integration that depends on it, usually at the worst possible hour. Aegis encodes one small defense against this class of mistake directly in the playbook layer: when it recommends perimeter blocks, it filters to reputationally-bad IPs only, so the actor's legitimate corporate egress — which the identity linkage correctly pulled into the incident — never lands on a deny-list.

What I'd tell you before you build one

What worked:

Normalize-first paid for itself immediately. Correlation, enrichment, triage, and reporting share one schema; adding a fifth source is one function.
Tiered escalation is the difference between a toy and a bill you can defend. The LLM is a scarce senior analyst, not a firehose destination.
Strict tool schemas beat prompt-engineering for structured output. Forcing the final answer through a closed-schema tool call eliminated an entire class of parsing failures.
Building the heuristic fallback as a real second implementation (not a stub) meant the demo works offline and gave the live agent a benchmark: both engines reason over identical enrichment, so their disagreements are informative.

Honest limitations:

The telemetry is synthetic and small. Seventeen hand-crafted events prove the architecture, not the recall. Real environments bring malformed timestamps, delayed log delivery, and volumes where the windowed pairwise correlation pass needs streaming or bucketing.
Identity matching is a naming-convention heuristic. Production needs a real identity graph (Entra ↔ AWS SSO ↔ HR system), and it will still be the hardest data-quality problem in the pipeline.
The threat-intel and identity enrichment are local lookups. They're pluggable by design — swapping in GreyNoise/AbuseIPDB is the intended path — but reputation-driven verdicts are only as good as the feed behind them.
The LLM's verdicts haven't been evaluated at scale. Before trusting it with suppression decisions, you'd want a labeled incident corpus and a measured false-negative rate. An "it sounded right on four incidents" evaluation is exactly the trap this article warns about.

My bottom line: I'd deploy this pattern today, but not with this blast radius. Start with the agent as an enrichment-and-draft layer — it investigates, writes the narrative, proposes severity and priority, and a human confirms every verdict. Suppression authority, the right to close an incident unseen, comes last and only per alert class, after the agent has run in shadow mode against human tier-1 decisions long enough to measure a false-negative rate you'd sign your name to. Agreement rate is the vanity metric; the number that matters is how often the machine says "benign" when the human said "incident." Until you can put a measured value on that, the correlate-first architecture is doing most of the heavy lifting anyway. And that part costs nothing.

Adam Lewandowski is a network and security engineer (CompTIA Security+, CCNA, CCNP, VMware VCP-DCV) with several years of hands-on enterprise operations across Cisco FTD/FMC/ISE, Windows Server, MECM, and VMware. He builds automation that takes the repetitive judgment calls out of security operations — without taking the humans out of the loop. Find him on LinkedIn.

DEV Community: Adam Lewandowski