🚨 Building an IOC Triage Pipeline with Suricata + ML + Docker

Honeypots generate tons of noisy logs. The challenge: how do you quickly tell which IPs deserve your attention and which are just background noise?
In this post, I’ll walk through how I built an IOC triage pipeline that ingests Suricata/Zeek telemetry, scores suspicious IPs, applies unsupervised ML, and outputs actionable blocklists.

🌐 The Problem

If you’ve ever run a honeypot like T-Pot, you know the drill:

• Gigabytes of Suricata/Zeek alerts
• Thousands of unique source IPs
• Endless false positives

Manually sorting through all this isn’t scalable.
I wanted a pipeline that could automatically:

1. Aggregate activity per IP
2. Score each IP on suspicious behavior
3. Use ML to flag anomalies
4. Output human-readable casefiles + blocklists

🛠️ The IOC Triage Pipeline

I built a Python tool (ioc_triage.py) that takes NDJSON logs and produces structured outputs.

Key Features

• Ingest Suricata/Zeek/T-Pot logs
• Aggregate features like flows/min, unique ports, entropy, burstiness
• Rule-based scoring (customizable via config.yaml)
• Unsupervised ML (IsolationForest + LOF + OCSVM, optional PyOD HBOS+COPOD)

• Fusion of rules + ML → combined tier (observe, investigate, block_candidate)

• Outputs:

  • Enriched per-IP CSVs
  • JSON casefiles
  • Blocklists (per-IP and prefix)

⚙️ How It Works

1. Ingest

Reads Suricata NDJSON logs:
bash

python scripts/ioc_triage.py \
  --input data/samples/raw.ndjson \
  --hours 72 -vv

2. Aggregate

Per source IP, it computes:

• Flows/minute
• Unique src/dst ports
• Burstiness (variance of activity)
• Port entropy
• Signature counts ###3. Score

Configurable rule weights in scripts/config.yaml:
yaml

score:
  weights:
    flows_per_min: 2.0
    unique_dst_ports: 1.6
    unique_src_ports: 1.3
    alert_count: 0.8
    max_severity: 0.6
  thresholds:
    block: 7.0
    investigate: 3.5

4. Machine Learning

Uses unsupervised anomaly detection:

• IsolationForest
• LocalOutlierFactor
• OneClassSVM (Optionally PyOD HBOS+COPOD)

Scores are normalized and combined into ml_score + ml_confidence.

5. Fusion

Rules + ML = tier_combined
→ final decision: observe, investigate, or block_candidate.

📦 Setup

Clone the repo:
bash

git clone https://github.com/YOUR-USERNAME/ioc-triage-pipeline.git
cd ioc-triage-pipeline

Install requirements:
bash

pip install -r requirements.txt

(Optional ML extras):
bash

pip install pyod scikit-learn

Or run via Docker:
bash

docker build -t ioc-triage .
docker run -it --rm -v $(pwd):/app ioc-triage \
    python scripts/ioc_triage.py --input data/samples/raw.ndjson --hours 72 -vv

🔍 Example Output

table

ip  score   ml_score    tier    ml_tier tier_combined   reason
61.184.87.135   9.455   0.944   block_candidate block   block_candidate flows/min high, burstiness high, multiple ports

Outputs:

• data/outputs/enriched.csv → per-IP features
• cases/.json → casefiles
• outputs/blocklist_combined.tsv → fused blocklist
• outputs/blocklist_combined_prefix.tsv → aggregated /24 + /48 prefixes

🙌 Why This Matters

This project turns raw honeypot noise into actionable intelligence:

• Analysts can focus on high-confidence threats
• Blocklists update automatically
• You can tune thresholds & ML contamination rates

It’s also great for students (like me!) to showcase ML + cybersecurity skills in a practical, portfolio-ready way.

📚 What’s Next?

• Try deep learning models (autoencoders, transformers)
• Add active enrichment (WHOIS, VirusTotal, AbuseIPDB)
• Build dashboards for live triage

👉 GitHub Repository

If you’re into honeypots, ML, or threat intelligence, give it a ⭐ on GitHub and let me know what features you’d like to see next!