Building AgentSmith-HUB: A High-Performance Security Data Pipeline with Real-Time Detection

#security #cybersecurity

Over the past few months, I’ve been working on a project called AgentSmith-HUB. It started as an experiment to build a lightweight, flexible rules engine for security data processing, but eventually became a full platform for real-time threat detection and response.

This post explains why I built it, how it works, and what kind of use cases it supports.

Why I Built It

Security teams often deal with massive volumes of logs and alerts. Traditional SIEMs can be heavy, expensive, or inflexible. In many cases, teams want to build their own detection logic, integrate with existing systems, and have full control over data pipelines.

I wanted something that:

Processes logs in real time
Allows flexible detection logic (regex, thresholds, enrichment)
Supports custom plugins for threat intel lookups, automated responses, or data transformations
Can run in distributed mode and scale easily
Is fast enough to handle high-throughput data

What It Is

AgentSmith-HUB is an open-source security data pipeline platform with a built-in real-time detection engine.

Main features:

Flexible rules engine with a simple XML-like syntax (, , , )
Plugin system (built-in and custom) for enrichment, alerting, and external integrations
Cluster/distributed mode for scalability
Full-featured web UI for building and testing workflows visually
MCP (Model Context Protocol) support, so it can integrate with LLM-based assistants for rule editing and operations
Integrates with Kafka, Elasticsearch, and cloud logging services

Performance

In testing (8 complex rules), it processed around 40,000 messages per second with sub-millisecond latency on a 2‑CPU, 4‑GB server.

Typical Use Cases

Intrusion detection with flexible rules and enrichment
Automated response workflows (block IP, send alerts, enrich data with threat intel)
Real-time log analysis and masking of sensitive information
A lightweight alternative to heavy SIEM setups

Example Rule

Here’s a simple rule that detects admin logins and appends additional fields:

<rule id="detect_admin_login" name="Detect Admin Login">
  <check type="EQU" field="username">admin</check>
  <append field="alert">admin login detected</append>
</rule>