Detection Engineering Lab: Building a Wazuh SIEM and Writing a Custom Rule for SSH Brute Force

This write-up documents a small SIEM detection lab I built to practice detection engineering from the defender's side: deploying a SIEM, generating adversary activity against a monitored host, validating that it is detected, and writing a custom correlation rule mapped to MITRE ATT&CK.

Repository with the rule, configuration notes and all screenshots: https://github.com/devyaninandanwar/wazuh-siem-detection-lab

Objective

To validate end to end that a SIEM can ingest endpoint logs, surface attacker activity, and support a custom detection, and to practice the triage thinking that goes with it, including recognizing a false positive.

Architecture

Three virtual machines in VirtualBox on a bridged network:

Host	Role
wazuh-manager	Wazuh server, indexer and dashboard (the SIEM)
monitored-endpoint	Ubuntu Server running OpenSSH and Apache, with the Wazuh agent
Kali Linux	Adversary simulation host

The Wazuh agent on the endpoint collects local logs (journald for SSH authentication events, and the Apache access log) and forwards them to the manager. The manager decodes the events, evaluates them against its ruleset, and indexes alerts for the dashboard.

Design decisions

The lab uses three separate hosts instead of running everything on one machine, for four reasons.

Log integrity
An attacker who compromises a host can alter or delete its local logs. Forwarding events to a separate manager preserves the evidence even if the endpoint is compromised.

Realistic data flow
In production, a SIEM collects telemetry from many hosts over the network. Separating the agent from the manager reproduces that flow end to end.

Resource isolation
The Wazuh stack (manager, indexer and dashboard) is memory-intensive. Keeping it off the endpoint avoids distorting the behavior of the host being monitored.

Observable attacker identity
Running the attacks from a dedicated Kali host means the alerts record a real source IP, which is the field an analyst uses to start an investigation.

Deployment issues

Two infrastructure problems are worth recording, since they affect anyone running this on a laptop.

Storage
The first manager VM had a 25 GB disk, and the all-in-one installer failed with "No space left on device". After rebuilding with 60 GB, the Ubuntu installer's LVM layout still allocated only about 29 GB to the root logical volume. The fix was lvextend -l +100%FREE on the logical volume followed by resize2fs, verified with df -h.

Memory
With 4 GB of RAM, the Wazuh indexer, manager and dashboard compete for memory, and after a reboot some services failed to start. Starting the indexer first and the dependent services afterwards resolved it.

Detection scenario 1: SSH brute force (MITRE ATT&CK T1110)

Simulated from the Kali host with Hydra against the endpoint's SSH service:

hydra -l ubuntu -P passwords.txt ssh://192.168.1.194

The agent's authentication logs triggered Wazuh's built-in rule 5760 (sshd: authentication failed) repeatedly. The event detail includes the targeted account, the source IP and port, and the raw log line. Together, these give an analyst the data needed to start an investigation.

Detection scenario 2: Network reconnaissance (corresponds to MITRE ATT&CK T1595, Active Scanning)

Before attacking a machine, an attacker usually scans it to find out what it is running. From the Kali host, I ran an Nmap service-version scan against the monitored endpoint:

sudo nmap -sV 192.168.1.194

The endpoint runs an Apache web server, so Nmap probed it too. Nmap deliberately requests web pages that don't exist, because the way a server answers for a missing page tells Nmap how that server behaves, and it needs that to interpret its other results. Apache returned a 404 error for each of these requests and recorded them in its access log on the endpoint.

The Wazuh agent on the endpoint reads that log and forwards it to the manager. The manager flagged the 404 errors under its web rules (31101). The raw log line also shows the Nmap user agent, which tells you which tool made the requests.

Detection engineering: a custom correlation rule

The built-in rule 5760 alerts on every failed login. A single failure is normal operational noise, so alerting on it alone produces low-value alerts. The detection of interest is a burst of failures. I wrote a frequency-based correlation rule that fires when rule 5760 triggers five times within 120 seconds:

<rule id="100002" level="10" frequency="5" timeframe="120">
  <if_matched_sid>5760</if_matched_sid>
  <description>SSH brute-force attack detected</description>
  <mitre>
    <id>T1110</id>
  </mitre>
  <group>authentication_failures,pci_dss_11.4,mitre,</group>
</rule>

Re-running the Hydra attack triggered rule 100002 at level 10, confirming the detection works against the simulated activity.

Alert triage: a rootcheck false positive

After deployment the SIEM raised 84 alerts for rule 510 (rootcheck), "Trojaned version of file detected", on system binaries such as /usr/bin/md5sum. On a freshly built host this warrants investigation before dismissal. Reviewing the event showed the rootcheck signature is a generic string pattern that matches any file containing certain references (for example /bin/sh), and no other indicators of compromise were present. My assessment was that this is a known false positive from generic rootcheck signatures.

The appropriate response in an operational environment is a documented, scoped tuning exception for those files, not disabling the rule. I documented this approach but did not implement the exception in this lab.

Roadmap

The lab is built to be extended. Planned next iterations:

• Same-source correlation Tighten rule 100002 with same_source_ip so it alerts only when the failures come from a single address, then retest against the same Hydra run.
• Automated response Use Wazuh active response to block the offending IP once the rule fires, taking the lab from detection to containment.
• Broader coverage Add further ATT&CK techniques and log sources, such as file integrity monitoring and privilege escalation scenarios.
• Tuning Implement the scoped exception for the rootcheck false positive and document the before and after alert volume.

Takeaways

The lab covered the full detection cycle: collecting endpoint logs in a SIEM, writing a rule for attacker behavior, validating it against simulated attacks, and triaging the alerts, including a false positive.

Which attack techniques would you simulate next in a lab like this? Share your ideas in the comments!

Comments

[Purvansh Bhatt]

Great write-up! Breaking down the rootcheck false positive triage and showing how to avoid alert fatigue with the custom frequency rule was super practical. For the next technique, testing cron persistence (T1053) against Wazuh’s FIM would be an awesome extension.