Building a SOC lab is more useful when the components are actually configured, tested, and troubleshot rather than simply installed.
For this stage of the lab, I set up a small virtualized environment in VMware Workstation: a Windows 11 endpoint feeding telemetry into Splunk Enterprise running on Xubuntu, with Sysmon added on top for more useful endpoint data. Along the way, the Universal Forwarder ran into a configuration problem that traced back to something as small as a file extension. Chasing it down taught me more about how the pipeline actually fits together than the install itself did.
1. Lab Architecture
The lab currently runs on two VMs.
SIEM
Xubuntu
Hostname: soc-siem
IP address: 192.168.242.128
Role: Splunk Enterprise
Splunk Enterprise listens for forwarded data on TCP 9997.
Endpoint
Windows 11
Hostname: WIN11-TARGET
IP address: 192.168.242.129
Role: Windows endpoint
This VM runs the Splunk Universal Forwarder and Sysmon. The Forwarder picks up selected Windows Event Log channels and ships them to the SIEM.
Architecture
I mapped the actual data flow in draw.io rather than reusing a generic SOC diagram. It only has two boxes right now, but I wanted the diagram to represent the lab as it exists, not as it will eventually look.
Sysmon runs alongside the standard Windows Event Log channels on the endpoint, and the Forwarder collects from both.
2. Installing the Splunk Universal Forwarder
Installing the Forwarder on the Windows 11 endpoint was the easy part. During setup, I pointed it at the SIEM directly:
192.168.242.128
with the receiving port:
9997
At this point I hadn't actually verified the network path. I'd just assumed that because the installer accepted the IP and port without complaint, the connection would work. That assumption turned out to be correct, but it's worth calling out because it's exactly the kind of thing you shouldn't take on faith once something isn't working later.
3. Configuring Windows Event Log Inputs
Installing the Forwarder doesn't mean anything gets collected until you tell it what to watch. That's handled through inputs.conf.
On the endpoint, I opened Notepad as Administrator and edited:
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf
with these inputs:
[WinEventLog://Application]
disabled = 0
[WinEventLog://Security]
disabled = 0
[WinEventLog://System]
disabled = 0
[WinEventLog://Microsoft-Windows-PowerShell/Operational]
disabled = 0
I saved the file, restarted the Forwarder service, and moved on, expecting to see events in Splunk shortly after.
4. Troubleshooting the inputs.conf.txt Problem
I didn't see anything. No Application logs, no Security logs, nothing.
My first instinct was to assume the whole setup was broken somewhere. Maybe the Forwarder wasn't actually pointed at the right indexer, maybe the receiving port wasn't open on the Xubuntu side. Rather than start changing settings at random, I worked backward through the pipeline one layer at a time.
Checking the network first
Before touching any Splunk configuration, I wanted to rule out the network entirely. From an elevated PowerShell session on Windows:
Test-NetConnection -ComputerName 192.168.242.128 -Port 9997
TcpTestSucceeded : True
The connection was fine. Whatever was wrong, it wasn't the network, so I stopped looking there and turned to the endpoint's own configuration.
Finding the actual problem
I listed the contents of the Forwarder's local config directory:
Get-ChildItem "C:\Program Files\SplunkUniversalForwarder\etc\system\local"
There was no inputs.conf. Instead there was inputs.conf.txt (Notepad had quietly appended the extension when I saved), and, on top of that, a directory named inputs.conf that had gotten created at some point, presumably from an earlier attempt where something tried to write to a path that didn't exist yet. Splunk had never actually been reading the file I thought I'd configured.
It's a small mistake, but it's the kind that's easy to miss precisely because the file looks right in the editor. Nothing about the content was wrong. The name was.
Fixing it
Remove-Item "C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf" -Recurse -Force
Rename-Item "C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf.txt" "inputs.conf"
Restart-Service SplunkForwarder
Removed the stray directory, renamed the real config file into place, restarted the service. The fix itself took seconds. Finding it was the actual work.
What stuck with me from this wasn't the rename. It was the order I worked in: check the network before assuming the SIEM is broken, check the endpoint's local config before assuming Splunk itself is misbehaving. Isolating layers instead of guessing is the difference between a five-minute fix and an afternoon of randomly changing settings.
5. Verifying Windows Event Log Ingestion
With the correct inputs.conf in place, I generated some activity on the endpoint and went back to Splunk Web on the Xubuntu VM to check.
In Search & Reporting:
index=*
Windows Event Log data was there.
Pipeline confirmed, end to end:
Windows Event Logs
↓
Splunk Universal Forwarder
↓
TCP 9997
↓
Splunk Enterprise
↓
Splunk Search
6. Expanding Endpoint Telemetry with Sysmon
Standard Windows Event Logs get you a baseline, but they're limited for anything resembling real investigation work. Sysmon fills that gap: process creation, network connections, and more, depending on the config in use.
I added the Sysmon Operational channel to inputs.conf:
[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = 0
Restarted the Forwarder:
Restart-Service SplunkForwarder
Generated some activity, checked Splunk again.
Sysmon Event ID 1 alone (process creation) already gives a lot more to work with than a generic Windows log entry. It's the difference between knowing something happened and knowing what process did it, when, and from where.
7. What I Took Away From This
The setup itself wasn't hard. The useful part was everything around the mistake.
Configs can look correct and still be wrong. inputs.conf became inputs.conf.txt without any error, warning, or indication that something had gone sideways. The file looked fine in Notepad. It just wasn't the file Splunk was reading. That's a cheap lesson to learn on a home lab and a much more expensive one to learn on the job.
Isolating the failure mattered more than fixing it. Checking network reachability before touching Splunk's config saved me from chasing the wrong problem. If Test-NetConnection had failed, I'd have known immediately the issue was on the SIEM side or somewhere in between, not in a config file.
And more logs isn't the same as more capability. Sysmon gives me a lot more raw material to work with, but a pile of events sitting in an index doesn't do anything on its own. That's the next problem to solve, not this one.
8. Current Lab State
SIEM
Xubuntu, Splunk Enterprise, receiving on TCP 9997
Endpoint
Windows 11, Splunk Universal Forwarder, Windows Event Logs + Sysmon
Telemetry flow
Windows 11 -- Windows Event Logs -- Sysmon -- Splunk Universal Forwarder -- TCP 9997 -- Splunk Enterprise (Splunk Search & Reporting)
The pipeline works. Right now it's collecting logs, not doing anything with them. That's the actual gap to close next.
Next Steps
- Build SPL searches against the Windows and Sysmon data actually sitting in the index.
- Turn some of those searches into detections instead of one-off queries.
- Put together dashboards for endpoint activity.
- Bring Kali Linux into the lab as a controlled attack source.
- Generate activity against the Windows endpoint and see what it actually looks like in Splunk.
- Use that telemetry to investigate, not just observe.
Right now this is a working log pipeline. The next few articles are about turning it into something that behaves more like a SOC.
Top comments (3)
So Lovely !!!
well documented!!!
Thank you David