Day 1 of Cybersecurity

Introduction

The title of the blog is clickbait and it is not actually my day 1 of Cybersecurity. I have been using Linux as on my host machine for almost 2 years, I have a lot of experience with network engineering, I am studying for the CCNA exam as well as I did a lot of labs on packet tracer and worked with real life physical network equipment, I have a physical home lab where I run my own DNS server (pi-hole), and I worked a lot with UFW, AWS NACLs and Security Groups, I am also experienced in ssh hardening and server administration, I can secure servers and restrict access only to authorised individuals. I also know VirtualBox very well, including VirtualBox network adapters, inter VM networking, additionally using pfSense as a virtual router to connect multiple VirtualBox networks. I also did some CTFs here and there, some tryhackme and hackthebox labs, used Kali to learn penetration testing basics from PortSwigger academy. And I know Docker very well, and how applications work, I have some experience with backend programming.

So I am not a complete beginner to Cybersecurity. I have been doing a lot of things related to Cybersecurity. But day before yesterday is when I truly started getting deep into Cybersecurity. All my previous work involved some amount of Cybersecurity in it in some ways but this is the first time I started doing a 100% Cybersecurity only project. My college required students in the 7th semester to form a team and submit a project, a research-oriented one instead of an application-oriented one (like the previous semester's Capstone project). My friends Savitha, Srimathi and I formed a team and decided to do SIEM assistant, a chatbot to interface with an existing SIEM solution so that SOC analysts can talk to a chatbot in human language to make it write complex queries and the SOC analyst can review it and execute them right in the chat interface and see the filtered output. This would help save a lot of time for SOC analysts so that they don't have to spend a lot of time writing complex queries in a query language and debugging those queries and can instead focus on investigating the threats.

But to do this I first have to understand Wazuh very well so I decided to do a Cybersecurity home lab in Wazuh, and I started it around day before yesterday evening. This is the first pure Cybersecurity resume worthy standalone project I ever started. I learnt SIEM basics from the internet and got a grip of what Wazuh is all about and decided to do a basic lab fully in Virtualbox with a Wazuh server and some log sources and do some attacks with Kali and generate some logs and analyse them and document it. This blog post is not the technical write up or the build log of that lab but my experience getting into Cybersecurity. For the lab required for the SIEM project, it will be hosted fully in AWS using Terraform because we are showcasing the assistant part in our project so we need a one click deploy for the lab.

Designing the Network Architecture

I initially planned on having different types of log sources. A Windows desktop log source, debian desktop log source, a pfSense firewall log source additionally with IDS/IPS, and a webserver that hosts a as a vulnerable web application as a log source, and DNS server log source. And planned to launch recons and attacks on the network and the webserver and investigate all the logs from Wazuh.

Here is what the initial plan looked like.

I put all the log sources in an internal network, and used a pfSense VM to connect it to the public internet. I chose internal because I wanted it to be isolated from the host, and I didn't choose NAT Network because I wanted control over the router instead of letting VBox manage it for me, since that is a log source.

And I didn't want to put Kali attacker in the same internal network as I didn't want to log Kali's outbound internet access in the corporate SIEM lab for obvious reasons. So I put the Kali VM in another host only ethernet adapter network (here it doesn't matter if it is host only or internal), and connected it to the public internet via another pfSense VM.

And I attached another network adapter to both pfSense VMs and put it in a separate internal network for the point to point link because I will have to set static routing between them so I need predictable stable IP addresses, which is why I didn't set up bridged adapter on the public link for those pfSense VMs to talk directly because if I did those will be assigned via DHCP from my real world router. So for public internet access I set NAT on the WAN interfaces of both pfSense as it didn't really matter at this point, since both pfSense has one for its own LANs and one for the point to point link called OPT1.

I have a lot of VM base images from which I just clone if I need to do a lab with multiple VMs. The base images are pfSense, debian desktops and servers and etc. Every VM in the diagram except for Kali, pfSense and Windows were debian desktops and servers. After cloning all that, I set up the required network adapters on each device and assigned them the right network. Now all that was remaining is the host configuration.

pfSense behaves differently with non LAN adapters

So I booted up the pfSense VMs first and set up the interfaces which was easy enough and set the LAN and WAN adapters appropriately along with DHCP for the LANs, and set static IPs on both the OPT1. Now I tried to ping one pfSense from the other using the IP set in another one's OPT1 but it just didn't work for some reason. I searched about it and only LAN is allowed to do whatever it wants, for OPT1 a separate firewall allow rule has to be set for communication to succeed. Coming from Cisco IOS routers, this was the first obstacle in this whole lab. So I set up the firewall rule and now I was able to ping both pfSense from each other.

So I went ahead and booted up the servers and setup static IP, then I booted up the desktop VMs and checked for connectivity. Then I added static routes to both pfSense and then tried pinging the machines in the LAN from the Kali. It worked fine. By this time it was night so I decided to do the remaining part of the lab, which I thought would be just installing Wazuh server and setting up agents and log sources and doing some attacks and analysing the logs from the Wazuh server, the next day which would be yesterday and I went to sleep thinking it would be over.

The Next Day

Wazuh Installation and Re-installation on Debian server VM

So I booted up the all the VMs and tried to install Wazuh on the medium sized debian server I dedicated for Wazuh. I also made pi-hole resolve wazuh-server.internal to the Wazuh server's IP beforehand. I opened the Wazuh docs quickstart page, copied the command which would download and run the install script, and executed it on the Wazuh server. It said OS not supported and only some Ubuntu, RHEL and Amazon Linux were supported. It told me to pass a flag with the install script to ignore that, and since I thought the debian VM can handle it I did that. So it began the installation and I had to wait for a lot of time as the downloading and installing was slow. After around 80% was done, it aborted the process and removed all the installed components and said the server is not big enough to handle the Wazuh and self deleted itself. So I allocated more resources to that server so that it can handle Wazuh and restarted the installation process.

It said a previous installation exists but the previous time the script clearly said it was removed. And the script said to pass another flag to overwrite the existing installation and I did that but it couldn't overwrite it fully as certain processes and files weren't fully removed earlier and several ports that would be used by the Wazuh server were still being listened on by detached processes. There was no standardized way to completely remove every left over thing. So I decided to purge the entire Wazuh server VM and mind you every install and uninstall commands took a lot of time. So I deleted the entire VM and decided to setup a fresh Ubuntu server that is big enough to handle Wazuh.

Ubuntu static IP issue

So I downloaded the Ubuntu noble ISO which took a lot of time, I waited for it. And after it finished downloading, I installed a base image from which I planned to clone another VM for the Wazuh server so that I don't have to install Ubuntu whenever I need a server VM. The installation also took a lot more time, so I had to wait. By the time the installation was over it was afternoon after my lunch time.

After finishing my lunch I got back to work and tried to set up static IP for the Wazuh server. For debian servers I just had to edit /etc/network/interfaces and put something like

iface eth0 inet static
    address 192.168.1.50
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 1.1.1.1

And then reload the service called networking and boom it works out of the box. But for Ubuntu, I have to edit a complicated yaml file and the syntax is too annoying even one extra indentation would mess up the whole thing but in debian indentation was not an issue. So I copy pasted the yaml values after giving the IP addresses to chatpgpt and making it generate the YAML file and ran sudo netplan apply and it finally worked.

Ubuntu not seeing the whole disk

I tried to install Wazuh server on the bigger Ubuntu VM, and mind you I did allocate enough resources to it so that I can install Wazuh on it and I did tell it to take up the full disk when installing the Ubuntu ISO and I never selected the LVM option. So when I downloaded and executed the install script I had to wait for a lot of time again. And it did the exact same thing it like earlier it finished around 80% of its job and then said the system is not big enough to handle Wazuh and self deleted all the components it installed. What?

So I checked the disk space on the server and it turns out only a small portion of the entire disk is visible to the host OS because by default Ubuntu server wouldn't install it over the entire disk and would just set up a LVM that I have to manually extend to the whole disk. So I did that and restarted the install script. Which again complained about existing installations that I had to pass the overwrite flag again and it took a lot of time to download the packages and when it is supposed to install them it couldn't overwrite again due to left over files and services and detached processes listening on ports. It was not possible to investigate all of that and remove one by one as the scope of this lab is log analysis so I decided to abandon this current installation and restored a snapshot, thankfully I had a snapshot even if I didn't I would have cloned from that Ubuntu base server VM (which I still have to modify to make it take up the entire disk).

And the first thing I did was extend the LVM to take up the whole disk. After that since it was fresh, I downloaded and executed the install script again. Which took a lot of time so I waited. And after a lot of time the server was installed and the username and password were printed on the terminal, so I opened up wazuh-server.internal in the browser from the debian administrative desktop (as it is an internal network) and it worked when I logged in. So finally the server was installed.

Accidental VirtualBox basic mode

Approximately around this time I accidentally clicked on Basic mode in the per VM settings in VirtualBox and that changed the entire VirtualBox manager to basic mode so the clone VM option wasn't there and there was only one network adapter option on each VM and it only had 2 options NAT and Bridged. I was so confused and crashing out trying to find the clone button for 5 entire minutes and I stumbled on the basic/expert setting in per VM setting again and set it to expert and got all my buttons back, I was finally relieved.

Configuring pfSense+Suricata IDS/IPS as a log source

I wanted network monitoring and IDS deployed on the corporate network and I wanted the IDS logs on the Wazuh. So I opened pfSense dashboard, added the Suricata package and set it up to send eve-json logs to the system. Then I set up remote logging in pfSense and made it send logs to Wazuh's IP on port :514. And then on wazuh server I installed rsyslog and made it save whatever comes on :514 to a separate file pfSense.log and edited Wazuh's xml conf to monitor that log file as well.

Now how does Wazuh server know it is pfSense that is sending the logs and no logs are being edited by unauthorized sources before it reaches the server? I don't know. I decided it is not my problem.

And after Suricata and pfSense logging to remote server to was set up I refreshed Wazuh and pfSense did send some logs to the server and it was visible under the agent id 000. So whatever comes to :514 it gets read as agent 000's activity. So no matter how many network devices are set up to remote syslog, it is all under agent id 000. How do we group the logs based on the source and type and classify them? I have decided it is not my problem and in a real organization I would escalate this issue to the senior security analyst as the scope of this lab is to just do some log analysis on Wazuh not configure multiple log sources and aggregate them with precision.

So I went ahead and did a nmap aggressive scan on the entire corp subnet expecting Suricata to log something related to it and after the scan was over I tried to refresh Wazuh dashboard and found out nothing. Why is Suricata not sending nmap scan incident logs to Wazuh server? I don't know. Probably because I haven't configured the exact rules to detect it. I have decided that I don't care to since IDS/IPS configuration is out of the scope of this lab and decided to drop Suricata entirely so that I can spend time on the actual scope of the lab.

I opened the pfSense dashboard and removed the entire Suricata. Now just the regular firewall logs come to Wazuh under agent 000. Which was fine for now and I moved on to installing agents on the remaining hosts.

My PC hanged

I use Linux by the way on my bare metal host machine as my daily driver that is why it was able to support all these VMs so far. Right before I started setting up agents on the hosts my PC just hanged. It was because 8 separate VMs, one windows and one bulky Ubuntu and one Kali, along with Brave browser, obsidian, and vscodium, 3 separate chromium processes were running. I just left my computer and took a break. When I came back it was locked but it was working, so I unlocked it and then stopped Brave, Obsidian and VSCodium. Then I shutdown and purged the webserver as nothing useful was on it. And I shut down the Kali VM and its pfSense. Some RAM was freed. Then the PC was working fine.

pfSense DNS resolve host override subdomain field confusion

I also decided to purge the pi-hole VM and make pfSense resolve domain names without an external server. So I edited the DHCP settings to make pfSense as the DNS server. And then in the host overrides, to add a manually entry to resolve wazuh-server.internal to the Wazuh server's IP address I opened up that setting page in pfSense dashboard and there were 3 options

1. host name
2. Domain
3. IP

In pi-hole GUI there are only 2 fields, FQDN and IP so I thought the first field host name is about the OS host name of the target server and entered the values like

1. wazuh-server
2. wazuh-server.internal
3. 10.0.0.4

And then nslookup wazuh-server.internal failed but other public domains were working fine (after the DHCP settings have been reloaded in VMs that use DHCP and in the hosts that have static IP I manually edited it to use pfSense as its DNS server). I found out through the hard way the first field host name is a sub domain field. So currently wazuh-server.wazuh-server.internal is being resolved to 10.0.0.4 and wazuh-server.internal is not even being cared about.

This was really annoying because why would it make me enter the subdomain in a separate field if I am gonna have to make a new entry for another subdomain for the same top level domain? I figured out I was supposed to leave the first field empty if I wanted to resolve the main domain and I did that and finally internal DNS was back to normal.

Reducing the scope of this project

Since the DNS server and web server are gone, we will only be collecting logs from Debian desktop, Windows Desktop, and the firewall. So the new network architecture looks like this.

2 pfSense VMs have 1 GB RAM each, and Wazuh server and Windows have 8 GB RAM each, Debian Desktop has 2 GB RAM and Kali has 8 GB RAM. So in total it is 32 GB RAM if all these VMs operated at maximum capacity but fortunately they won't and I am also going to reduce the RAM of Kali and other things when I do the attacks and ssh brute force on the hosts. And I have to run OBS to record the attacks and logs generation and inspection so it is a big problem for future me.

Originally I planned to do webserver log monitoring and DNS server log monitoring so I can host a vulnerable web application on the webserver and attack it with Kali so I can have those logs in Wazuh too and also I planned to do DNS poisoning and capture logs from pi hole so another investigation can be done. Now due to my host's limitations I will only be doing File Integrity and Registry edit monitoring in windows and default monitoring in debian desktop and the only thing I will be doing with Kali is ssh brute force.

Deploying the agents

Deploying the agents on the hosts was the easy part, I literally had to enter some values in the Wazuh dashboard and I got the commands to copy paste on the host and thankfully it worked out of box in the debian desktop. I tried installing some packages and removing them and it was shown correctly in the Wazuh server and the login and logout details. Thankfully this part worked out of the box.

And I deployed an agent in Windows and did some registry edits and ran notepad as admin and edited some files, including the host files and hoped that it would show up in the Wazuh dashboard but nothing showed up. Turns out I have to install a separate thing on Windows for it to actually send all these data.

By this time it was night so I decided to do rest the next day and just went to sleep.

Conclusion

This is the moment that I realised, this is Cybersecurity. Yes, this. All this. I am a junior and I am having to deal with all of these in my first cybersecurity lab, so I am pretty sure professionals have it 1000 times worse. I have decided this is worth it. I am getting into Cybersecurity.

I also spent a lot of time documenting all these from my memory and made sure I didn't miss anything that happened yesterday because a lot of people that get into Cybersecurity and become a professional don't share their technical struggles. Maybe some on YouTube but I've only heard people say it is hard, I haven't seen anyone show the "why" it is hard publicly like this. I wanted to document it and share so that I can look back at this from the future and think "things were so easy back in the day", and recruiters if they stumble on this, they would know I am a guy that actually does things so I would be hire-able, and also people that are starting fresh could see this and other posts I will make in the future so they can see the reality of getting into Cybersecurity, the technical details part.

I am gonna go ahead and configure the agents in debian and Windows after posting this and do the brute force and log analysis and document that part too. The deadline for this lab is Monday so I do have a lot of time to do that. The deadline exists because we are supposed to show weekly progress report to our college.

Thanks for reading so far and goodbye.

Comments

[Shoban Chiddarth]

This is the longest post out of all my posts and I typed all this by hand without any AI assistance.