Building a Network Intrusion Detection System (NIDS) with Snort on Linux: A Complete Hands-on Guide

1. Introduction: In today’s rapidly evolving digital environment, organizations face a growing number of cyber threats such as port scans, brute-force attacks, denial-of-service (DoS) attacks, and web-based exploitation attempts. Detecting these threats in real time is essential for maintaining network security and system availability.

This project focuses on the design and implementation of a Network Intrusion Detection System (NIDS) using Snort and Wireshark on Ubuntu Linux. The main goal of this project was to monitor network traffic, identify suspicious activities, and generate alerts based on custom-defined detection rules.

Through this implementation, I gained hands-on experience in network traffic analysis, intrusion detection, Snort rule writing, and basic incident response mechanisms.

1. Objectives: The main objectives of this project were:

To install and configure Snort IDS on Ubuntu Linux
To monitor live network traffic for malicious activity
To create and test custom Snort rules
To detect common network attacks in a lab environment
To analyze suspicious packets using Wireshark
To perform basic incident response actions after attack detection

1. Lab Environment: The project was implemented in a controlled lab environment with the following specifications:

Component Specification
Operating System Ubuntu 22.04 LTS
RAM 4 GB
CPU 2 Cores
Storage 20 GB Free Space
IDS Tool Snort
Packet Analyzer Wireshark

1. Project Architecture: The architecture of this project consists of an attacking system, a target monitoring system, and traffic inspection tools.

Attacker Machine (Kali Linux / Test System)
|
| (ICMP / TCP / HTTP / SSH Attack Traffic)
v
Target Ubuntu Monitoring System
├── Snort IDS
├── Custom Detection Rules
├── Wireshark Packet Analysis
├── iptables Firewall
└── fail2ban (SSH Protection)

This setup allowed me to generate controlled attack traffic and verify whether the intrusion detection system was able to identify suspicious behavior successfully.

1. Tools and Technologies Used: The following tools and technologies were used during this project:

Snort – for real-time intrusion detection
Wireshark – for packet capture and traffic analysis
Ubuntu Linux – as the deployment environment
iptables – for firewall-based blocking
fail2ban – for brute-force mitigation
Nmap – for port scan simulation
Hydra – for SSH brute-force simulation
ping / flood traffic – for ICMP flood testing
sqlmap – for SQL injection testing

1. Installation and Configuration:
   6.1 System Update:
   The system was first updated to ensure all packages were current.

   cmd: sudo apt update && sudo apt upgrade -y`

6.2 Install Required Dependencies:
The necessary libraries and development tools required for Snort were installed using the following command:

cmd: sudo apt install -y build-essential libpcap-dev libpcre3-dev \
cmd: libdumbnet-dev bison flex libnghttp2-dev zlib1g-dev \
cmd: libssl-dev libnetfilter-queue-dev libdnet-dev

6.3 Install Snort:
cmd: Snort was installed and verified using:
cmd: sudo apt install snort -y
cmd: snort -V

This step confirmed that Snort was successfully installed and ready for configuration.

6.4 Install Wireshark:
Wireshark was installed for network traffic capture and packet inspection.

cmd: sudo apt install wireshark -y
cmd: sudo dpkg-reconfigure wireshark-common
cmd: sudo usermod -a -G wireshark $USER

Wireshark was used throughout the project to visually inspect suspicious packets and confirm Snort alerts.

1. Snort Rule Configuration:
   7.1 Understanding Snort Rule Structure:
   Each Snort rule consists of two main parts:

   Rule Header → Defines traffic conditions
   Rule Options → Defines alert message, classification, SID, thresholds, and payload matching

General Snort rule format:

 [action] [protocol] [source IP] [source port] -> [destination IP] [destination port] (options)

7.2 Creating a Custom Rule File:
A custom rule file was created to store locally written detection rules.

  cmd: sudo nano /etc/snort/local.rules

1. Custom Detection Rules: The following custom Snort rules were created to detect common attack patterns.

8.1 ICMP Flood Detection (DoS Attack):
This rule detects excessive ICMP echo requests from the same source within a short time window.

alert icmp $EXTERNAL_NET any -> $HOME_NET any (
    msg:"ICMP FLOOD ATTACK DETECTED";
    threshold:type both, track by_src, count 100, seconds 5;
    classtype:attempted-dos;
    sid:1000001;
    rev:1;
)

Purpose:
  Detects possible ICMP flood or ping flood attacks.

8.2 Nmap SYN Port Scan Detection:
This rule detects rapid SYN packets sent to the monitored host, which may indicate an Nmap SYN scan.

alert tcp $EXTERNAL_NET any -> $HOME_NET any (
    msg:"NMAP SYN SCAN DETECTED";
    flags:S;
    threshold:type both, track by_src, count 20, seconds 10;
    classtype:attempted-recon;
    sid:1000004;
    rev:1;
)

Purpose:
  Detects suspicious reconnaissance activity such as port scanning.

8.3 SSH Brute Force Detection:
This rule monitors repeated connection attempts targeting the SSH service.

alert tcp $EXTERNAL_NET any -> $HOME_NET 22 (
    msg:"SSH BRUTE FORCE ATTEMPT";
    flow:to_server,established;
    detection_filter:track by_src, count 10, seconds 60;
    classtype:attempted-recon;
    sid:1000009;
    rev:1;
)

Purpose:
   Detects repeated SSH authentication attempts from a single source IP.

8.4 SQL Injection Detection:
This rule attempts to identify common SQL injection patterns in HTTP requests.

alert tcp $EXTERNAL_NET any -> $HOME_NET 80 (
    msg:"SQL INJECTION ATTEMPT";
    flow:to_server,established;
    content:"union"; http_uri; nocase;
    content:"select"; http_uri; nocase;
    classtype:web-application-attack;
    sid:1000012;
    rev:2;
)

Purpose:
  Detects simple SQL injection attempts in web traffic.

Note: This rule was implemented for educational demonstration purposes and would require more tuning for real-world production environments.

8.5 Cross-Site Scripting (XSS) Detection:
This rule detects possible XSS payloads in HTTP requests.

alert tcp $EXTERNAL_NET any -> $HOME_NET 80 (
    msg:"CROSS-SITE SCRIPTING (XSS) ATTEMPT";
    flow:to_server,established;
    content:"<script"; http_uri; nocase;
    pcre:"/(\<script|\%3Cscript)/Ui";
    classtype:web-application-attack;
    sid:1000013;
    rev:2;
)

Purpose:
  Detects simple reflected or encoded XSS payload attempts.

1. Including Custom Rules in Snort Configuration:
   The local rule file was included in the Snort configuration file:

   cmd: sudo nano /etc/snort/snort.conf

The following line was added:

 include $RULE_PATH/local.rules

This allowed Snort to load the custom detection rules during execution.

1. Running Snort in IDS Mode:
   Snort was executed in IDS mode to inspect live traffic and generate alerts in real time.

   cmd: sudo snort -A console -q -c /etc/snort/snort.conf -i eth0
   Explanation:
   -A console → Displays alerts in terminal
   -q → Quiet mode
   -c → Specifies configuration file
   -i eth0 → Selects the network interface

2. Attack Simulation and Testing:
   To verify the effectiveness of the intrusion detection system, several attack simulations were performed in a controlled environment.

   11.1 Port Scan Detection Test:
   A SYN scan was launched using Nmap:

    cmd: nmap -sS [TARGET_IP]
   Expected Snort Alert
    [**] [1:1000004:1] NMAP SYN SCAN DETECTED [**]
    [Priority: 3]
    04/03 14:30:12 192.168.1.100 -> 192.168.1.10
   

   11.2 ICMP Flood Test:
   An ICMP flood was generated using:

     cmd: ping -f [TARGET_IP]
    Expected Outcome:
       Snort generated repeated alerts once the threshold was crossed.
   

   11.3 SSH Brute Force Test:
   A brute-force attempt was simulated using Hydra:

    cmd: hydra -l root -P pass.txt ssh://[TARGET_IP]
    Expected Outcome:
       Snort detected repeated SSH connection attempts from the attacking source.
   

   11.4 SQL Injection Test:
   SQL injection traffic was simulated using:

      cmd: sqlmap -u "http://[TARGET_IP]/page?id=1"
    Expected Outcome:
      Snort generated an alert when matching suspicious SQL keywords were identified.
   

3. Packet Analysis with Wireshark:
   Wireshark was used to inspect and validate the traffic that triggered Snort alerts.

Traffic Analysis Activities Performed
Observed SYN packets during Nmap scanning
Monitored repeated ICMP echo requests during flood testing
Analyzed SSH traffic patterns during brute-force attempts
Inspected suspicious HTTP requests containing SQLi/XSS payloads

Using Wireshark alongside Snort improved visibility into the exact packet behavior and helped confirm whether alerts were legitimate or false positives.

1. Detection Results: The custom intrusion detection system successfully identified multiple attack patterns during testing.

Attack Type Command Used Detection Status
ICMP Flood ping -f target Detected
Port Scan nmap -sS target Detected
SSH Brute Force hydra -l root -P pass.txt ssh://target Detected
SQL Injection sqlmap -u "http://target/page?id=1" Detected

The results demonstrate that Snort, when configured with custom rules, can effectively detect suspicious traffic patterns in a controlled lab setup.

1. Incident Response Actions Taken:
   After attack detection, basic incident response measures were applied to reduce further malicious activity.

   14.1 Blocking Malicious IP Addresses:
   The attacking IP was blocked using iptables:

     cmd: sudo iptables -A INPUT -s 192.168.1.100 -j DROP
   

   14.2 Rate Limiting ICMP Requests:
   To mitigate ICMP flood attacks, rate limiting was applied:

       cmd: sudo iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/second -j ACCEPT
   

   14.3 Enabling SSH Protection with fail2ban:
   fail2ban was installed and enabled to reduce SSH brute-force attempts.

          cmd: sudo apt install fail2ban -y
          cmd: sudo systemctl enable fail2ban
          cmd: sudo systemctl start fail2ban
   

   14.4 Viewing Firewall Rules:
   Firewall rules were verified using:

           cmd: sudo iptables -L -n
   

2. Challenges Faced:
   During the implementation of this project, several practical challenges were encountered:

Configuring Snort correctly on Ubuntu required careful attention to rule paths and interface settings
Some initial rules generated noisy alerts and required threshold tuning
Application-layer attacks such as SQL injection and XSS were more difficult to detect reliably than network-layer attacks
Testing needed to be performed in a controlled environment to avoid accidental impact on normal traffic

These challenges helped improve my understanding of real-world intrusion detection limitations and the importance of tuning.

1. Limitations: Although the project was successful, it also had some limitations:

The IDS was tested only in a small lab environment
Detection logic for web attacks was basic and signature-based
Some attacks may evade detection through encoding or obfuscation
The system currently focuses mainly on detection, not full automated prevention
Performance under high enterprise-level traffic was not evaluated

These limitations highlight areas for future improvement.

1. Future Improvements: This project can be further enhanced in several ways:

Integrate ELK Stack for centralized logging and visualization
Add Zeek for deeper protocol and behavioral analysis
Automate response actions using Python scripts
Configure email or Slack alerts for real-time notifications
Improve rule quality using more advanced Snort signatures
Store and correlate alerts for better incident investigation

1. Key Learning Outcomes: This project provided strong practical exposure to several core cybersecurity concepts.

Skills Gained
Hands-on experience with Snort IDS
Network traffic analysis using Wireshark
Writing and tuning custom detection rules
Understanding attack patterns such as:
Port scanning
ICMP flood attacks
SSH brute-force attempts
Basic web attack payloads
Performing basic incident response
Applying layered defense concepts using IDS + firewall + fail2ban

1. Conclusion: This project demonstrated how a Network Intrusion Detection System (NIDS) can be designed and implemented using Snort and Wireshark in a Linux environment.

By configuring Snort, writing custom detection rules, simulating attack scenarios, analyzing traffic, and applying response measures, this project provided practical exposure to real-world network defense techniques. It also emphasized the importance of traffic visibility, rule tuning, and defense-in-depth in cybersecurity operations.

Overall, this project significantly improved my understanding of network monitoring, threat detection, and incident response, and it served as a valuable hands-on learning experience in cybersecurity.

1. References: i. Snort Official Documentation ii. Wireshark Official Documentation iii. Nmap Documentation iv. Hydra Tool Documentation v. sqlmap Documentation