Packet-Level Security: How to Monitor and Detect Network Attacks with Nmap and Wireshark

A Step-by-Step Lab for Understanding SYN Scans, Session Flow, and Implementing IDS Logic
Introduction

Network security is fundamentally a battle fought at the packet level. This write-up details a hands-on lab designed to move beyond theory and directly observe the signatures of network activity—both normal and potentially malicious—using powerful tools like Nmap, tcpdump, and Wireshark. Our objective was simple: simulate a basic scan and normal traffic, analyze the results, and lay the groundwork for effective Intrusion Detection System (IDS) rule creation.

Lab Setup and Execution

Our setup involved a Kali Linux machine acting as both the attacker and the monitor, targeting a server at 192.168.1.16.

Execution Steps:

1. Start Packet Capture (The Monitor): We initiated a raw capture on the monitor interface to ensure we didn't miss a single frame. tcpdump -i wlan0 -w capture.pcap
2. Run Nmap Scan (The Attack Simulation): A stealthy SYN scan was performed across common ports. The -sS flag sends SYN packets but does not complete the handshake, making it harder to log but easy to detect at the packet level. nmap -sS -p1-1000 -T4 -oN nmap_scan.txt 192.168.1.16
3. Generate Normal Traffic: To establish a baseline, we generated legitimate web traffic (HTTP/HTTPS) and started an SSH session (ssh user@192.168.1.16).
4. Testing for Anomalies: Small, repeated requests were sent to generate a pattern that simulates rapid interaction or an aggressive test condition.

Results and PCAP Analysis (Wireshark Deep Dive)

The Nmap results confirmed open ports 22, 80, and 443. However, the real insights came from the capture.pcap file analyzed in Wireshark.

Traffic Type    Wireshark Observation   Security Implication
Nmap Scan    Numerous SYN packets without corresponding ACK flags.  High Confidence Scan: The attacker is probing without completing the TCP handshake, a classic SYN scan.
HTTPS Traffic   TLS Client Hello from 192.168.1.13 to 192.168.1.16. Normal Encrypted Session Start: Expected handshake for secure web traffic.
SSH Session End A RST, ACK flag observed from the server (port 22) to the client port.  Abrupt Connection Close: Indicates the session was forcefully ended (e.g., terminal closed, network error, or firewall reset).
Large Data  Frames of 1460 / 1514 bytes.    Normal Data Flow: Standard large TCP segments (payloads) being transferred.
Test Conditions A few TLS Handshake Failure events. Anomaly: May indicate a configuration error (cipher mismatch) or aggressive testing environment.

Designing a Simple IDS Rule

Based on the SYN scan observation, we can create a foundational rule for an IDS like Suricata or Snort to detect the initial signs of a SYN flood or aggressive port scan:

alert tcp any any -> $HOME_NET any (msg:"Possible Aggressive SYN Scan Detected"; flags:S; threshold:type threshold, track by_src, count 15, seconds 5; sid:1000001; rev:1;)

Interpretation: If any single source IP (track by_src) sends 15 or more SYN packets (flags:S) in a 5-second interval, trigger this alert. This provides immediate, low-false-positive detection for scanning activity.

Mitigation and Defense

Effective network security relies on multiple layers:

• IDS/IPS: Deploy Suricata or Snort and constantly tune custom rules.
• Segmentation (VLANs): Separate your development/test environment from production networks.
• Firewall Hardening: Implement explicit DENY ALL rules, only allowing required ports (22, 80, 443, etc.).
• Rate Limiting: Configure firewalls or load balancers to limit the number of new SYN connections per source IP (e.g., using SYN cookies).
• Centralized Logging (SIEM): Ship all IDS alerts and system logs to a central location for correlation and long-term analysis.

Conclusion

This lab demonstrated that the true nature of an attack is revealed in the network packets. By understanding TCP/IP flags and protocol sequences, security professionals can write precise detection rules and build robust, proactive defense strategies.