DEV Community

Timothy Imanuel
Timothy Imanuel

Posted on

Network Forensics: Statistical Flow Analysis – Reading the Story in Network Metadata (Week 6)

This week, we focus on statistical flow analysis: instead of examining every byte of every packet, investigators study summaries of network conversations (flow records) to spot compromised hosts, data leakage, and suspicious patterns of behaviour.

Why Flow Analysis?

Flow analysis helps investigators answer key questions about network activity:

  • Identify Compromised Hosts: Compromised systems often send out more traffic than usual, use unusual ports, or communicate with known malicious systems.
  • Confirm or Disprove Data Leakage: The volume of exported data can show whether sensitive information has left the network.
  • Individual Profiling: Flow data can reveal a user's normal working hours, periods of inactivity, and sources of entertainment.
  • Correlate Activity: Exchanges between hosts can be linked together to build a timeline of events.

A flow record is a subset of information about a flow. It typically includes the source and destination IP address, source and destination port (where applicable), protocol, date, time, and the amount of data transmitted.

Flow Record Processing System

A flow record processing system is made up of four components:

  • Sensor: The device that monitors traffic on a network segment and extracts the important details into a flow record.
  • Collector: One or more servers that listen on the network for flow record data and store it to disk.
  • Aggregator: When multiple collectors are used, their data is combined on a central server for analysis.
  • Analysis: Once exported and stored, the flow data can be analysed with commercial, open-source, or homegrown tools.

Sensors

  • Network Equipment: Many switches support flow record creation and export (e.g., Cisco uses NetFlow, while SonicWall supports both IPFIX and NetFlow). Be cautious of "sampling," as sampled data is not comprehensive.
  • Standalone Appliances: Used when existing network equipment does not support flow data.
  • Argus (Audit Record Generation and Utilization System): Comes as two packages (Argus Server and Argus Client), is libpcap-based, supports BPF filtering, and its documentation specifically mentions forensic investigation. It exports data in Argus' compressed format over UDP.
  • Softflowd: Passively monitors traffic and exports records in NetFlow format. It runs on Linux and OpenBSD and is libpcap-based.
  • Yaf (Yet Another Flowmeter): Works with libpcap and live packet transfer, exports in IPFIX format over SCTP, TCP, or UDP, and supports BPF filters.

Sensor Placement & Modifying the Environment

Investigators often have little control over sensor placement, since most infrastructures are not built with flow monitoring in mind.

  • Placement Factors: Minimise duplication, keep time synchronised across devices, and remember that collecting only on external devices like firewalls ignores valuable internal traffic. Prioritise resources and avoid overloading network capacity.
  • Leverage Existing Equipment: Switches, routers, firewalls, and NIDS/NIPS can often already produce flow data.
  • Upgrade Equipment: If existing hardware will not work, deploy replacements.
  • Deploy Additional Sensors: Use port mirroring or a network tap to send packets to a standalone sensor.

Flow Record Export Protocols

  • NetFlow: Cisco's proprietary format. It maintains a cache tracking the state of all active flows; completed flows are marked as "expired" and sent to a collector as a NetFlow Export packet. Newer versions (NetFlow v9) are transport-independent (UDP, TCP, and SCTP), while older versions only support UDP and IPv4.
  • IPFIX: The open standard that extends NetFlow v9. It handles bidirectional flow reporting, reduces redundancy, improves interoperability, and uses templates that define which data the sensor exports.
  • sFlow: Supported by many devices (though not Cisco). It uses statistical packet sampling rather than recording every packet, so while it scales very well, it is generally not well suited for forensic analysis.

Collection and Aggregation

When deciding where to place collectors, investigators should consider:

  • Congestion: Flow records generate their own traffic, so choose a location with low network impact.
  • Security: Export flow records on a separate VLAN where possible, isolate physical cables, and encrypt using IPsec or TLS.
  • Reliability: Consider using TCP or SCTP instead of UDP.
  • Capacity & Strategy: Decide whether one sensor or many is needed, and let the analysis strategy shape all of the above.

Commercial collectors include Cisco NetFlow Collector, ManageEngine NetFlow Analyzer, and WatchPoint NetFlow Collector. Open-source options include SiLK (the most powerful but with the steepest learning curve, using flowcap and rwflowpack), flow-tools (modular and extensible, UDP input only), nfdump/NfSen (using the nfcapd collector daemon with UDP sockets or pcap files), and Argus (supporting Argus format, NetFlow v1–9, and IPFIX).

Flow Record Analysis Techniques

Flow records store a summary of the traffic crossing a network. Forensic data carving does not apply here, but the data is still extremely valuable. Analysis should be shaped by your goals and available resources (time, staff, equipment, and tools), and usually begins with a triggering event such as the IP address of a compromised system, a time frame, known ports, or unexplained flows.

  • Filtering: Narrow down a large pool of evidence by removing extraneous data, isolating activity related to specific IP addresses, and filtering for known patterns of behaviour.
  • Baselining: Because flow data is much smaller than full packet captures, it can be retained for longer. This allows investigators to build network baselines (general trends over time) and host baselines, since most flow patterns change dramatically when a host is compromised or under attack.
  • "Dirty Values": Search for suspicious keywords, IP addresses, ports, and protocols.
  • Activity Pattern Matching: Examine IP addresses (internal or Internet-facing, country of origin, registered owner), ports (well-known services, scanning activity), protocols and flags (connection attempts, successful scans, data transfers), directionality (downloads vs. uploads), and volume (many small packets can indicate port scanning, while large transfers are cause for concern).

Simple and Complex Patterns

  • Many-to-One: A DoS attack, a syslog server, a "drop box" data repository, or an email server at the destination.
  • One-to-Many: A web server, an email server at the source, a spam bot, a warez server, or network port scanning.
  • Many-to-Many: Peer-to-peer file sharing or widespread port scanning.
  • One-to-One: A targeted attack or routine server communication.
  • Fingerprinting (Complex Patterns): Matching complex flow patterns to specific activities. For example, a TCP SYN port scan shows one source IP, one or more destination IPs, incrementally increasing destination ports, a high packet volume in a short time, and outbound TCP flags set to SYN.

Flow Record Analysis Tools

  • SiLK: rwfilter extracts flows of interest by time and category (generally as functional as BPF), while rwstats, rwcount, rwcut, and rwuniq handle basic manipulation. rwidsquery takes a Snort rule or alert file and builds a matching rwfilter, and rwpmatch saves only the packets that match given flow metadata. Advanced users can script it with the PySiLK Python interpreter.
  • Flow-tools: Covers collection, storage, processing, and sending. Key tools include flow-report (ASCII reports), flow-nfilter (filtering), and flow-dscan (identifying suspicious traffic).
  • Argus Client Tools: ra reads, filters, and prints Argus data with BPF-style filter expressions (e.g., src host, dst host); racluster aggregates records; rasort sorts them; ragrep performs regular expression matching; and rahisto generates frequency distributions for metrics like flow duration, ports, bytes, and packet counts. A basic capture looks like argus -w ip_capture_file - ip and not icmp, and settings such as ARGUS_INTERFACE=eth0 live in /etc/argus.conf.
  • FlowTraq: A commercial tool by ProQueSys that supports many formats and can sniff traffic directly. It lets users filter, search, sort, and produce reports, and is designed for forensics and incident response.
  • Nfdump/NfSen: Nfdump aggregates flow records by specific fields, limits by time range, generates statistics (IP addresses, interfaces, ports), anonymises IPs, and supports BPF-style filters. NfSen provides a graphical, web-based interface on top of it.
  • EtherApe: A libpcap-based graphical tool that visually displays network activity in real time, with colours designating protocols such as HTTP, SMB, ICMP, and IMAPS. Note that it does not take flow records as input.

Reference: Davidoff, S., & Ham, J. (2012). Network Forensics: Tracking Hackers Through Cyberspace. Boston: Prentice Hall.

Top comments (0)