- When to capture: triggers, scope, and privacy guardrails
- Capture strategies and
tcpdumpfilters that scale - Follow streams and decode obscure failures in Wireshark
- How to spot retransmits, packet loss, and latency in traces
- Practical Application: capture-to-RCA checklist and evidence packaging
Packets are the single most reliable instrument you have during a network incident — they show what was actually on the wire, with timestamps, sequence numbers, and protocol state. Treat capture discipline and a consistent evidence package as part of your incident playbook: the right pcap at the right scope converts speculation into reproducible root cause.
The problem you face in real incidents is threefold: you either don’t have packet evidence, you have too much evidence, or the evidence you have cannot be legally or safely shared. Symptoms look familiar — intermittent application timeouts that leave no trace in logs, user-reported slowdowns across geographies, or an SLA breach with no obvious root cause. Those symptoms demand precise, time-bounded captures and a defensible handling process so the PCAPs can be analyzed, shared, and archived without creating privacy or legal risk .
When to capture: triggers, scope, and privacy guardrails
Capture when the packet-level view will materially shorten MTTD/MTTK/MTTR: user-impacting outages, reproducible failures during a maintenance window, security incidents where content may show exfiltration, or when an SLA metric crosses a threshold and you need packet evidence for the provider conversation. Capture only after authorization and with the minimum scope necessary: time-box the capture, restrict endpoints, and prefer header-only captures if payload is not required. Formalize that authorization in your monitoring/IR policy and record it with the evidence package. NIST’s de-identification guidance and forensic readiness documents provide the framework for deciding when data must be de-identified and how to preserve chain-of-custody for network evidence .
Important: PCAPs commonly contain PII and credentials. Treat every capture as potentially sensitive: record who approved it, why it was taken, what filters were applied, and where the raw file will be stored. NISTIR 8053 discusses de-identification strategies you should consider before sharing traces externally .
| Trigger | Minimum capture scope | Retention guideline |
|---|---|---|
| Production outage impacting customers | Host(s) + upstream hop(s); 1–5 minutes before/after incident | Keep raw for short term; extract & redact for sharing; hash and archive per policy |
| Security incident (possible data exfil) | Full payload capture (preserve evidence) | Follow forensic chain-of-custody; legal counsel involved |
| Performance validation after deploy | Target service IPs/ports; header+payload for 60–300s | Summary retained, raw trimmed if not needed |
Cite the legal team when in doubt. In the U.S. and EU you may have obligations under wiretap, stored-communications, or data-protection laws; for operational troubleshooting you usually rely on internal monitoring/consent exceptions — but that should be explicit in policy and documented with each capture .
Capture strategies and tcpdump filters that scale
Capture strategy is a set of trade-offs: fidelity vs. size vs. privacy vs. capture health. Your toolset should include tcpdump (or dumpcap/tshark if you prefer the Wireshark toolchain), a robust BPF capture filter, snaplen sizing, buffer tuning, and rotation or ring buffers for long captures. Use capture-time filtering (BPF) to avoid a “haystack” of irrelevant packets — BPF runs in the kernel and prevents unnecessary packet copies to user space, which reduces kernel drops. The pcap/BPF syntax is expressive: host, net, port, portrange, and/or/not, and byte-offset expressions let you slice by almost any header field at capture-time .
Practical tcpdump knobs you’ll use constantly:
-
-i <iface>— interface to capture on. -
-s <snaplen>— snapshot length;-s 0typically means capture the full packet. Keep snaplen minimal if payload is not needed.-s 1500often captures full Ethernet frames without extra noise.-s 96captures headers only in many cases. Use-s 0only when full payload is required because larger snaplen increases processing cost and can cause drops. -
-B <KiB>— set libpcap buffer size (bigger for high-throughput links). -
-w <file>and-W/-C/-G— rotate by file count, size or time to avoid huge single files; use ring-buffer patterns for automated captures. -
--immediate-mode(or-U) — flush packets to disk immediately on some platforms (helps live pipelining). -
-Z <user>— drop privileges after opening interface (security best practice). - Use kernel-side BPF:
tcpdump -i eth0 -w /tmp/cap.pcap -s 1500 'host 10.0.0.10 and tcp port 443'— the filter is compiled into BPF so only matching packets are copied out .
Example patterns (capture-time BPF):
# Capture only traffic to/from a service IP and port (low-volume, high-fidelity)
sudo tcpdump -i eth0 -s 0 -B 4096 -w /var/captures/svc-20251201.pcap 'host 10.0.0.10 and tcp port 443' #
# Hourly rotating files, keep 24 files
sudo tcpdump -i eth0 -s 0 -B 8192 -w 'edge_%Y%m%d_%H%M%S.pcap' -G 3600 -W 24 'net 10.0.0.0/8 and tcp' #
A few practical notes from the field:
- Mirror/SPAN ports can overload a switch’s mirror queue and drop packets — measure
dropped by kernelcounters fromtcpdumpsummary and use larger capture buffers or hardware taps for high line-rate captures . - Avoid capturing at application endpoints when the process must remain pristine for legal or forensic reasons — prefer a passive tap or a dedicated capture appliance where possible.
- For high-performance environments use specialized capture NICs/SmartNICs or host kernel features (TPACKET_V3) and tune ring buffers;
tcpdump -Band--immediate-modematter here .
Follow streams and decode obscure failures in Wireshark
The fastest route from a pcap to an answer is isolating the flow and reading it as the application did. Use Wireshark’s Follow TCP Stream (or the tshark -q -z follow,tcp,... equivalent) to reconstruct the byte stream in correct sequence — this collapses retransmits/out-of-order packets into the application view and exposes protocol-level errors or application-layer timeouts .
When you select a packet and run Analyze → Follow → TCP Stream, Wireshark applies a display filter for just that tcp.stream and presents the reassembled payload. For scripted workflows, tshark -q -z follow,tcp,ascii,<stream> offers the same output on the CLI .
What to check in the initial triage of a TCP flow:
- Three-way handshake and options:
tcp.flags.syn==1will show the SYN; inspecttcp.options.mss,tcp.options.wscale, and whetherSACKis negotiated. Window scaling and SACK change how you interpret subsequent loss behavior. Use Wireshark’s TCP header tree or display filters liketcp.options.wscaleto expose these options. - Initial RTT sample: Wireshark exposes
tcp.analysis.initial_rttandtcp.analysis.ack_rttfields which you can export to CSV for histograms. Usetshark -r file -Y "tcp.analysis.ack_rtt" -T fields -e tcp.analysis.ack_rttto extract RTT samples for statistical analysis. - Application-level errors: reassembled payload often contains HTTP status codes, SQL errors, or application timing markers — viewing the stream in ASCII/hex will show the problem in sequence. If TLS is in use, supply the session keys (
SSLKEYLOGFILE) to Wireshark or configure decryption keys in preferences to reveal the HTTP layer.
Example: isolate a stream and export RTTs:
# isolate all TCP retransmissions for manual inspection
tshark -r full.pcap -Y "tcp.analysis.retransmission" -T fields -e frame.number -e tcp.stream -e ip.src -e ip.dst -e tcp.seq -E header=y -E separator=, > retrans.csv #
# extract ack RTTs for a client subnet into CSV for histogramming
tshark -r full.pcap -Y "tcp.analysis.ack_rtt and ip.dst==10.0.0.0/24" -T fields -e tcp.analysis.ack_rtt -E header=y -E separator=, > rtt_samples.csv #
How to spot retransmits, packet loss, and latency in traces
Wireshark’s tcp.analysis suite flags expected events: tcp.analysis.retransmission, tcp.analysis.fast_retransmission, tcp.analysis.spurious_retransmission, tcp.analysis.duplicate_ack, tcp.analysis.lost_segment, tcp.analysis.zero_window, and tcp.analysis.ack_rtt — these are your primary indicators when triaging loss and latency problems .
Practical triage steps for a degraded TCP flow:
- Confirm the handshake completed with expected MSS/Window options; if window scaling was not negotiated, advertised windows may be misleading. Use
tcp.flags.syn==1andtcp.stream eq <n>to get the context. - Look for
tcp.analysis.duplicate_ackfollowed bytcp.analysis.fast_retransmission— three duplicate ACKs generally trigger fast retransmit as defined by the TCP congestion-control RFCs. That threshold and fast-retransmit behavior are standardized in RFC 5681. - If retransmits appear without duplicate ACKs and with a long gap, you may be seeing RTO-driven retransmits; RTO computation and exponential backoff behavior are described in RFC 6298 — look for
tcp.analysis.rtoannotations and check whether retransmit doubling is happening . - Distinguish loss from reordering:
tcp.analysis.out_of_ordervstcp.analysis.retransmissionplustcp.analysis.spurious_retransmission— spurious retransmits indicate sender-side heuristics or RTO misestimation rather than real persistent loss.tcp.analysis.lost_segmentsuggests that Wireshark inferred missing packets (either not captured or truly lost).
Quick tshark diagnostics:
# count retransmits per 60s interval
tshark -r full.pcap -q -z "io,stat,60,COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission" #
# list flows with highest retransmit counts
tshark -r full.pcap -q -z conv,tcp | head -n 40 # inspect top TCP conversations by packets/bytes and spot retransmit-heavy flows #
Use timestamps carefully: multi-vantage captures must be time-synchronized (NTP/PTP). Wireshark supports time shift for traces when clocks are not synchronized; capture metadata should note the NTP state of each capture host .
Practical Application: capture-to-RCA checklist and evidence packaging
This is the operational checklist I use on real incidents — follow it sequentially and record each artifact. Use the tool examples alongside.
-
Authorization & context (documented)
- Who authorized the capture, the business reason, and the legal basis (operational monitoring, consent, incident response). Record this in
README.txt. Reference NIST guidance for evidence handling and forensics readiness .
- Who authorized the capture, the business reason, and the legal basis (operational monitoring, consent, incident response). Record this in
-
Capture plan (scope, snaplen, duration, filters)
- Decide
snaplen(-s) based on need (-s 1500headers+normal payload;-s 96headers-only;-s 0for full frames if required) and select a tight BPF. Kernel-side BPF is your first line of data reduction. Record the exact BPF expression.
- Decide
-
Live capture (use
tcpdumpordumpcapfor non-root capture)- Example live command (hourly rotated ring):
sudo tcpdump -i eth0 -s 1500 -B 8192 -w '/var/captures/edge_%Y%m%d_%H%M%S.pcap' -G 3600 -W 48 'host 10.0.0.10 and tcp port 443' #
-
Immediate verification
- Watch
tcpdumpsummary forpackets capturedvsdropped by kernel. Runcapinfos full.pcapto confirm earliest/latest timestamps, duration, and packet counts.capinfosyields metadata you should include in the evidence manifest.
- Watch
-
Trim to evidence window
- Use
editcap -A "<start time>" -B "<end time>"to extract the incident window andeditcap -s <snaplen>to truncate payload if sharing is necessary. Add a capture comment or README viaeditcap --capture-comment "Authorized by ..."to embed context in the file (pcapng supports comments).
- Use
Example:
# extract time window and reduce payload to headers
editcap -A "2025-12-01 10:02:30" -B "2025-12-01 10:07:00" full.pcap incident-window.pcap
editcap -s 128 incident-window.pcap incident-window-trimmed.pcap #
- Integrity & provenance
- Compute cryptographic hashes and sign them if required:
sha256sum incident-window-trimmed.pcap > incident-window-trimmed.pcap.sha256
ls -l --full-time incident-window-trimmed.pcap > incident-fileinfo.txt
- Record capture host,
tcpdump/tsharkversions (tcpdump --version,tshark -v), interface name, NIC driver & timestamping mode (ethtool -i eth0), and the exact capture command line inREADME.txt. NIST SP 800-86 explains documenting and protecting forensic evidence as part of incident response.
-
Merging and multi-vantage correlation
- If you captured at multiple points, time-shift if needed with
editcap -tand merge withmergecap -w merged.pcap a.pcap b-shifted.pcap. Includecapinfosoutputs for each source in the package so recipients can validate timestamps and offsets.
- If you captured at multiple points, time-shift if needed with
-
Analysis exports for the RCA package
- Export the isolated flow, the follow-stream dump, CSVs of RTTs or retransmits, and a short narrative with packet references (frame numbers) that support each claim. Use
tsharkto produce CSV data andcapinfosfor metadata.
- Export the isolated flow, the follow-stream dump, CSVs of RTTs or retransmits, and a short narrative with packet references (frame numbers) that support each claim. Use
# single-stream pcap and follow output
tshark -r full.pcap -Y "tcp.stream eq 42" -w stream-42.pcap # isolate flow
tshark -r stream-42.pcap -q -z follow,tcp,ascii,0 > stream-42-follow.txt # human readable reassembly
-
Redaction & de-identification before sharing
- If the file contains PII, anonymize or redact before sharing externally. Follow NISTIR 8053 recommendations on de-identification and document the de-identification method used (what fields were removed/pseudonymized). Tools such as
editcap(snaplen truncation) or specialized anonymizers (prefix-preserving IP anonymizers) are commonly used; the key is to preserve analytic value while removing identifiers .
- If the file contains PII, anonymize or redact before sharing externally. Follow NISTIR 8053 recommendations on de-identification and document the de-identification method used (what fields were removed/pseudonymized). Tools such as
-
Package & deliver
- Create a zipped evidence bundle that contains:
-
incident-window-trimmed.pcap(or sanitized pcap) incident-window-trimmed.pcap.sha256-
README.txtwith command-line, authorizations, capture host and time, and high-level findings -
capinfosoutputs and CSV exports for RTT/retransmit metrics - short RCA narrative with references to
frame.numberentries
-
- Keep the raw (unsanitized) capture in a secured evidence store per your retention policy; share only the sanitized package externally.
- Create a zipped evidence bundle that contains:
Callout: Use
capinfosto produce a one-line metadata summary and include it with every evidence package.capinfosprovides packet counts, duration, first/last timestamps, and capture comment fields that are invaluable for verifying what was shared .
Final word
Collecting packets deliberately — authorized, scoped, and well-documented — turns chaotic incident reports into reproducible RCAs and defensible evidence. Make tcpdump your capture workhorse, use BPF to reduce noise at the kernel level, use Wireshark/tshark to follow streams and run tcp.analysis checks, and package every pcap with metadata and hashes so your findings are repeatable and shareable under privacy and legal constraints .
Sources:
De-Identification of Personal Information (NISTIR 8053) - Guidance on de-identification techniques and considerations for sharing sensitive data drawn from captures.
Guide to Integrating Forensic Techniques into Incident Response (NIST SP 800-86) - Forensic readiness, evidence handling, and chain-of-custody practices used to justify packaging and retention steps.
tcpdump(1) manual (man7.org) - tcpdump options and runtime behavior referenced for -s, -B, -w, -G, rotation, and buffer sizing.
pcap-filter(7) – BPF syntax (man7.org) - Capture-time filter syntax and kernel-side advantages for BPF expressions.
Wireshark User’s Guide — Following Protocol Streams - Explanation of Follow TCP Stream and time-reference features used in stream reconstruction and timestamp handling.
Wireshark Display Filter Reference: TCP - tcp.analysis.* fields and other TCP analysis flags referenced for retransmit/loss/RTT detection.
tshark(1) manual (Wireshark) - CLI equivalents for Follow TCP Stream, statistics exports, and scripted extraction examples.
editcap(1) manual (Wireshark) - Commands for trimming, snaplen adjustment, time slicing, and embedding capture comments in pcap/pcapng.
mergecap(1) manual (Wireshark) - Merging multiple captures, timestamp adjustments, and IDB handling for multi-vantage analysis.
capinfos(1) manual (Wireshark) - Metadata extraction used for evidence manifests (earliest/latest packet, counts, durations).
RFC 5681 — TCP Congestion Control - Standard behavior for fast retransmit/fast recovery and the three-duplicate-ACK heuristic referenced in analysis.
RFC 6298 — Computing TCP's Retransmission Timer - RTO computation and exponential backoff information cited when interpreting RTO-based retransmits.
Top comments (0)