DEV Community: Robin Hayer

The tool said it wrote 48 packets. Only 44 were there

Robin Hayer — Mon, 21 Sep 2026 00:58:20 +0000

Exit code zero said success, but the packet count said otherwise.

Recap

I have been developing a tool wrapped around tshark. The first blog post was about hitting a wall on a 2.5 GB file. Later, I talked about parallelizing the PCAP processing in my second blog post where I ran into a file corruption bug.

Now here comes the real culprit: the PcapSplitter's FiveTupleSplitter caused file truncation/corruption on TCP session reuse (i.e., a new SYN packet arrives for an already tracked 5-tuple hash).

How I found it

Post two ended with me replacing PcapSplitter. This is what came before that replacement.

The initial signal came from "Total Block Length" errors thrown by tshark on some output files. That told me something was wrong, not what.

The actual signal came when I built a small reproduction. PcapSplitter reported 12 files and 48 packets, but on disk, there were 11 files and 44 packets. Exit code zero and printed "Finished" on standard output.

Theory One: File Descriptor Exhaustion, wrong

Someone on Reddit suggested file descriptor exhaustion. It was plausible with one output file per flow, where I had 95-125 flows, and a failed open() with an unchecked return would produce silent drops.

But instead of relying on theory, I tested it. Then I found that at low ulimit -n it silently drops most packets and still exits zero. Sharp threshold, reproducible. It wasn't my corruption. Raising the limit didn't fix the original problem.

Theory Two: Hardcoded LRU Limit, again wrong

After testing file descriptor exhaustion, I found that the PcapSplitter library has a hardcoded MAX_NUMBER_OF_CONCURRENT_OPEN_FILES = 250 with an LRU that closes and reopens handles past it. It was a perfect candidate.

I tested it too. It held up fine. My minimal reproduction was 13 connections, nowhere near the cap.

Two good theories, one found a different real bug, the other held up fine. Neither was mine.

What it was exactly

The correct part of the splitter was assigning a new file number when a TCP session reuses a 5-tuple, but the filename function builds the name from IP and port only. So, in this case, both sessions get the same filenames. Then main.cpp sees a file number it has never seen, and opens that file fresh, without append. This truncates the existing file or causes a race condition between two active file writer handlers.

1. File Number Allocation on Session Reuse (`ConnectionSplitters.h`)

When a fresh SYN arrives for a 5-tuple hash that has been seen before, FiveTupleSplitter::getFileNumber deliberately assigns a new file number to isolate the logical session:

if (isSyn && m_TcpFlowTable.find(hash) != m_TcpFlowTable.end() && m_TcpFlowTable[hash] == false)
{
    m_FlowTable[hash] = getNextFileNumber(filesToClose);   // <-- Allocates a NEW file number
}

2. The Filename Disconnect (`ConnectionSplitters.h`)

Right below this logic, FiveTupleSplitter::getFileName overrides the base class implementation but completely ignores the fileNumber parameter passed into it:

std::string getFileName(pcpp::Packet& packet, const std::string& outputPcapBasePath, int fileNumber)
{
    // ...
    sstream << "connection-";
    if (packet.isPacketOfType(pcpp::TCP)) {
        // ...
        // Filename is derived strictly from the packet's IP/Port values:
        updateStringStream(sstream, getSrcIPString(packet), srcPort, getDstIPString(packet), dstPort);
        return outputPcapBasePath + sstream.str();   // <-- fileNumber parameter is UNUSED on the TCP/UDP path
    }
    // ...
}

Because fileNumber is dropped, the new session produces the identical filename string as the older session.

3. The Truncation / Race Condition (`main.cpp`)

In main.cpp, the active writer cache map (outputFiles) is keyed by the integer fileNum, not the filepath string. When the newly allocated fileNum is evaluated, it goes down the fresh initialization branch:

// Since fileNum is a newly generated integer, it won't be found in the map
if (outputFiles.find(fileNum) == outputFiles.end()) 
{
    std::string fileName = splitter->getFileName(parsedPacket, outputPcapFileName, fileNum) + outputFileExtension;

    if (isReaderPcapng) {
        outputFiles[fileNum].reset(new pcpp::PcapNgFileWriterDevice(fileName));
    } else {
        outputFiles[fileNum].reset(new pcpp::PcapFileWriterDevice(fileName, rawPacket.getLinkLayerType()));
    }

    // CRITICAL: Plain open — no append flag — on a path that already holds valid session data!
    if (!outputFiles[fileNum]->open()) 
        break;
}

Impact & Consequences

File Truncation: If the previous file representing the same session was closed by the LRU mechanism, the new file writer triggers a standard file open on the existing path, that truncates the existing PCAP data captured from the previous session.
Write Race / Corruption: If duplicate files representing the same session are still open and active, it creates a race condition between two active file writer handlers. This breaks the linear layout of the PCAP/PCAPNG format and leaves the file corrupted.

Reporting the bug and fixing upstream

I filed the issue with a 13-connection reproduction in #2248.

The maintainer found his own commit 6379100 and said he couldn't remember why he'd made the change. He pushed back on my first fix, where I suggested appending the number to every filename. But it changes output for everyone, which is why he pushed back. Fair point.

Then I proposed a middle ground where we only suffix on actual collision, first session keeps its name. He accepted it, asked for tests. I wrote the code and tests.

Merged #2249

Takeaways

Exit code zero is not always success. Verify what a tool claims against what it produced.
A possible theory that reproduces a real bug can still be the wrong theory.
Bugs usually come from two reasonable components disagreeing about a definition. Not necessarily a broken piece.

Concurrency Without a Parallel Parser: Splitting PCAPs by Session

Robin Hayer — Wed, 26 Aug 2026 10:00:00 +0000

tshark needs linear state, so I moved the concurrency upstream — and hit a corrupted-output problem on the way.

Where the last post left off

In my first post about the 2.5 GB wall, streaming fixed memory. It didn't fix throughput.

The pipeline was still one file, one tshark process, one core. Piping stdout to stdin doesn't parallelize anything — it just stops the pipeline from holding the whole file in memory at once. Processing time dropped, but the shape of the work didn't change: still sequential, still bottlenecked on a single core no matter how large the machine underneath it was.

A few people asked the obvious question in the comments: why not just add goroutines?

I tried. It didn't work, and the reason it didn't work is more interesting than the fix.

Why the parser can't parallelize

Goroutines help when work can be split into independent pieces. tshark's dissection isn't independent — it's linear state.

It reads a capture packet by packet, and what it reads in one packet changes how it decodes the next. TCP stream reassembly needs the packets in order to rebuild the byte stream. Connection state tracks which flow a packet belongs to. Dissectors that depend on earlier context — retransmission detection, duplicate ACK detection, anything under tcp.analysis.* — read and update shared conversation tables as they go.

None of that is optional. tshark wasn't written single-threaded because nobody got around to parallelizing it — the dissection model requires strict ordering. Skip packets out of order and the state tshark is tracking for that stream goes wrong.

So wrapping the consuming side in goroutines doesn't help, because the problem was never on the consuming side. My Go program reading tshark's output line by line was already fast. The producing side — tshark itself, walking the file front to back — is sequential no matter what I do to the code around it.

If I wanted concurrency, it had to happen before tshark ever saw the file.

Move the concurrency upstream

If one tshark process can't go faster, run several over pieces of the file at once.

The obvious version of that is wrong. You can't cut a pcap into equal-sized chunks — a TCP stream that straddles a cut point loses the packets on the other side of the boundary, and the dissector on that chunk has no idea the connection existed before the cut. Retransmission flags, sequence tracking, anything stateful breaks the moment a conversation is split across two files.

The fix is to split along conversation boundaries instead of byte boundaries. Group packets by session — same 5-tuple, same connection — so every chunk holds complete conversations and nothing crosses a chunk boundary mid-stream. No state is lost, because no state ever needed to survive past a single chunk.

Once the split is done that way, the chunks are independent by construction. Each one goes to its own tshark process, running in parallel, merged back together at the end. The dissection itself stays linear — it just becomes N linear passes running at once instead of one linear pass running alone.

That was the plan. Getting there took one extra tool falling apart on me.

The bug

I reached for PcapSplitter, part of PcapPlusPlus, running in connection mode — split by 5-tuple, exactly what I needed.

In connection mode it holds one output file open per distinct flow for the length of the pass. On the files that mattered, that was 95 to 125 distinct flows open at once. Under that load, the output came back corrupted.

Two distinct failure signatures:

total block length N of an EPB is too small for M bytes of packet data
total block length 0 of an unknown block type is less than the minimum block size 12

I reproduced both on the master build and on the v25.05 stable release, so this wasn't something already fixed between versions with me sitting on a stale checkout. I reproduced both on pcapng and on legacy pcap after converting with editcap -F pcap, so it wasn't specific to the pcapng container format either. And I ran the source file through pcapfix before assuming the input was the problem — it came back clean. The corruption was being introduced by the split, not inherited from the source.

Same root cause both times: too many output file handles held open at once by one process. I didn't dig further into PcapSplitter's internals to find the exact line — I didn't need to, because the fix wasn't going to be a patch. I replaced it.

Reimplementing it in-process

I rewrote the session split using gopacket, inside my own program, instead of shelling out to a separate tool. No external process, no file handles held open by something I don't control — the splitting and the writing happen where I can see and bound exactly what's open at once.

Ran it 20 times across both files that had triggered the original corruption, in both the Go and Python versions of the pipeline. 20 out of 20 succeeded, including full concurrency on the file that had reproducibly failed before.

The honest ending

Here's the part worth sitting with.

Session splitting only kicks in above 100,000 packets — below that, one tshark process handles the whole file fine. I went back through the real corpus this pipeline processes day to day: 57 files. Exactly 3 of them cross that threshold.

I built session splitting, and everything downstream of it, for a problem that shows up in about 1 out of every 19 files. It was the right thing to build — the 3 files that need it would otherwise take hours or fail outright — but it's worth saying plainly: most of the corpus never touches this code path at all.

The 2.5 GB Wall: Why My tshark Wrapper Died, and How Streaming Fixed It

Robin Hayer — Mon, 10 Aug 2026 04:22:47 +0000

The tool that worked

I built a CLI tool that wrapped tshark for PCAP analysis. Grab the file, send shell commands, let tshark analyze it, capture the output, shape it, return it. Quite simple, right.

It was a good simple design, and it was the right design for what I was doing. I was analyzing small PCAP files and it worked smoothly. No hidden bugs, no errors, nothing to fix. When something works, you stop thinking about it.

Then came 2.5 GB

It was fine until I hit a 2.5 GB file. Six to seven hours of processing, and repeated crashes with out-of-memory errors.

I had never seen anything like this before. I ran it again. Same thing. Hours in diagnosis, still nothing. My first instinct was that I was messing something up in the shell commands, or that something was wrong with tshark itself.

Honestly, I was happy about it. I had never been challenged like this before. So I took the challenge.

The one thing I was sure of was the scale: this file held around 1.9 million packets. The problem was either on tshark's side or mine. It turned out to be both.

What was actually wrong

Three things, and they looked like one.

First, I was running three separate queries against the same file — one for analytics, one for rows, one for full dissection. Three full passes over 2.5 GB. I had written them as three because they answered three different questions, and at small file sizes that cost nothing. At 2.5 GB it cost everything.

Second, tshark is single-threaded. For a capture this size, there is no parallelism to fall back on.

Third — and this was the part I was most responsible for — I was asking for full JSON dissection. tshark built the entire output in memory, and then my program parsed all of it in memory. Two full copies of a dataset that was already too large.

I couldn't go further with that architecture. If I don't have memory, I can't process it. No amount of tuning fixes a design that requires holding everything at once.

Cutting three passes down to one

The first fix was the obvious one once I saw it: stop reading the file three times.

I consolidated the three queries into a single optimized query that returned everything I needed in one pass. Same output, one traversal instead of three.

That alone took processing from six to seven hours down to one to two hours. No architecture change, no extra hardware. Just not doing the same expensive work three times.

It was a big win and it wasn't enough. The memory pressure was still there, and one to two hours for a single file is still a bad number.

Streaming the output

The second fix came from a concept I knew in theory but had never needed: streaming.

Instead of waiting for tshark to finish and hand me a complete result, I connected tshark's stdout directly to my Go program's stdin. Now tshark emits packets and my program consumes them as they arrive. Processing overlaps generation. Memory stays flat, because at any moment I'm only holding one packet, not 1.9 million.

Then I narrowed the query itself. I stopped asking for full JSON dissection and switched to -T fields and -T ek, requesting only the fields I actually needed. Full JSON dissection means tshark decodes every layer of every packet and serializes all of it. Most of that work was thrown away by my program a moment later. Narrowing the query removed the work instead of optimizing it.

Together those two changes rebuilt the pipeline: tshark emits selected fields packet by packet, my Go program reads line by line, processes, and writes output as it goes.

The OOM crashes were gone instantly. Processing dropped to about 70 minutes on the same file — down from six to seven hours originally, on the same machine, with no extra cores.

What it didn't fix

Streaming solved memory. It did not solve throughput.

The operation was still single-threaded. Nothing about piping stdout to stdin makes tshark use more cores. And if multiple files arrive at once, the problem comes straight back — one file at a time, one core, everything else waiting.

That's the wall I hit next, and it's a different problem with a different fix.

If you've hit something similar, I'd like to hear how you handled it. I'm writing up the concurrency side next.