The tool that worked
I built a CLI tool that wrapped tshark for PCAP analysis. Grab the file, send shell commands, let tshark analyze it, capture the output, shape it, return it. Quite simple, right.
It was a good simple design, and it was the right design for what I was doing. I was analyzing small PCAP files and it worked smoothly. No hidden bugs, no errors, nothing to fix. When something works, you stop thinking about it.
Then came 2.5 GB
It was fine until I hit a 2.5 GB file. Six to seven hours of processing, and repeated crashes with out-of-memory errors.
I had never seen anything like this before. I ran it again. Same thing. Hours in diagnosis, still nothing. My first instinct was that I was messing something up in the shell commands, or that something was wrong with tshark itself.
Honestly, I was happy about it. I had never been challenged like this before. So I took the challenge.
The one thing I was sure of was the scale: this file held around 1.9 million packets. The problem was either on tshark's side or mine. It turned out to be both.
What was actually wrong
Three things, and they looked like one.
First, I was running three separate queries against the same file — one for analytics, one for rows, one for full dissection. Three full passes over 2.5 GB. I had written them as three because they answered three different questions, and at small file sizes that cost nothing. At 2.5 GB it cost everything.
Second, tshark is single-threaded. For a capture this size, there is no parallelism to fall back on.
Third — and this was the part I was most responsible for — I was asking for full JSON dissection. tshark built the entire output in memory, and then my program parsed all of it in memory. Two full copies of a dataset that was already too large.
I couldn't go further with that architecture. If I don't have memory, I can't process it. No amount of tuning fixes a design that requires holding everything at once.
Cutting three passes down to one
The first fix was the obvious one once I saw it: stop reading the file three times.
I consolidated the three queries into a single optimized query that returned everything I needed in one pass. Same output, one traversal instead of three.
That alone took processing from six to seven hours down to one to two hours. No architecture change, no extra hardware. Just not doing the same expensive work three times.
It was a big win and it wasn't enough. The memory pressure was still there, and one to two hours for a single file is still a bad number.
Streaming the output
The second fix came from a concept I knew in theory but had never needed: streaming.
Instead of waiting for tshark to finish and hand me a complete result, I connected tshark's stdout directly to my Go program's stdin. Now tshark emits packets and my program consumes them as they arrive. Processing overlaps generation. Memory stays flat, because at any moment I'm only holding one packet, not 1.9 million.
Then I narrowed the query itself. I stopped asking for full JSON dissection and switched to -T fields and -T ek, requesting only the fields I actually needed. Full JSON dissection means tshark decodes every layer of every packet and serializes all of it. Most of that work was thrown away by my program a moment later. Narrowing the query removed the work instead of optimizing it.
Together those two changes rebuilt the pipeline: tshark emits selected fields packet by packet, my Go program reads line by line, processes, and writes output as it goes.
The OOM crashes were gone instantly. Processing dropped to about 70 minutes on the same file — down from six to seven hours originally, on the same machine, with no extra cores.
What it didn't fix
Streaming solved memory. It did not solve throughput.
The operation was still single-threaded. Nothing about piping stdout to stdin makes tshark use more cores. And if multiple files arrive at once, the problem comes straight back — one file at a time, one core, everything else waiting.
That's the wall I hit next, and it's a different problem with a different fix.
If you've hit something similar, I'd like to hear how you handled it. I'm writing up the concurrency side next.
Top comments (0)