10GB, waiting for me in the morning.
It came from one printf I dropped into a loop before going to bed. Almost every line says the same thing: everything was fine.
So which line is the one where it stopped being fine?
When a bug won't hold still for a debugger, we all fall back to printing. And then we print too much. Unlike production logs, development logs have no retention policy, no rotation, and no design. You add a line because you're stuck, you mean to remove it when it's fixed, and you forget.
This article lines up four situations from the development side: output balloons, reproduction takes hours, threads interleave, and you need to compare "before it broke" with "after." Each one has a workable command-line answer. And each one has a point where that answer stops — and for all four, it turned out to be the same point.
Up front: UwView (free) opens a 10 GB debug log readable from the first moment; UwView Pro reopens it instantly with line numbers from the second time on (details at the end)
1. printf debugging produced 10GB — reading it before you delete it
Situation
Look at that 10GB more closely. It came from a single added line inside a loop — a printf, a console.log, a Log.d — left running overnight while you waited for the bug to show up. By morning it was past 84 million lines.
There's essentially one shape of line in there. Of those 84 million, maybe a few dozen are worth reading.
Why it happens
Debug output scales with execution count, not with the difficulty of the bug.
There's no way around that. You added the print because you don't know where it breaks; if you knew, you'd have added a condition. "I have no idea what's happening" and "I can't narrow it down" are the same state, so until you know, you print everything. The result is an extremely low density of useful information. Saying that a few kilobytes of a 10GB file matter is not an exaggeration.
Worse, that 10GB can't be thrown away. Especially if reproducing it was expensive. With no guarantee you can recreate the same conditions, you can't delete it until you've finished reading it.
Command-line triage, and where it stops
Start by looking at the shape of the whole thing.
wc -l debug.log # line count first
awk '{print $3}' debug.log | sort | uniq -c | sort -rn | head # distribution of message types
grep -n 'state=BROKEN' debug.log | head # positions of suspicious lines
sed -n '84213900,84213960p' debug.log # pull the surrounding window
grep -n for the position, sed -n for the context. That two-step is genuinely powerful and it settles most investigations.
It stops in two places. First, it assumes you already know what a suspicious line looks like. If you were smart enough to emit a state=BROKEN marker, you win — but if you understood the failure well enough to emit that marker, you probably didn't need this log. In practice the discovery is "the normal lines start looking subtly different around here," and that is the kind of thing you only notice by skimming with your eyes.
Second, skimming doesn't work at 10GB. Editors won't open it, less will but shows no line numbers, so you lose all sense of where you are in the file. It's exactly the wall from part 1.
2. A bug that takes three hours to reproduce — don't start over every morning
Situation
The bug only appears after about three hours under load. You got lucky and reproduced it. You open the log, dig in, and end the day at "here's what I know so far."
The next morning you open the log to continue. You wait for the index. You hunt for the line you'd reached yesterday. You try to remember the filter you were using. That's your whole morning.
Why it happens
The state of the investigation lives outside the tool.
Chasing a three-hour-repro bug necessarily spans several days. But most viewers and editors throw away "where you were looking" and "what you were filtering by" the moment they close. All that survives is the file, so tomorrow you start from opening it again.
And the cost of reopening scales with size. If an 8GB log takes two minutes to open, five reopens is ten minutes. That sounds small. What actually costs you is the reluctance it creates: you stop testing cheap hypotheses because opening the file is annoying. The quality of the investigation gets eroded by the tool's latency.
Command-line triage, and where it stops
You can externalize some of the state.
grep -n 'txId=8842' huge.log > findings/txid8842.txt # save the positions you found
sed -n '12000000,12000500p' huge.log > findings/window.log # carve out a working window
Writing line numbers into your notes helps too. "Anomaly starts at 84,213,912" means tomorrow you can sed straight there.
It stops because carving destroys context. findings/window.log only knows about its 500 lines, so the moment you want another 2,000 lines earlier, you're back to the original. And back to reopening it.
There's a deeper circularity: line numbers are only meaningful against the original, and opening the original is exactly what costs you every time. Your note says 84213912, but reaching it means opening 8GB. This is why "just take notes" doesn't actually solve it.
3. Multithreaded logs interleave into noise
Situation
Sixteen worker threads, one log. It's in timestamp order, which is correct, but consecutive lines come from different threads. Following one thread's story means your eye jumps every few lines.
12:04:31.220 [w-07] fetch start id=8841
12:04:31.221 [w-03] parse done id=8830
12:04:31.221 [w-11] fetch start id=8842
12:04:31.223 [w-07] fetch done id=8841
Ten minutes in, you've lost track of which thread you were following.
Why it happens
The order the file is written in (time) doesn't match the order you want to read in (thread).
This isn't a design mistake. Time order is correct and there's no alternative. But when debugging, what you want is usually "what happened to w-07" — a vertical narrative — while the file is laid out as horizontal cross-sections. That orthogonality is the whole difficulty.
The nasty part is that some bugs are only visible in the interleaving. Deadlocks and race conditions are relationships between threads in time, so separating the threads makes them disappear. You want to split it, and you must not split it all the way.
Command-line triage, and where it stops
Extraction is easy.
grep '\[w-07\]' app.log | less # just one thread
grep -E '\[w-(07|11)\]' app.log # two threads, still in time order
awk '{print $2}' app.log | sort | uniq -c | sort -rn # which threads are loudest
grep --color=always -E '\[w-07\]|\[w-11\]|$' app.log | less -R # colour without filtering
That last trick — appending the empty alternative |$ — is the classic way to highlight matching lines without discarding the rest. It's how you keep context and still get colour.
It stops in three places. First, piping into less -R loses the original line numbers, so you can't come back to a position later. Second, the grep expression becomes unmanageable as the colour count grows; sixteen threads in sixteen colours is not realistic. Third, every switch between "filtered" and "full" means retyping the command. The round trip — look at only w-07, back out for surrounding context, filter again — happens dozens of times per session. Even at a few seconds each, your train of thought breaks every time.
4. Comparing "when it worked" with "after it broke"
Situation
It worked last week. It fails this week. You have both logs. Line up the same operation in each and the difference should be right there.
So you run diff, and it emits several million lines. Timestamps, request IDs, and thread names differ on every line, so every line registers as a difference.
Why it happens
Line-level diff can't tell which parts of a line are supposed to change.
diff compares text; it knows nothing about log structure. When a human reads 12:04:31.220 [w-07] fetch start id=8841, they unconsciously file the timestamp, the thread, and the ID under "always different" and fetch start under "meaningful." diff has no such distinction, so it weighs all of it equally.
And what you actually want isn't a line diff at all — it's a sequence diff. "It used to go fetch → parse → commit, and now parse is missing." "Retries went up to three." The thing you're comparing is the order of events, not strings.
Command-line triage, and where it stops
The standard move is to normalize away the volatile parts first.
sed -E 's/^[0-9:.]+ //; s/id=[0-9]+/id=N/g; s/\[w-[0-9]+\]/[w]/' good.log > good.norm
sed -E 's/^[0-9:.]+ //; s/id=[0-9]+/id=N/g; s/\[w-[0-9]+\]/[w]/' bad.log > bad.norm
diff <(sort good.norm | uniq -c) <(sort bad.norm | uniq -c) # compare by occurrence count
Keeping only the skeleton of each line and comparing counts surfaces things like "this message appeared 0 times when healthy and 4,120 times when broken." That works well.
It stops in two places. First, normalizing erases the original line numbers. You learn that retries increased, but not where in bad.log they started increasing. So you go back to the original and start again from grep -n.
Second, counting throws away order. The sort | uniq -c above compares sets, so it can't detect "the sequence got reordered" — a very common failure shape. Seeing order means opening both files at once and following the same region side by side with your eyes. And when both files are several gigabytes, "open both at once" is the part that doesn't happen.
What all four had in common
| Situation | How it shows up | Where it stops |
|---|---|---|
| printf output reached 10GB | Useful density is vanishingly low | No tool that lets you skim it |
| Three-hour reproduction | Every morning starts with reopening | Investigation state lives outside the tool |
| Threads interleaved | The vertical story is cut horizontally | Cost of the filter/full round trip |
| Healthy vs. broken comparison | diff reports every line | Normalizing erases line numbers |
None of these are blocked on analysis technique. They're blocked on one thing: the operation you need stops being worth its cost once the file is big.
Skimming, reopening, toggling between a filter and the whole file, viewing two files side by side — on a small file, everybody does all of these without thinking. At a few gigabytes each one starts charging tens of seconds to minutes, so people stop doing them. The investigation gets shallower as a result, which means the tool's latency is quietly lowering your bug-discovery rate.
This shape has recurred through the series: part 1's tools that demand a full read before showing anything, part 2's round trip between the hit list and the scene, part 3's re-read on every interpretation switch, part 4's trade of size against readability. Development logs get treated as disposable because they're short-lived — but they get stuck in exactly the same places production logs do.
The tool I use
UwView (free), which I develop, is a viewer built to drop the assumption that size makes these operations expensive. Even a 10GB debug log displays, scrolls, and searches from the moment it opens; the index is built in the background and line numbers appear when it completes. That makes the section-1 job — skimming until the shape of the lines changes — something you can actually do. Highlight colours are applied without removing lines, so you can follow a thread by eye with the context intact, as section 3 needs. It never writes to the original.
And when you're chasing a slow-to-reproduce bug across several days, or lining up a healthy log against a broken one, take a look at UwView Pro. It saves the index and the compression, so from the second open onward the file opens instantly with line numbers — no more mornings that start with reopening. Compressed-cache search and ~1/9 storage come with it, so holding on to a reproduction log you can't delete costs you less disk (all OS, one-time or monthly).
One honest limit: opening a file and editing it are different problems. Bulk-deleting or reshaping lines in a huge debug log is a separate piece of work that's still in development — there's a progress report in this comparison.
Links
- Part 1: four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
- Part 2: four ways to trace causality in logs: https://uvp.y42u.net/en/blog/uwview-ps02-log-causality-tracing-en/
- Part 3: four character-encoding traps: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
- Part 4: deciding between delete, keep, and compress: https://uvp.y42u.net/en/blog/uwview-ps04-log-retention-decision-en/
- How the colour highlighter works: https://uvp.y42u.net/en/blog/uwview-v11-color-highlighter-en/
- Measured across three file sizes (with conditions): https://uvp.y42u.net/en/blog/uwview-pro-benchmark-3sizes-en/
- Source code (GitHub): https://github.com/amru195704/UwView
From the developer: My apps, Kindle books, and open-source projects are listed at GitHub: amru195704.
A note
The information in this article is provided for reference and is not guaranteed to be accurate or complete. Log volume, reproduction time, and time-to-open vary widely by language, runtime, and storage configuration. Command examples may need adjusting for your environment (GNU vs. BSD, your shell, thesedandawkimplementations, etc.). If you find an error or inaccuracy, please point it out in the comments and it will be corrected after verification.
Top comments (0)