DEV Community

y4u
y4u

Posted on Originally published at uvp.y42u.net

Following a Log That Won't Stop Growing — Four Situations Where tail -f Can't Keep Up

#logging #devops #sre #linux

Three thousand lines a second is unreadable.

Your eyes manage maybe a dozen lines a second. So where did that ERROR go — the one that went past at two hundred times reading speed?

A log that keeps growing is hard in ways a static log isn't. Mid-incident triage, an overnight load test, an embedded serial console, a dev box turned up to TRACE: four different situations, and the dead end has the same shape in all of them. Here they are in order.

Up front: UwView Pro saves the index and the compression, so even on a file that keeps growing you can repeat "reopen and read to the end" in 0.02–0.07 s from the second open onward (measured on a 47.73 GB text file; one specific setup, results vary), and reapply a saved search condition as-is. Note that following a file while it is being written — real-time tail — lives in the free edition of UwView (details at the end)

1. The screen only shows "now" — after the ERROR has scrolled away

Situation

You're in the middle of an incident, with tail -f on the application log.

It pours. You thought you saw an ERROR and tried to follow it, but it's already off the top. Hit Ctrl+C and nothing after that point reaches the screen at all — while the file keeps growing.

Scrolling back up gets you a few thousand lines. At 3,000 lines a second, that's a few seconds' worth.

Why it happens

tail -f is a tool for forwarding appends to the end of a file. It has no notion of going backwards.

Then human reading speed multiplies in. You can follow maybe a dozen lines a second, and a handful if you're actually extracting meaning. At 3,000 lines a second, the gap between output rate and reading rate is more than two orders of magnitude. With a gap that wide, "watch all of it" was never on the table.

Adding a pipe to narrow things down introduces a second problem. When its stdout is a pipe or a file, grep switches to block buffering, so nothing reaches you until a few kilobytes have accumulated. A few seconds of lag mid-incident is a few seconds of lag on the decision.

Working with general-purpose tools, and where it stops

Turn off the buffering, then narrow.

tail -f app.log | grep --line-buffered -E 'ERROR|FATAL'

tail -f app.log | stdbuf -oL grep -E 'ERROR' | tee triage.log

tail -n 200000 -f app.log

tail -F app.log
Enter fullscreen mode Exit fullscreen mode

--line-buffered and stdbuf assume GNU coreutils and GNU grep. BSD and busybox behave differently, so check your own man pages.

Three limits.

First, you have to choose the condition before you start. Once you're running grep 'ERROR', realising you also want WARN doesn't help: the WARN lines that went by while you were watching only ERROR never appeared. The stream doesn't wait, so there's no do-over.

Second, you cannot go backwards. -n 200000 rewinds to a point at the moment you start. Deciding ten minutes in that you want to see ten minutes ago means stopping tail -f and running something else — and while it's stopped, you see nothing.

Third, there are no line numbers. tail's output doesn't carry them. Appending grep -n gives you "which line arrived on the pipe," not a coordinate in the original. Not being able to write "behaviour changes from this line of the original" in the incident report is exactly the excerpting problem from Part 16.

2. The overnight load test — picking up "yesterday's" thread first thing

Situation

A 24-hour load test, unattended overnight.

You arrive to find an 18 GB log. The monitoring graph says latency spiked some time around 3 a.m. The terminal where you'd left tail -f running shows — a dropped session. The scrollback wrapped hours ago.

That afternoon a developer asks to look at the same window from a different angle. You face the 18 GB again.

Why it happens

Three things stack up.

Terminal scrollback wraps by line count. Defaults sit in the thousands to tens of thousands. At a few hundred lines a second, that's minutes. The screen you were "watching" isn't kept.

ssh sessions drop. A network blip, a sleeping laptop, a VPN reconnect. None of it was designed to survive eight unattended hours.

And nothing records how far you got. This is the one that really costs. Picking up the next morning and re-examining in the afternoon both start from zero, because there's no record of where you were. Reopening the same 18 GB several times a day is Part 10's "ten reopens a day," transplanted into an overnight test.

Working with general-purpose tools, and where it stops

Keep the screen somewhere durable, and cut by time.

tmux new -s load -d 'tail -f run.log | tee -a session.log'

sed -n '/2026-09-13T03:/,/2026-09-13T04:/p' run.log > 3am.log

awk '$0 >= "2026-09-13T03:00" && $0 < "2026-09-13T04:00"' run.log
Enter fullscreen mode Exit fullscreen mode

Three limits.

First, tee -a means storing it twice. Another log appears alongside the 18 GB — and if you run the test nightly, it appears nightly (Part 8 is about the storage side).

Second, excerpting renumbers from 1. What line of the original is line 500 of 3am.log? You need that coordinate the moment you paste it into the test report.

Third, both sed -n and awk scan from the top every time. One pass over 18 GB is minutes, and comparing the 3 a.m. hour against the 4 a.m. hour is two passes. Jumping by timestamp was Part 7's subject — but when the target is still growing, the tail moves while you scan.

3. Reading days of serial-console capture end to end

Situation

You ran an embedded board for three days and captured everything on the serial line. 2.4 GB of text.

There are several reboots in there. Some were planned; some weren't. You want to know which reboots were the abnormal ones.

Your editor hangs on it. less opens it, but the screen goes strangely coloured and the layout breaks.

Why it happens

A serial capture isn't "text for people to read." It's the byte stream that went to the terminal, verbatim.

  • Control characters and escape sequences, raw. Colour codes like ESC[32m (ESC is 0x1B), cursor moves, carriage returns used to overwrite a line. The bootloader and kernel were writing to a terminal, so of course they're in there.
  • Mixed line endings. Bare CR, CRLF, and LF in one file, varying with how picocom / minicom / screen was configured. With bare CR, what counts as "one line" stops agreeing between tools — the same accident as the factory logs in Part 18.
  • No timestamps, or relative ones. The board may only have milliseconds since boot. Answering "when" across three days means keeping a separate mapping to wall-clock time.
  • Two different kinds of mojibake. Byte-level corruption from a baud-rate mismatch, and vendor messages emitted in Shift_JIS. The second is fixed by switching encodings; the first is not recoverable (Part 3).

Working with general-purpose tools, and where it stops

See what's mixed in, then count.

cat -v capture.log | head -40

perl -pe 's/\e\[[0-9;]*[a-zA-Z]//g' capture.log > clean.log

grep -c 'Booting Linux' capture.log

tr -d '\r' < capture.log | wc -l
Enter fullscreen mode Exit fullscreen mode

Three limits.

First, cleaning it produces a different file. clean.log is readable, but it isn't the original, and dropping CR changes the line count outright. The coordinate you'd give the board vendor — "from this line of the original" — is lost in the cleanup step.

Second, baud-corrupted text doesn't match a search. If the o in Booting came through as a corrupted byte, grep 'Booting Linux' won't count that reboot. The answer says four; it might really be five. No amount of cleverness in the search term fixes that. Reading it through is what fixes it.

Third, it can't answer "when." With only milliseconds since boot, grep gives you a place but not a time. Finding "the second day, late afternoon" in three days of capture comes down to reading around it and estimating.

4. TRACE produced 40 GB in an hour — designing for the flow rate itself

Situation

To catch a bug that won't reproduce, you turned the log level up to TRACE and started a run.

40 GB in an hour. The disk warning fires. And the bug hasn't shown up yet. You want several more hours, which means clearing what's already there.

Why it happens

Log level multiplies rather than adds.

What was one line per request becomes 200 at TRACE. At 100 requests a second, that's 20,000 lines a second. Extrapolating linearly from how DEBUG felt is guaranteed to miss by an order of magnitude.

And everything around it is still configured for DEBUG.

  • Rotation can't keep up. "100 MB × 10 generations" cycles several times an hour, so the earliest part — the part you most want — is the first to go (evidence vanishing in the rotation gap was Part 16).
  • Compression eats CPU. The post-rotate gzip runs on the same host as the application and contaminates the latency numbers you're measuring.
  • The dev box doesn't have the disk. You can't reproduce production's flow rate locally (Part 17 is that friction).

Working with general-purpose tools, and where it stops

Emit less, decide retention first, discard while streaming.

<!-- TRACE only where you suspect (logback) -->
<logger name="com.example.payment" level="TRACE"/>
<root level="INFO"/>
Enter fullscreen mode Exit fullscreen mode 
logrotate -f "$LOGROTATE_CONF"

tail -F app.log | grep --line-buffered 'txnId=7f3a' > trace-7f3a.log

watch -n 60 'df -h /var/log; ls -l --block-size=M /var/log/app.log'
Enter fullscreen mode Exit fullscreen mode

Three limits.

First, narrowing erases the lines you assumed were irrelevant. This is the nastiest one here. You set TRACE on com.example.payment only — and the cause turns out to be contention with a different module, whose evidence was never emitted. You raised the level precisely because you didn't know what was relevant, and now you're being asked to narrow. The order is backwards (the same contradiction shows up in Part 5's printf debugging).

Second, it disappears while you wait for rotation. More generations and the disk won't hold; fewer and the old end goes. Either way, you're forced to choose before you've decided anything.

Third, without a way to open 40 GB, you end up throwing away the log you just produced. This genuinely happens. You turn on TRACE, get 40 GB, grep a handful of lines because nothing will open it, and delete the rest. Raising the output volume loses its meaning at the point where it can't be read (Part 1's "won't open" comes back around here).

What all four had in common

Situation How it flows General-purpose approach What's left over
Mid-incident tailing Thousands of lines/s, two orders above reading speed `tail -f \ grep --line-buffered`
Overnight load test 8 unattended hours, 18 GB tmux + tee, sed -n by time Stored twice. Renumbered from 1. No coordinate to resume from
Serial console 3 days, 2.4 GB, control characters mixed in cat -v / sed to strip ANSI / grep -c Cleaning yields a different file. Corrupted terms don't match
Dev box at TRACE 40 GB/hour, growing multiplicatively Per-logger levels / logrotate / filtered tee Narrowing suppresses the evidence. Rotation deletes it. Unreadable means wasted

Four unrelated situations. They jam in the same place because a flowing log has three properties at once:

  • "Now" and "a moment ago" don't fit on one screen. A follower sticks to the end and has no reverse gear; the tools that do go backwards (less, sed -n) aren't watching the end while they do it. Pick one and you lose sight of the other.
  • There are no do-overs. On a static file you can change the condition and try again as often as you like. On a flowing one, the lines that pass while you're changing the condition are simply gone from view. Hence "decide the right condition first" — an impossible request.
  • No coordinates survive. Not in the follower's output, not at the end of the pipe, not in the cleaned-up file. You can't get back to "that line" afterwards.

With those three together, "read it as it flows" and "go back and read it later" become two separate jobs — tail -f for the first, less or sed for the second. Two tools isn't the problem. The coordinate not surviving the handoff between them is the dead end.

What's needed is to make the second job cheap.

  • Open everything that exists right now, even while the file grows — look at the past in another window without stopping the follower.
  • Make reopening cheap — the faster the flow, the more often you reread to the end. If one wait is short, the count stops mattering.
  • Don't rebuild the condition — apply last night's filter to this morning's file as-is.

Back to that ERROR that scrolled away. It didn't vanish. It's still in the file, with a line number. It only left the screen. What couldn't keep up was the eye; the record held.

The tool I use

UwView (free), which I develop, is a viewer that displays, scrolls, and searches huge text from the moment it opens. It never loads the whole file into memory, so it opens files larger than RAM. The index is built in the background; when it finishes, line numbers appear.

For flowing logs, here's what the free version covers.

  • It opens files that are being written. It opens for shared reading, so you can read while the application keeps appending. Section 1's "look at ten minutes ago in another window without stopping tail -f" works.
  • Real-time tail — following appends — is in the free version. It detects growth and auto-scrolls to the end. As noted below, Pro does not have it (desktop only; it doesn't work in the browser build).
  • How much you can read isn't decided by RAM. Section 4's 40 GB, section 2's 18 GB, and section 3's 2.4 GB all open unsplit (measured ceiling on the free version: 47.73 GB, about 890 million lines — one specific setup).
  • It never writes to the original, and never excerpts or splits it. No clean.log from section 3, no 3am.log from section 2, which means line numbers stay the original's.
  • Encoding can be switched while the file is open (UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16, auto-detected). Section 3's "only the vendor messages are Shift_JIS" needs no intermediate file.
  • Highlighting colours lines rather than removing them. Colour section 1's ERROR|FATAL and you can pick them out of the stream without filtering — and because nothing is removed, their surroundings stay on the same screen.

Section 2's "pick up where I left off" and section 4's 40 GB of retention are UwView Pro territory.

  • The index and the compression are saved: from the second open onward, the same file comes back instantly, with line numbers (0.02–0.07 s measured on a 47.73 GB text file; one setup, results vary). When the workflow is "reopen and read to the end," the number of reopens is the cost. Section 2's morning and afternoon, and section 1's repeated review, are exactly that.
  • Search conditions can be saved and reapplied: last night's filter goes straight onto this morning's file. Section 2's "from zero every time" disappears (the filter popup).
  • Drill-down search: filter a result by another term, and another. The original line numbers survive to the last step. Section 1's "started with ERROR, now I also want WARN" becomes something you can do afterwards, without racing the stream (drill-down search).
  • ±N is independent per step: ±1 while narrowing, ±200 while reading. Section 3's "found the reboot line, now show me the 100 lines before it" works directly (the free version is fixed at ±1; a variable ±N is Pro).
  • Sequence search: find only the places where w1 → w2 → w3 appear in that order. Section 3's planned and unplanned reboots can be separated by the order of the messages that precede them (how it works). Honestly: each step scans the text from the previous position, so it takes about as long as a full-text search.
  • Tally (frequency ranking): counts per captured value, and clicking a row descends to where it occurred. It's section 3's reboot count, with the difference that you can get back to the actual lines after counting.
  • Stored at roughly 1/9 and still searchable: section 4's 40 GB an hour stays compressed and searchable, without zgrep's decompress-as-you-scan on every query (Part 8).
  • Larger files: 258.68 GB and 4.5 billion lines measured on Pro (one specific setup; results vary).

The honest limits

This article is about following a log, so the most important limit comes first.

  • UwView Pro does not have real-time tail. Following appends lives in the free edition of UwView. Pro manages edits as a diff file (.ewvz), which sits badly with an original that keeps changing, so the feature isn't there. If you need to follow a log while it's being written, use the free version; if you only need to reopen and read to the latest, Pro does that fine. Worth checking before you buy.
  • No alerting and no notifications. "Page me when FATAL appears" is monitoring's job. Section 1 assumes a human is watching.
  • No automated cross-log correlation. Lining section 2's test log up against the monitoring graph belongs to another tool.
  • No automatic stripping of control characters or ANSI escapes. Section 3's capture is displayed as it is — read through, not scrubbed.
  • No involvement in log levels or rotation. How much section 4 emits is the application's business.
  • Lines display up to the first 8,192 characters. On a file where one line gets extremely long — section 3's bare-CR capture, for instance — the rest is elided.
  • Text only. A capture recorded in binary has to be written out as text first.

All four items above are read-only work. Non-destructive diff editing (Edit Upgrade) exists as a separate licence, but what you need here is the View side.

The same things, from the command line

v1.6.0 added a uvp command (and uvf for the free build). It uses the same .uwvz as the GUI, so an index built from the shell is already there when you open the file in the app. Mapped onto this article's four parts:

uvp app.log -E 'ERROR|FATAL' -C 5

uvp app.log -E 'ERROR|WARN|FATAL' 'txnId=7f3a' -C 5

uvp run.log '2026-09-13T03:' 'latency' -C 10 -out 3am-latency.txt.gz

uvp capture.log -seq 'reboot: Restarting system,Booting Linux' -C 10

uvp app.log.uwvz -uniq 'txnId=([0-9a-f]+)' -head 20

uvp app.log.uwvz 'txnId=7f3a' -open
Enter fullscreen mode Exit fullscreen mode

Exit codes are grep's — 0 found, 1 not found — plus 2 when the 1,000,000-hit cap is exceeded. Put if uvp run.log 'FATAL'; then at the end of section 2's overnight job and the morning starts with the one thing you needed to know. Stated plainly, though: there is no tail equivalent in the CLI either. Following a log while it's being written is the free UwView's real-time tail; what uvp covers is the other side — taking the stream back, fast, in a form you can keep.

Stated honestly: on the first question ripgrep is 15–20% faster, because uvp builds its index first. On a 3 GB file that fits in RAM, rg stays ahead on the second question too. uvp pays off past 10 GB, when you ask the same file more than one question (the measurements; Mac M4, external USB SSD, OpenStreetMap XML — one setup, results vary).

And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly).

Links

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Line counts, sizes, filenames, log contents, and configuration values are illustrative and do not describe any real product or incident. The behaviour of tail, grep, sed, awk, tr, cat, stdbuf, tee, logrotate, tmux, and watch varies by implementation (GNU/BSD/busybox), version, and build options, as do option names and defaults — in particular the difference between tail -f and tail -F, and whether --line-buffered or stdbuf is available at all. Check your own man pages and the official documentation. Before changing log levels in production, confirm disk capacity, rotation settings, and performance impact, and follow your organisation's change-management process. Measured figures come from one specific setup and are not a guarantee of the same result. If you spot an error, a comment is welcome and I'll check and correct it.

Top comments (0)