DEV Community: y4u

We Lose to klogg on Opening. uvf Wins the Whole Run — Re-measured at 50 GB

y4u — Fri, 18 Sep 2026 14:23:44 +0000

If all you do is open a 51 GB file, klogg is faster. 52.55 s against 100.6 s. That one we lose.

Run it through to the search, though, and it turns over: klogg takes 108.14 s, uvf … -open takes 53.69 s.
About 2×. The reason is not speed. It is how many times the file gets read.

First — what `uvf … -open` is

uvf … -open runs through this whole article, so here is what it means.

UwView has two faces: a window (GUI) and a command line (uvf). Same tool — the window is for
opening and reading a huge file, the command is for finding things in it.
-open is the option that joins the two.

uvf <file> <pattern> -open

Type that and uvf reads the file once, searches it, hands the byte offsets of the matching lines
straight to the window, and the UwView window comes up. The window does not search again.
A line-numbered list of hits is there from the first frame, and clicking any of them puts you at that
spot in the file.

In other words, it removes the round trip where you grep to find roughly where something is, then
open the same file again in a separate viewer and search all over again.
Every uvf … -open figure in this article is that one command, timed from pressing enter to the
list of hits being on screen.

uvf is free (it ships with the free edition of UwView) and runs standalone on Windows, macOS and Linux
-open has existed since v1.6.0, but v1.6.4 is where it became a single pass and v1.6.5 is where the wait for the index disappeared. Everything measured here is v1.6.5

The loss, first

The same data at 3 GB, 10 GB and 51.25 GB (OpenStreetMap Japan expanded to XML, plus splits of it),
just opening it, and just searching for one term.

Opening

	3 GB	10 GB	50 GB
klogg	3.65 s	10.98 s	52.55 s
UwView (free)	5.27 s	19.62 s	100.6 s

Searching (東京)

	3 GB	10 GB	50 GB
klogg	0.56 s	11.75 s	55.59 s
UwView (free)	0.20 s	16.72 s	88.9 s

Hit counts were 11,274 / 11,393 / 94,979 and matched between the two at all three sizes.
From 10 GB up, klogg wins both opening and searching.

Convert the seconds to read throughput and it is obvious what is happening.

	3 GB	10 GB	50 GB
klogg, opening	792 MB/s	891 MB/s	930 MB/s
klogg, searching	(cached)	832 MB/s	879 MB/s
UwView free, opening	549 MB/s	498 MB/s	486 MB/s
UwView free, searching	(cached)	585 MB/s	550 MB/s

This drive reads at about 950 MB/s raw. klogg saturates it. Our free GUI runs at half.
That is not a bug — it matches the figures we have published all along (473–491 MB/s to open,
575–581 MB/s to search). It is how the thing is built, and it is a debt we owe. More on that below.

And yet the whole run turns over

What you actually want is not "open it". It is "find it and read it." So we timed that.

Time to find it and read it	3 GB	10 GB	50 GB
UwView free GUI (open + search)	5.47 s	36.34 s	189.50 s
klogg (open + search)	4.21 s	22.73 s	108.14 s
`uvf … -open` (v1.6.5)	3.28 s	10.41 s	53.69 s
ratio (vs klogg / vs free GUI)	1.28 / 1.67	2.18 / 3.49	2.01 / 3.53

klogg reads the file once to open it and again to search it. The UwView free GUI also reads it twice,
and at half the speed — which is how you get to 3.5×.

uvf file 'pattern' -open reads it once.

$ time uvf osm/japan-latest.osm 東京 -open
11.32s user  7.31s system  34% cpu  53.690 total

51.25 GB ÷ 53.69 s = 910 MB/s — essentially this drive's raw read speed (≈ 950 MB/s).
CPU sits at 34%; the rest is waiting on the disk, so the medium is the entire bottleneck.
Inside those 53.69 seconds it searches, builds the index, hands everything to the window and puts a
line-numbered list of hits on screen. The window itself then appears in under a second.

At 10 GB it is 10.405 s (940 MB/s); at 3 GB, 3.279 s (882 MB/s). At every size, it costs one pass over the file.

For reference, the uvf command measured on the search alone runs at 871 / 933 / 962 MB/s.
-open at 882 / 940 / 910 sits in the same band, dropping about 5% only at 50 GB — that is the window starting up.

258 GB behaves the same way

We checked the largest file too — the whole United States from OpenStreetMap expanded to XML,
258.68 GB and 4.5 billion lines, in one unsplit file.

$ sudo purge
$ time uvf osmUs/us-260726.osm 'New York' -open
56.24s user  40.60s system  36% cpu  4:25.21 total

265.21 s (4 min 25.2 s), 930 MB/s — the medium's speed again.

The first run gave 250.68 s (984 MB/s). That was above the drive's ~950 MB/s raw read, so we put it down to
cache left by a run we had stopped partway, and re-took it after sudo purge. The difference is 5.5%.
We are publishing the slower number — the one without cache.

Now line that up against how long each of them takes just to finish opening the file.

258.68 GB	Time	Throughput
klogg, to finish opening	258 s	956 MB/s
`uvf … -open` (find it, all the way to the screen)	265.21 s	930 MB/s
UwView Pro, to finish opening (building the `.uwvz`)	328 s	752 MB/s
UwView free GUI, to finish opening	530.1 s	465 MB/s

In about the time klogg needs merely to finish opening the file, we have already searched it and put the hits
on screen (258 s against 265.21 s — 7 seconds, 2.8%, apart). klogg starts searching from there.

Doing the same thing in the free GUI costs 8 min 50.1 s to open plus 7 min 20.0 s to search — 16 min 10 s, or 3.66×.

(Both free-GUI figures were re-taken the same day and came out the same — the earlier run gave 532.6 s / 449.4 s, 0.5% and 2.1% apart. That is 465 / 561 MB/s, the same band as 486 / 550 at 50 GB.)

And of course, if you keep coming back to the same file, that is Pro. Once the .uwvz exists, a search takes
34.8 s and opening it again is instant. One pass, or many — that is the dividing line.

Why one pass is enough

When uvf first shipped in v1.6.0, -open passed only the pattern to the window, and the window
searched again on its own. The CLI read everything, then the window read everything.

v1.6.4 — the CLI hands over the byte offsets of the lines it found. The window stops re-searching
v1.6.5 — it also collects the index markers while searching and hands those over too. The window assembles the index instead of re-reading the file, and since the line numbers travel with it, the results list has line numbers from the first frame

It is reading the whole file anyway, so counting one more thing costs almost nothing. And indeed the
timings above are near-identical to the search-only figures for uvf (3.32 s / 10.48 s / 50.82 s).
The handoff adds essentially zero.

The 3 GB case narrows

Look at the ratios again: 1.28× against klogg at 3 GB, against 2.01× at 50 GB.

The reason is simple — 3 GB fits in memory (32 GB here). klogg and the free GUI still read it twice,
but the second read comes from cache, so reading twice costs almost nothing (klogg's 0.56 s search at
3 GB, the free GUI's 0.20 s).

From 10 GB up it does not fit, so the second read really does hit the disk, and the gap opens.
This is the same thing we have been saying for months: 3 GB is a draw; the difference starts at 10 GB.

The free GUI's 0.20 s at 3 GB is near the limit of what a person can time by hand. Even if it were
really 1.0 s, the total is 6.27 s — 1.9× — and the conclusion does not move.

The debt we owe

This measurement made our own weak spot plain.

The free GUI opens at 486 MB/s. klogg opens at 930 MB/s. Very nearly a factor of two.
The uvf command was rebuilt as a single pass in v1.6.3 and reaches 962 MB/s, so we already have proof
that this machine and this disk can do it. If the GUI's index build is brought to the same way of reading,
opening 50 GB should go from 100.6 s to somewhere near 52 s.

klogg saturating the medium is, for us, a measured target. We will close it in a coming release.

The side that keeps an index

Everything above is the free edition. The paid uvp converts the file once into a .uwvz
(about one ninth of the original, with a line index) and never touches the original again.

Search	klogg	`uvp` (with `.uwvz`)	Difference
10 GB	11.75 s	1.80 s	6.5×
50 GB	55.59 s	6.34 s	8.8×

Opening it again	klogg	`uvp`	Difference
10 GB	10.98 s (every time)	0.01–0.07 s	150–1,100×
50 GB	52.55 s (every time)	0.01–0.07 s	750–5,000×

klogg rebuilds its index every time you open the file. That is the dividing line between
"look at it once" and "come back to it again and again".

Which to use

Looking once → klogg is enough. It opens fast, it is free, and it is open source
Find it, then read it → uvf … -open. One pass over the file gets you all the way to the window. Free
Coming back to the same file → UwView Pro. Opening again takes 0.01–0.07 s, searching takes 6.34 s

How it was measured (you can repeat it)

File: OpenStreetMap Japan XML, one file of 51,254,526,392 bytes (the 10 GB and 3 GB files are splits of the same data)
Term: 東京, fixed string. Hit counts confirmed at 11,274 / 11,393 / 94,979 — all three matched
sudo purge before every run, then a 10 s settle; everything measured cold
Mac M4 / 32 GB / external USB SSD; klogg 24.11.0.1685; UwView free GUI and uvf v1.6.5
The GUI runs were timed by hand (open = until the index finishes and you can move anywhere in the file; search = until the hit count settles)

uvf is free and runs on Windows, macOS and Linux as a single binary from
GitHub Releases.
If you want an index so you can come back to the same file, that is UwView Pro.

ripgrep Is 2.9x Faster With no-mmap — Measured Against the Default mmap Read on One 51 GB File, on Three OSes

y4u — Fri, 18 Sep 2026 14:20:27 +0000

Searching a 51 GB file with ripgrep, adding --no-mmap can make it faster. On Windows, by up to 2.89×. On macOS, nothing changes. On Linux it goes either way, depending on what you are searching for.

This is not a story about a flaw in ripgrep. Memory-mapped reading is the right default most of the time. It is just that "most of the time" turned out not to mean "always".

The numbers first

The same single 51.25 GB file (OpenStreetMap Japan expanded to XML — 51,254,526,392 bytes), the same two searches, every run measured immediately after dropping the cache.

Machine	Search	default (`--mmap`)	`--no-mmap`	Effect
Windows 11	fixed string	83.74 s	68.50 s	1.22× faster
Windows 11	regex + invert	282.84 s	97.83 s	2.89× faster
macOS	fixed string	55.38 s	55.70 s	no change
macOS	regex + invert	59.40 s	58.47 s	no change
Linux (VM, 8 GB RAM)	fixed string	~83 s	~100 s	~1.2× slower
Linux (VM, 8 GB RAM)	regex + invert	~172 s	~105 s	1.64× faster

Six cases: three got faster, two stayed the same, one got slower.

Three things follow.

On macOS you can ignore all of this. Either way is the same. The default is already doing the best it can.
On Windows the default costs you on a 51 GB file — nearly 3× on the heavier search.
On Linux it flips with the workload. Light search: the default wins. Heavy search: --no-mmap wins.

Why we ended up measuring this

We were comparing our own CLI — uvf, which ships with the free edition of UwView — against ripgrep across 3 GB, 10 GB and 50 GB on macOS, Windows and Linux. The 50 GB case on Windows looked wrong.

Summed over the same seven searches, Windows against macOS came out at 1.09× for 3 GB and 1.45× for 10 GB. That is a believable machine difference. But 50 GB jumped to 2.25×, and inside it one item — -E -v, a regex printing the lines that do not match — took 283 s against macOS's 58 s. 4.9×.

"The machine is slower" does not explain that, because there is no reason for the ratio to get worse as the file grows. So we wrote a script to take it apart.

How it was measured

File: OpenStreetMap Japan XML, one file of 51,254,526,392 bytes
Search 1 (fixed string): rg -n -F '東京' <file> — 94,979 hits
Search 2 (regex + invert): rg -n -v '^ +<' <file> — 3 hits
ripgrep 15.2.0 (15.1.0 on Linux)
Cache dropped before every single run, then a 10 s settle (sudo purge on macOS, drop_caches on Linux, RAMMap -Et from an elevated shell on Windows)
Same script everywhere; hit counts recorded each time and checked to match between --mmap and --no-mmap

The machines:


macOS	Apple M4 / 32 GB / external USB SSD
Windows 11	HP Spectre x360 (Core i7-1165G7, 4C/8T, 15 W class) / 15.6 GB / Intel Optane H20 with SSD 512 GB
Linux	VMware on the same Mac, 2 vCPU / 8 GB

Do not compare seconds across machines. The comparison is the two bars inside one machine.

It was neither the disk nor the CPU

On Windows the first suspect was file placement and fragmentation. It was not that. Reading the same 3 GiB with no tool in the way gave 1159 MB/s at the start of the 51 GB file, 1134 MB/s in the middle and 1168 MB/s at the end. Flat. This drive does 1.1 GB/s raw, so reading 51.25 GB has a floor of about 45 s.

The second suspect was thermal throttling — a 15 W ultrabook under five minutes of load is a fair thing to suspect. Also not it. The clock read 2803 MHz before and after, and running the same 3 GB search five times in a row gave 7.1 s from the second run onward, steady.

What was left was the reading. And adding --no-mmap turned 282.84 s into 97.83 s.

Why (this part is a guess)

A memory map makes a file look like memory. In practice, every page you touch is fetched from disk by the OS. When the file fits in RAM this is excellent — it removes a copy.

It stops being excellent when the file does not fit. 51.25 GB fits neither in the Windows machine's 15.6 GB nor in the Mac's 32 GB, so the whole read becomes a long sequence of mapping pages in and throwing them out again. How much that costs apparently differs by operating system — that is our working explanation.

To be honest about it: this is reasoning backwards from the measurements, not something we verified. In particular we cannot cleanly explain why Linux flips — default faster for the light search, --no-mmap faster for the heavy one. If you know, we would like to hear it.

What to actually do

On Windows, if you are searching one file larger than about 10 GB with ripgrep, try --no-mmap once. It is one word, and it does not cost you anything.

rg --no-mmap -n 'pattern' huge.log

On macOS, do nothing. There is no difference.

On Linux, it is worth trying for heavy searches — regex, -v, -i and combinations of them. For a plain fixed string the default was faster in our runs.

Again: this is not a knock on ripgrep. Defaulting to mmap is the right call for the normal case — walking a source tree, reading logs of a few megabytes to a few gigabytes. What we measured sits at the far edge of the range: one file that does not fit in memory.

About our own tool, briefly

Since we were the other side of this comparison, here is where uvf sits.

uvf does not memory-map. It reads sequentially into a fixed buffer. As a result it behaves the same on all three systems — there is no good case and no bad case. On Windows at 10 GB with a fixed string, cold, ripgrep took 15.91 s and uvf took 7.22 s: 2.2×. On macOS the two are level (3.26 s against 3.32 s cold at 3 GB), so we read that gap as exactly the amount Windows loses to mmap.

The flip side is a real one. On Linux the default mmap is faster for light searches, and uvf is not using it. There is speed left on the table there. Right now we prefer having the same behaviour everywhere, but it is on the list.

uvf is free, runs on Windows, macOS and Linux, and is a single binary from GitHub Releases. If you want to search the same file repeatedly there is a paid edition (uvp) that builds an index beside it — but for searching, uvf is enough.

Trying it yourself

Two runs against the same file, dropping the cache in between.

# first (the default)
rg --mmap -n -F 'pattern' huge.log > /dev/null

# drop the cache
#   macOS  : sudo purge
#   Linux  : sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
#   Windows: RAMMap64.exe -accepteula -Et   (elevated)

# second
rg --no-mmap -n -F 'pattern' huge.log > /dev/null

Measuring without dropping the cache tells you nothing except that the second run is warmer, so do not skip that step. And check that both runs print the same number of lines — if they do not, something about the measurement is wrong rather than something about ripgrep.

I Thought We Lost at 3 GB — Turns Out We'd Won (As Long as the Cache Is Cold)

y4u — Wed, 16 Sep 2026 13:41:57 +0000

A grep on 3 GB belongs to ripgrep if the file is in the cache. If it is not, it belongs to uvp.

In the previous article I wrote that 3 GB is ripgrep's ground and I concede it. When the file fits in memory, an index has little to offer — believing that, I had been measuring warm every single time. Drop the cache and measure again, and it is the other way round. So here is the 3 GB picture again, that included.

Up front

I compared ripgrep / sed / gzip against uvp on a 3 GB file (3,032,812,644 bytes cut from OpenStreetMap Japan XML) under the same rules (Apple M4 / 32 GB / external USB SSD, /usr/bin/time -p real. Each search was measured cold — straight after sudo purge — and warm, the run immediately after, with the .uwvz already built).

Pure search swaps with the state of the cache. Warm, ripgrep (0.33 s vs 1.01 s); cold, uvp (3.26 s vs 1.25 s = 2.6×). It comes down to whether uvp's fixed cost — startup plus index load, just under a second — is larger or smaller than rg's whole scan, and the absolute gap is under 0.7 s either way
Even so, there are four situations where you win at 3 GB: ① fixing and saving (replace 3.5× and extract 12.6× over sed), ② operations rg does not have (sequence detection, ±N-line drill-down, click-to-drill), ③ logs kept as gz/zip (expand + rg 6.47 s vs .uwvz direct 0.68 s = 9.5×), and ④ not yet knowing what you are looking for (narrow in the CLI, look in the GUI, narrow again — 0.24 s per stage)
In short: on a day that ends with grep, use rg. The moment the investigation involves reading, narrowing, counting, fixing or saving, 3 GB becomes uvp's ground too

First, drop the cache and measure again

First, pure search, measured in two states. Cold is straight after sudo purge (the file is not in the page cache); warm is the run immediately after (it is in memory). uvp 1.6.3, ripgrep 15.2.0, with the .uwvz already built.

Cold (opening a file after a while)

Search	`rg`	`uvp`	rg ÷ uvp	Hits
Fixed string	3.26 s	1.25 s	2.6×	11,274
Ignore case `-i`	3.30 s	1.58 s	2.1×	940,482
Regex `-E`	3.32 s	1.70 s	2.0×	928,301
Regex, anchored	3.24 s	1.47 s	2.2×	222,072
Regex + ignore case	3.29 s	1.69 s	1.9×	491,643
Exclude `-E -v`	3.57 s	1.85 s	1.9×	1
Exclude `-v`, plain term	3.51 s	3.96 s	1/1.1	7,698,790

Six of seven go to uvp, by roughly 2×. The reason is exactly the reason at 10 and 50 GB: it reads 1/12 of the bytes (3,032,812,644 against 251,829,224).

Warm (asking again, right after reading it)

Search	`rg`	`uvp`	rg ÷ uvp
Fixed string	0.33 s	1.01 s	1/3.1
Ignore case `-i`	0.65 s	1.22 s	1/1.9
Regex `-E`	0.52 s	1.28 s	1/2.5
Regex, anchored	0.48 s	1.16 s	1/2.4
Regex + ignore case	0.61 s	1.40 s	1/2.3
Exclude `-v`, plain term	2.10 s	3.74 s	1/1.8
Exclude `-E -v`	3.13 s	1.74 s	1.8×

Warm, six rows go to rg. 3 GB fits in 32 GB of RAM, so one read puts the whole file in the cache and rg runs straight over it in 0.33 s, while uvp pays its startup and index load first. I concede this. The gap is 0.7 s, and conceding it costs nothing in practice.

One exception: -E -v is faster on uvp even warm (1.74 s against 3.13 s). Excluding works over every line, so ripgrep's cache advantage stops helping and the difference in bytes read shows through.

At 10 GB there is no warm state at all

10 GB	Cold	Warm
`rg` (fixed string)	10.96 s	10.89 s
`uvp` (`.uwvz` 1.16 GB)	2.37 s	1.55 s

The second run is the same 10.96 s. The file does not fit in 32 GB of RAM, so every run reads it off the drive again. "At 3 GB, use rg" holds only when the file fits in memory and someone read it a moment ago. Right after a reboot, on a log you last opened days ago, on a USB HDD or a network share, the difference in bytes read shows up even at 3 GB.

The problem is that an investigation almost never ends after one grep.

① A day you fix and save — when the job writes output, bytes read matter directly

Search is "read and count". Extract and replace are "read and write out". That changes things.

Test	CLI	`uvp`	Ratio
Extract hit lines (`sed -n '/term/p'`)	9.96 s	0.79 s	12.6×
Replace all (`sed 's/A/B/g'`)	15.04 s	4.28 s	3.5×
Same (`rg --passthru -r`)	—	4.28 s	1.7×

sed processes one line at a time, so even 3 GB takes 10–15 s. uvp pulls only the lines it needs out of a 0.25 GB index and writes them, so it wins even at 3 GB. The replace output is byte-identical to sed and to rg --passthru -r in every combination tested.

Add up a day's flow — "investigate, fix, save" — and it looks like this.

Open → 2 queries → replace and save (3 GB)	Total
rg + sed	17.7 s
uvp (including index build)	7.6 s (2.3×)

uvp's 7.6 s includes 1.2 s to build the index on the first run. It still wins. One replace turns sed's 15 s into uvp's 4.3 s, and next to that 10-second gap the 1.2 s for the index is noise.

② A day you use something rg does not have

Some rows could only go into the comparison as "reference", because rg has no equivalent command.

Operation	`uvp` (3 GB)	In rg you would…
Sequence detection `-seq login,timeout,restart` (only flows in this order)	0.99 s	write a state machine in awk
Second term within ±N lines `-C 3 <term2>` (co-occurrence nearby)	1.23 s	`rg -C 3 \
Click a tally value to drill into it	instant in the GUI	read {% raw %}`rg -o \
Jump from a hit to the original text with line numbers	instant in the GUI	note the line number from {% raw %}`rg -n`, then `sed -n 'Np'`

Before "faster or slower", these are rows where the comparison itself does not exist. "Only the flows where a timeout follows a failed login within three lines", "only the hits with another term within five lines" — use any one of these in an investigation and whether the file is 3 GB stops mattering.

③ A day the logs are kept as gz / zip — stop expanding them every time

Logs are normally kept gzipped. To search them you expand and then rg — and zgrep does not save you the expansion itself.

I compressed the same 3 GB with gzip -6, zip -6 and .uwvz, and compared expand-then-search with search directly.

3 GB	Time to compress	Size	Search (expand + rg / direct)
gzip -6	16.8 s	0.30 GB (1/10.1)	6.47 s (expand 3.49 + rg 2.98)
zip -6	19.4 s	0.30 GB (1/10.1)	8.71 s (expand 8.08 + rg 0.63)
`.uwvz`	1.2 s	0.25 GB (1/12.0)	0.68 s (direct search)

Same size as gzip, 14× faster to create, 9.5× faster to search. gzip expands everything every time you search; .uwvz carries an index and is searched without expanding. The second and third questions stay at 0.68 s. You could keep the expanded file around and get rg's 0.33 s from the second question on, but then there was no point compressing.

And .uwvz restores the original byte-for-byte with -extract (free). Change the storage format from gz to .uwvz and the per-search expansion disappears, even at 3 GB. At 50 GB it is 7 s against gzip's 2 minutes — 17×.

④ A day you do not yet know what you are looking for — this is the real point

In the table of losses, 東京 at 3 GB returns 11,274 hits. rg's 0.33 s does not include the time for a person to read those 11,274 lines.

There are two kinds of investigation. Checking whether a string is there — and the kind where you do not yet know what to search for. The second is trial and error itself: scan the 11,274, look around the interesting ones, narrow by another term, count, go back. In a terminal you rebuild the command each time for 0.33 s — except that what you actually pay is the time to rebuild it and the time to follow output scrolling past.

Put -open at the end of a uvp command and the CLI result is handed straight to the GUI.

uvp japan-dv-ai.uwvz '東京' -uniq 'k="([^"]+)"' -sort count -head 20 -open

The CLI finishes the search and the tally, and hands the result to the GUI through a temporary file (the GUI does not search again; v1.6.2)
The GUI reads the same .uwvz, so reopening does not read 3 GB (reopening a 47.73 GB file takes 0.02–0.07 s from the second time on; 3 GB is less)
Click a row in the list to jump to it in the text. ±N lines of context. Click a value in the tally to drill into it as the next stage
Narrow again with another term — 0.24 s per stage (measured at 50 GB; 3 GB is less)
Once you know what you are after, drop back to the command line and process in bulk. The GUI and the CLI use the same .uwvz, so moving between them costs no waiting

So the thing to compare is not "one grep, 0.33 s vs 1.01 s" but "total time and number of actions to reach the answer". rg alone means rebuilding the command, reading the output, rebuilding again for every re-narrowing. GUI alone means a slow open. Put an index between them so the CLI and the GUI use the same thing, and narrow broadly with a command, look closely on screen, drop back to a command connects with no waiting.

The CLI and the GUI call the same functions (drill-down, tally, sequence), so the count the CLI prints and the count the GUI shows always agree. That was verified byte-for-byte against rg / sed across 81 combinations plus 12 gz/zip inputs plus every 258 GB case, with zero defects on the product side.

Summary — four situations where you win at 3 GB

Situation	Measured at 3 GB
A day that ends with grep	Warm, rg wins (0.33 s vs 1.01 s). Cold, uvp wins (3.26 s vs 1.25 s = 2.6×). `-E -v` goes to uvp even warm (1.74 s vs 3.13 s)
① A day you fix and save	Replace 3.5×, extract 12.6× over sed. 2.3× on "open → 2 queries → replace and save"
② A day you need something rg lacks	Sequence detection, nearby co-occurrence, click-to-drill, line-number jump. No comparison exists
③ A day the logs are gz/zip	Expand + rg 6.47 s vs direct 0.68 s (9.5×). Compressing takes 1/14 of gzip's time
④ A day you do not yet know what you are looking for	Narrow in the CLI, look in the GUI, narrow again. 0.24 s per stage. The unit of comparison changes from "one grep" to "time to the answer"

"A single grep on 3 GB" is ripgrep's ground, and it can be conceded. The moment an investigation involves any one of reading, narrowing, counting, fixing or saving, it becomes uvp's ground — and past 10 GB, pure search wins by 5–10× as well.

The tools

uvp ships with UwView Pro v1.6.3 (Windows, macOS and Linux, one licence for all three; one-time or monthly; 14-day free trial, during which uvp and editing both work). Replace (-replace) is part of the Edit Upgrade. Restoring from .uwvz (-extract) is free. The free UwView ships uvf, which does search and -open only and has no index (v1.6.3 rebuilt it as a single pass: 3.32 s cold at 3 GB, level with rg).

All figures are my own measurements on my own machines, not an independent benchmark. ripgrep is among the fastest there is for searching across many files, and this article does not contest that. If you spot a configuration issue, tell me; I will check and correct.

Conditions

Apple M4 / 32 GB / macOS 26.3.1 / external SSD
ripgrep 15.2.0 / BSD sed / gzip and zip at level 6 / the cold and warm search figures are uvp 1.6.3, measured 2026-09-17; replace, compression and GUI figures are uvp 1.6.2 / sudo purge before every run, cooldown between runs
3 GB: 3,032,812,644 bytes cut from OpenStreetMap Japan XML. The .uwvz is 251,829,224 bytes (1/12.0); gzip is 300,890,910 bytes (1/10.1)
The search tables assume the .uwvz is already built. Including the build, the first run is 4.64 s cold (rg 3.26 s)
Everything here was measured on a Mac (Apple M4 / 32 GB / external USB SSD). Windows and Linux — a low-powered laptop, under VMware — may behave differently; those are being measured separately and will get their own article

Picking the One Place to Beat ripgrep — Read 1/9 of the Bytes, Then Hand the Result From the CLI to the GUI

y4u — Wed, 16 Sep 2026 10:16:05 +0000

You cannot out-engineer ripgrep.

ripgrep finishes a 258 GB, 4.5-billion-line file in 278 seconds. wc -l finishes the same file in 268. The gap is 4%, and for those 278 seconds the CPU is mostly asleep. When I saw that number, I knew there was no algorithmic path to catching up. So where do you fight?

Up front

I ran ripgrep and my own index-based uvp against a single 258 GB, 4.5-billion-line file under the same rules (Apple M4 / 32 GB / external SSD, sudo purge before every run, /usr/bin/time -p real, all figures second run = index present).

ripgrep runs at 1.04× the time it takes to read the file once. It does not saturate a single core (user/real = 0.04). There is no way to beat that on efficiency
There is exactly one place to win: the number of bytes read. Index plus compression takes 258.68 GB down to 28.6 GB (1/9.04). Read 1/9 the bytes, and the time follows: 278.35 s → 35.51 s
The first question is always a loss (building the index takes 333.67 s). Break-even is at question 1.37. At 3 GB, ripgrep wins no matter how many questions you ask
One by-product. The moment you hold an index, the CLI result can be handed straight to the GUI and narrowed again there as many times as you like. A drill-down stage costs 0.24 s (50 GB, v1.6.2), which makes that round trip practical

I once wrote about picking a single axis against commercial editors instead of counting features. This is the CLI edition. The opponent is ripgrep, and the losses come first.

The premise: fight fair and you lose

Before any comparison, the physical floor.

$ /usr/bin/time -p wc -l us-260726.osm
4509830821
real 268.73

268.73 s = 963 MB/s. That is the time to read 258 GB once. Against that, ripgrep:

Q	Term	Hits	`rg -c`
1	New York	100,492	275.80
2	Brooklyn	30,871	278.35
3	Central Park	14,306	276.75
4	Statue of Liberty	88	293.85
5	coffee_shop	36,506	346.21
		median	278.35

1.04× the physical floor. Same for 88 hits or 100,000. ripgrep is a tool that "finishes when it has finished reading", and the reading speed is set by the disk.

The CPU numbers make it sharper.

rg -c 'New York'   real 275.80   user 11.55   sys 51.60

Of 275 seconds, 11 were spent on the CPU. On a 10-core M4, that is 4% of one core. It does not need more — the disk cannot keep up.

That is the starting point. In an I/O-bound world, a faster CPU changes nothing, and neither does parallelism. Polish the algorithm all you like; the 268.73-second wall does not move.

(A side note: this is also why "written in Rust, therefore fast" is not a general rule. amber, which parallelises within a single file, takes 357.69 s on the same file — slower than ripgrep. Burning 25–36× the CPU time does not help when there is one disk. Measurements are in the 258 GB article.)

The one move left: read fewer bytes

If the disk is the ceiling, the only lever is how much you read.

uvp builds an index and a compressed copy first.

$ /usr/bin/time -p uvp convert us-260726.osm
real 333.67

333.67 s — 1.24× the physical floor — to read 258 GB once and produce a .uwvz. The result is 28,608,409,953 bytes = 1/9.04 of the original. Every later search reads only those 28.6 GB. The original 258 GB can be deleted.

	Bytes read	Floor at 963 MB/s	Measured (median)	Ratio to floor
`rg -c`	258.68 GB	268.73 s	278.35 s	1.04×
`uvp`	28.6 GB	29.72 s	35.51 s	1.20×

Both are pinned to their own I/O floor. The only difference is how many bytes each must read. And per byte, ripgrep is the more efficient one (1.04 versus 1.20). What wins is not the algorithm; it is the prepayment.

The CPU picture flips too.

rg   real 275.80   user  11.55   → user/real = 0.04 (waiting on I/O)
uvp  real  34.70   user 168.63   → user/real = 4.86 (~5 cores decompressing and searching)

The moment you read 1/9 the bytes, the disk has slack and the CPU becomes the bottleneck. The bound flips — that is what this approach really is.

The table of losses

Pick one axis and you lose on the others. All of them, up front.

Situation	Result	Why
First question	ripgrep 278.4 s vs uvp 369.3 s (1/1.33)	You pay 333.67 s for the index. A consistent 1.2–1.3× loss at every size (3 GB 1/1.18, 10 GB 1/1.31, 50 GB 1/1.29)
3 GB	ripgrep faster from the second question on too (1/2.3)	The file sits in the OS cache and rg scans it in 0.33 s. uvp always pays 0.6–0.7 s of startup plus index load. Absolute gap: 0.4 s
`-head N` (first few only)	`rg \	head` 0.05 s vs uvp 0.32 s
Efficiency per byte	rg 1.04× vs uvp 1.20×	As above. Decompression costs
Drill-down with context (`-C N`)	~8.5 s per stage (50 GB)	It looks outside the previous hit lines, so it re-reads the `.uwvz`. At ±0 it is 0.24 s

3 GB is ripgrep's ground, and I concede it. The gap is 0.4 s, so conceding costs nothing. The index starts to matter once the file no longer fits in memory.

Which question pays for the index

CLI(N) = N × 278.35            (rg, median of 5 questions)
uvp(N) = 333.67 + N × 35.58    (index build + mean .uwvz search)

N* = 333.67 ÷ (278.35 − 35.58) = 1.37

Questions N	ripgrep	uvp	rg ÷ uvp
1	278.4	369.3	1/1.33 (ripgrep wins)
2	556.7	404.8	1.38×
5	1,391.8	511.6	2.72×
20	5,567.0	1,045.4	5.33×

It flips at the second question. The formula applies to any index-based approach (T_index ÷ (T_scan − T_search)). 10 GB and 50 GB show the same shape: from the second run on, 5.5–6.9× at 10 GB and 7.3–10.2× at 50 GB.

That is the whole "speed" story. Honestly, on its own it would barely merit an article. Of course the second search is faster once you have built an index.

The by-product: the CLI and the GUI became one surface

The prepayment bought something I had not planned for.

Put -open at the end of a uvp command and the CLI's result is handed straight to the GUI.

uvp us-260726.osm.uwvz 'New York' -C 0 -grep hospital -v -grep school -v -grep Brooklyn -open

Step by step:

The CLI searches the 258 GB file (via its 28.6 GB index), narrows through three stages to 35 lines — 34.88 s
Those 35 lines are handed to the GUI through a temporary file. The GUI does not search again (v1.6.2; until then only the conditions were passed and the GUI re-ran the search, so on a huge file you waited for the same search twice)
The GUI opens. The index already exists, so reopening does not read 258 GB
Click a row in the list to jump to it in the text. Show ±N lines of context. Narrow again with another term — that costs 0.24 s per stage (v1.6.2)

Step 4 is the point. In the interest of honesty: there was a period when only the drill-down was oddly slow in the comparison against rg. Single searches ran 7–9× faster, yet stacked drill-down stages managed only 3–4×. It turned out one code path was re-reading per stage; v1.6.2 fixes it. At 50 GB, three stages went from 29.46 s to 8.10 s — 0.24 s per stage. Three stages on 258 GB now take 34.88 s, the same as a single search (34.85 s). The cause and how it was isolated will get their own article.

With that, the following workflow finally holds together:

Narrow in a script, look at it as a person, and narrow again while looking.
Not waiting 278 s for every re-narrowing (CLI only), not re-reading 258 GB every time the GUI opens (GUI only) — build the index once, and let both the CLI and the GUI use it.

A grep pipeline is "one pass no matter how many stages", and that is the right design for a CLI. But change the condition and try again and it is another full pass. The GUI is quick to change conditions but slow to open. Put an index in between and both of those "slow"s disappear.

Because the CLI and the GUI call the same functions (drill-down, tally, sequence), the count the CLI prints and the count the GUI shows always agree. Verification: 81 command combinations plus 12 gz/zip inputs plus every 258 GB case were checked byte-for-byte against rg / sed, with zero defects on the product side.

What was done, in short

ripgrep cannot be beaten on efficiency. 1.04× the physical floor, 4% CPU. It is finished
The only place to win was "bytes read". Read 1/9 and you get 1/8 the time — no more, no less
The first question is always a loss. Break-even at question 1.37. At 3 GB you lose every question. That ground is conceded
As a by-product of the prepayment, the CLI and the GUI share one index and pass results between them
The one place I picked is not "fast" but "inside a single investigation, moving between CLI and GUI without waiting"

The tools

uvp ships with UwView Pro v1.6.2 (Windows, macOS and Linux, one licence for all three; one-time or monthly; 14-day free trial, and uvp works during the trial). The free UwView ships uvf, which has no index — search and -open only, and no speed to speak of.

All figures are my own measurements on my own machines, not an independent benchmark. ripgrep is among the fastest there is for searching across many files, and this article does not contest that. It is only about asking a gigantic single file many questions. If you spot a configuration issue, tell me; I will check and correct.

Conditions and commands

Apple M4 / 32 GB / macOS 26.3.1 / external SSD (963 MB/s)
ripgrep 15.2.0 / uvp 1.6.1–1.6.2 / sudo purge before every run, 150 s (258 GB) or 60 s (50 GB) cooldown between runs
OpenStreetMap USA XML 258,679,440,228 bytes / 4,509,830,821 lines; OpenStreetMap Japan 51,254,526,392 bytes

/usr/bin/time -p wc -l us-260726.osm                       # physical floor
LC_ALL=C rg -c -F 'New York' us-260726.osm                 # rg
uvp convert us-260726.osm                                  # build the .uwvz index
uvp us-260726.osm.uwvz 'New York' --no-line-number         # uvp (second run)
uvp us-260726.osm.uwvz 'New York' -C 0 -grep hospital -v -grep school -v -grep Brooklyn -open

The Morning the Monthly Batch Died — Four Realities of Running a Data Pipeline

y4u — Tue, 15 Sep 2026 16:12:08 +0000

line 48,213,996. That's all you got.

Can you look at that line right now?

The pipeline hands you a coordinate. The thing it points at is somewhere else entirely.

Pipeline failures don't behave like application failures. What breaks isn't your code — it's the input data, produced outside your control, and the evidence is one line inside tens of gigabytes. Four scenes here — reading around the line that died, hunting the odd rows in a training corpus, reopening the same monthly input every month, and looking at the original before you transform it — each starting from what actually jams.

Up front: In UwView Pro you type the line number and land on 48,213,996 — and you land in the original, not an extracted fragment, so you can widen the context as far as you like without re-reading the file. The index and the compression are saved to a sidecar, so next month the same input file reopens with line numbers still attached in 0.02–0.07 s (measured on a 47.73 GB text file; one specific setup, results vary). When you move on to fixing things, the original stays byte-identical: edits accumulate in a diff file, and you can stop halfway and resume tomorrow (details at the end)

This article is about reading the raw data on either side of a pipeline, on your own machine. What you may handle, move, or retain is governed by your organisation's policy and your client's.

1. All that's left is a line number

Situation

The monthly batch dies. The alert is waiting for you in the morning.

You open the log. There's one line in it.

ERROR: failed to parse record at line 48213996: unexpected field count (expected 12, got 13)

That looks like enough. A line number, an expectation, an actual. And you still can't fix it.

Why there are thirteen fields is not something you can determine without seeing the line. A delimiter inside the data, an unclosed quote, a missing newline that welded two records together — all of these happen, and which one happened is decided by the bytes, not by reasoning.

The input file is 31 GB.

Why it happens

A parser is not built to report a broken line. It's built to stop.

What it can tell you is where it stopped and how reality differed from its expectation. Why the count became thirteen lies outside its concerns entirely.

Worse, the reported line number is not necessarily the line you want.

An unclosed quote makes the parser swallow dozens of lines as one record. The error surfaces where it finished swallowing; the cause is dozens of lines earlier
Mixed line endings make the parser's idea of a line differ from your editor's (part 13)
Header-skipping implementations report a number that's off by one
Parallel readers sometimes report a position relative to a chunk, not the file

So a line number is where you start looking, not where the answer is. Which is why "extract just that line" doesn't get you there — you need to be able to widen.

General tools, and where they stop

You go look at the line.

sed -n '48213976,48214016p' input.tsv

sed -n '48213976,48214016p;48214017q' input.tsv

awk -F'\t' 'NF!=12 {print NR": "NF}' input.tsv | head -50

That third command is the one that matters. Whether there is one broken line or four thousand changes the entire response — one you fix by hand, four thousand means you go argue with whoever generated the file.

Three limits.

First, sed makes you choose the range before you know what you need. You look at ±20, realise the cause is further back, and reissue with ±200. Each attempt counts 48,213,976 lines from the top of a 31 GB file. Every wrong guess is a full pass.

Second, the original coordinates vanish the moment you excerpt. Line 1 of that sed output is line 48,213,976 of the original, and that mapping now exists only in your head. Find something suspicious inside the excerpt and you're doing arithmetic to say where it really lives. What goes in the handover document is only as good as that arithmetic.

Third, awk over the whole file means one pass per idea. Check field counts; then quote counts; then line lengths. Every new way of looking costs another 31 GB read. Your investigation runs at the speed of I/O, not at the speed of thought (part 15).

2. The odd rows in a 50 GB corpus

Situation

You're assembling training data. The concatenated corpus is 50 GB.

You write the preprocessing, kick off a run. Hours later the loss curve has a shape it shouldn't. Or the outputs contain strings you don't recognise.

The cause is somewhere in the data. But "somewhere" isn't something the training side can localise for you.

Why it happens

Because you cannot state the criterion in advance.

The pipeline failure in part 1 had an explicit one: twelve fields. Corpus validation has none. You're looking for "odd rows", and odd gets defined after you see them.

List what actually turns up and the impossibility of enumerating it ahead of time becomes obvious.

Mass-duplicated lines — a crawler's error page, tens of thousands of times, byte-identical
HTML and JSON debris — tags the extractor missed, un-unescaped &, raw script tags
One absurdly long line — a Base64-embedded image arriving as a single multi-megabyte row
Encoding accidents — one source concatenated in Shift_JIS, read back as UTF-8 (part 3)
Personal data that shouldn't be there — not a matter of counts; one occurrence is one too many

That last one is what makes "sampling is fine" indefensible. Ten thousand clean sampled rows tell you nothing about the rest.

General tools, and where they stop

Start with distributions.

awk '{print length}' corpus.txt | sort -n | tail -20

sort corpus.txt | uniq -c | sort -rn | head -20

grep -cP '[\x00-\x08\x0B\x0C\x0E-\x1F]' corpus.txt

grep -cE '</?(script|div|span|br)\b' corpus.txt

Line length is the cheapest signal. A multi-megabyte line is itself the evidence.

Three limits.

First, sort does not survive 50 GB gracefully. Counting exact duplicates the honest way means sorting the file, which wants temp space on the order of the file itself plus the time to match. --parallel and -T help, but this is no longer in the "let me just check something" weight class.

Second, a count doesn't give you the thing. grep -c says 1,247. Whether those 1,247 are one problem or seven different problems is decided by looking. You can grep them out, but extraction erases their position in the corpus — and "a cluster from one source" versus "scattered throughout" is a judgement you can only make with position in hand.

Third, validation is a round trip. Find an anomaly, fix the preprocessing, rebuild, validate again. If one lap is forty minutes, every lap is forty minutes. The lap count is why corpus work never seems to end.

3. Reopening the same 31 GB every month

Situation

The monthly batch input is a large file with roughly the same shape every time.

You opened it last month. And the month before. Each time you do the same things: check the header, count records, look at the date range, search for the failure patterns you already know about.

The work itself takes thirty minutes. But opening takes several minutes, and each search costs tens of seconds. Every month, the same wait.

Why it happens

Because the wait is being accounted for as a one-time cost.

Three minutes is tolerable in the moment — it's a coffee. The problem is that it compounds across twelve months and however many people do this, and, more fundamentally, that a three-minute reopen makes people check less.

"While I'm here, let me also look at…" stops happening. Reopening is expensive, so you try to do it all in one go
You think of a search after closing the file, and you don't run it
The net effect: the thoroughness of your checking silently adjusts itself to the wait time

And one more thing. Monthly work carries no record of what last month's you actually checked. You rebuild the same queries from memory. Each rebuild differs slightly, and comparability quietly disappears — the same failure mode as an investigation procedure that lives only in one person's head.

General tools, and where they stop

It's routine work, so scripting it is the right instinct.

#!/bin/sh
f="$1"
echo "== header =="; head -1 "$f"
echo "== lines  =="; wc -l < "$f"
echo "== date range =="; head -2 "$f" | tail -1 | cut -f1; tail -1 "$f" | cut -f1
echo "== known bad patterns =="
grep -cE '\t\t|^\t|\t$' "$f"        # empty fields
awk -F'\t' 'NF!=12' "$f" | wc -l    # field count anomalies

Now the checks are fixed and comparable month to month. Writing the checklist down is worth something on its own.

Two limits.

First, a script only does the checks you decided on. When a number differs from last month, what happens next is not in the script. From there you open the file and look — and the wait arrives exactly there. Scripting removes the routine part only.

Second, the read-from-scratch structure is untouched. The script above walks 31 GB three times: once for wc -l, once for grep -c, once for awk. It did the same last month. Nothing from the previous read survives anywhere.

4. Look at the original before you transform it

Situation

A partner sends you 258 GB of XML to ingest.

There's a spec. There's a schema. You write the transform, put it on the pipeline. It runs for hours and dies. Or it completes and the record count doesn't match what you expected.

Either way the next step is the same. You go look at the original.

And 258 GB of XML doesn't open in most editors.

Why it happens

Because a spec describes what should be, and the file describes what happened.

Between organisations, those two diverge essentially always.

Tags not in the spec — the sender extended something and your copy of the spec predates it
Different handling of absence — empty element or no element at all. The spec is silent; the parser is not
Character references and escaping — &amp; is a fossil of a double-escaping step upstream
The format changes mid-file — receive several years in one delivery and you may be crossing a system replacement on the sender's side
A truncated tail — a transfer that stopped halfway looks perfectly fine at the head

That last one is the emblem of the whole problem. A defect you could see by glancing at the end of the file stays invisible because you can't open the file. You find out hours into the transform instead.

General tools, and where they stop

Inspect without opening.

head -c 2000 huge.xml
tail -c 2000 huge.xml

grep -oE '<[a-zA-Z_][a-zA-Z0-9_:.-]*' huge.xml | sort | uniq -c | sort -rn | head -30

xmllint --noout --stream huge.xml

grep -n -m 5 -A 10 '<UnexpectedTag' huge.xml

tail -c is cheap and decisive. If the file doesn't end in a closing tag, the investigation is over and you're asking for a resend.

Three limits.

First, grep doesn't know XML. You can count tag names, but nesting depth and parentage are outside its world. "Which record is this tag inside?" needs either a structure-aware tool or a pair of eyes.

Second, xmllint --stream stops at the first error. A hundred defects report as one. Fix, rerun, learn about the next one — a hundred times, at tens of minutes a lap.

Third, the going-and-looking round trip is heavy. You're told line 18 million, you look, it isn't enough, you look earlier, then earlier still. Whether that loop is cheap or expensive is decided entirely by whether the original opens.

What the four had in common

Scene	What you're given	Where it jams	General-tool approach	What's left over
Batch died	One line number	Reported position ≠ cause position	`sed` to excerpt ±N	Range must be chosen first; excerpting kills coordinates
Corpus validation	No criterion	"Odd" gets defined by looking	Length / duplicate / control-char distributions	Counts without the rows or their positions
Monthly input check	Last month's procedure	Checking thins out to fit the wait	Freeze the checks in a script	Re-read from zero; nothing carries over
Pre-transform check	A spec	Spec and file diverge	`head` / `tail` / tag inventory	Heavy round trips; one error at a time

Four different stages, four different people. The right-hand column rhymes anyway, because all four require seeing a specific place in the actual file, with its context — and that is exactly the expensive part.

In a pipeline, data is something that passes through. Read, transformed, written. A human looking at the bytes is the exceptional case — and so nobody has tooling for the exceptional case. When it dies, you improvise a window out of sed and awk.

Three conditions, then.

Jump by line number, and land in the original — with an excerpt, widening the context means rebuilding it
Make the second open cheap — neither validation nor monthly checking is a one-shot job. If you can't reduce the number of laps, reduce the cost of a lap
Keep the original when you start fixing — migration and data repair need the baseline available to the very end

Back to line 48,213,996. The line number wasn't insufficient. The route to that line just stopped at the edge of 31 GB.

The tool I use

I build UwView (free), a viewer that makes huge text readable, scrollable and searchable from the moment it opens. It doesn't pull the file into memory, so files larger than RAM open fine. Indexing runs in the background and line numbers appear when it completes (most viewers show you only the head until indexing is done). It never splits or extracts, so the original stays one file, unmodified.

Jump by line number. Type 48,213,996 and you're there — in the original, so you widen the context as far as you want. No choosing a range up front, and a wrong guess costs nothing
Reach the tail in one move. Part 4's "does it end in a closing tag" is answerable on 258 GB right after opening — and unlike tail -c, you can walk back upward from there
Switch encodings without reopening (UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16, auto-detected). Part 2's Shift_JIS contamination is confirmable without producing a converted copy

Beyond that is UwView Pro.

The index and compression are saved. This is part 3, directly. From the second open onward the file comes back with line numbers, instantly (0.02–0.07 s measured on a 47.73 GB text file; one specific setup, results vary). "Reopening is expensive, so I'll check less" stops being a decision you have to make
Save the queries, resume tomorrow. Against part 3's rebuilt-from-memory searches, the conditions themselves become an artefact. Sessions restore, so monthly work starts as a continuation
Drill-down search. Narrow a result set further, again. Part 2's "are those 1,247 one problem or seven?" gets sorted on screen without producing extract files. Tabs carry term (count) and the original line numbers survive to the last stage (drill-down article)
Tally (frequency ranking). Count occurrences captured by a regex and click a row to descend to it — close to part 2's distribution work, without walking the file once per idea. It does not sum or average; the lineage is grep -oE | sort | uniq -c
Store at roughly 1/9 and still search it. Keeping twelve monthly inputs a year becomes compatible with rechecking them (part 8)

And the part this series kept flagging — "once you've found it, fixing it means rewriting the original" — got an answer in v1.4.0 with Edit Upgrade. It lands squarely on migration and data-repair work.

The original is never rewritten. Edits and bulk replacements accumulate in a diff file (.ewvz). Fixing the broken row from part 1 leaves the baseline intact to the end
Repair cost doesn't scale with file size. In development measurement, 21,994 replacements across a 10 GB, 100-million-line file took 16.8 s (measurement article; macOS, external SSD, .uwvz-backed file, single run, results vary)
Stop halfway, resume tomorrow. Pause and continue without resaving the whole file — the property that multi-day work like part 2's corpus validation actually needs

Stated plainly: UwView is neither a validator nor an ETL tool.

Part 1's "pull every row with the wrong field count" is awk's job. This tool's turn comes after — putting the resulting line number onto the real thing
It does not understand CSV or XML structure. No sorting by column name, no type checking, no schema inference, no nesting validation. Expect a "huge CSV editor" or an "XML editor" and you'll be disappointed
Tally has no sum or average. Counting part 2's duplicates is sort and uniq
It does not run, schedule, retry, or monitor pipelines
Edit Upgrade operates on .uwvz-backed files, not on a raw CSV in place — you convert first, and it works as an add-on to a View License (it does not run standalone)
It handles text. Parquet and binary database dumps are out of scope

The same things, from the command line

v1.6.0 added a uvp command (and uvf for the free build). It uses the same .uwvz as the GUI, so an index built from the shell is already there when you open the file in the app. Mapped onto this article's four parts:

uvp input.tsv 'order_id=A1B2C3' -C 40

uvp input.tsv 'order_id=A1B2C3' -C 40 -open

uvp corpus.txt -uniq '^(.{0,30})' -head 20

uvp 2026-09.tsv 'ERROR' 'timeout' -out check-202609.txt.gz

uvp huge.xml.uwvz '<UnexpectedTag'
uvp huge.xml.uwvz -extract -out restored/   # restoring the original is free

Exit codes are grep's — 0 found, 1 not found — plus 2 when the 1,000,000-hit cap is exceeded. if uvp input.tsv 'ERROR'; then works as written, so it drops straight into part 3's check script.

Stated honestly: on the first question ripgrep is 15–20% faster, because uvp builds its index first. On a 3 GB file that fits in RAM, rg stays ahead on the second question too. uvp pays off past 10 GB, when you ask the same file more than one question (the measurements; Mac M4, external USB SSD, OpenStreetMap XML — one setup, results vary).

What it does is find things, show you the real thing, and let you repair it without breaking the original. If you're doing monthly input checks or multi-day migration repairs, try it.

14-day free trial (everything in View + Edit, no payment details)
UwView Pro (View) — persistent index, compressed-cache search and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly)
Edit Upgrade — non-destructive diff editing, bulk replace on huge files, pause and resume (add-on to a View License)

Links

Part 1 — Four standard tools that sink under a huge file: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
Part 3 — Four encoding traps and how to isolate them: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
Part 8 — Compressed storage and searchability at once: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
Part 9 — Reading structured data raw: https://uvp.y42u.net/en/blog/uwview-ps09-read-raw-structured-data-en/
Part 13 — Four techniques for inspecting huge data: https://uvp.y42u.net/en/blog/uwview-ps13-huge-data-inspection-en/
Part 15 — Four limits of command-line craft: https://uvp.y42u.net/en/blog/uwview-ps15-cli-craft-limits-en/
Part 17 — Friction between logs and your dev environment: https://uvp.y42u.net/en/blog/uwview-ps17-dev-env-log-friction-en/
Part 20 — Surviving an audit by design: https://uvp.y42u.net/en/blog/uwview-ps20-audit-log-retrieval-en/
Measuring bulk replace on 100 million lines: https://uvp.y42u.net/en/blog/uep-100m-lines-replace-all-en/
We shipped a uvp command (measured against ripgrep): https://uvp.y42u.net/en/blog/uvp-cli-release-vs-ripgrep-en/
Drill-down search: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
Sequence search: https://uvp.y42u.net/en/blog/uvp-sequence-search-en/
Source code (GitHub): https://github.com/amru195704/UwView

From the developer: A full list of my apps, Kindle books and open-source work lives at GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. The error messages, line numbers, file sizes and counts are illustrative and do not refer to any real dataset or engagement. Quote handling in CSV/TSV, line-ending behaviour, and XML character-reference handling vary by parser and library version. Command examples may need adjusting for your environment (GNU vs BSD, differences among awk, sed, sort, grep, xmllint, availability of grep -P). Always confirm option names and defaults against your local man. Figures described as measured come from one specific setup and are not a guarantee of the same result; disk type, filesystem, fragmentation, encryption, page-cache state and concurrent processes change outcomes substantially. Handling of training data and migration data is governed by your organisation's policy, your client's, and applicable law. If you spot an error, please leave a comment and I'll check and correct it.

"Please Produce the Logs from Three Years Ago" — Four Ways to Survive an Audit by Design

y4u — Tue, 15 Sep 2026 16:10:37 +0000

Two weeks to produce them.

You retained them, yes. But how many days does it take to get them back out?

Retention period and retrieval time are two different designs. Skip one and the other breaks.

Being asked for logs by an auditor runs on a different clock than incident response. An incident is about now. An audit is about three years ago, in a shape somebody else decided. Four scenes here — a request scoped by date, checking raw logs before ingestion, the economics of retention, and archives that outlive everything around them — each starting from what actually jams.

Up front: UwView Pro's sequence search returns only the places where terms appear in that order, so an audit trail like "request → approval → execution" becomes the query itself. Drill-down search stacks the narrowing and keeps the original line numbers all the way to the last stage, so an excerpt can carry the one thing a submission needs: which line of which original. The index and the compression are saved to a sidecar, so from the second open onward the file comes back with line numbers in 0.02–0.07 s (measured on a 47.73 GB text file; one specific setup, results vary). Everything runs on your own machine; the file is never sent anywhere (details at the end)

This article covers retrieving and reading a log you already hold. What you must retain, for how long, and whether a submission satisfies a requirement belong to your organisation's policy and the applicable law and regulators.

1. The retention period is met. The retrieval was never designed.

Situation

The request arrives: "Produce the operation logs for one account, for October 2023."

Policy says three years. You're inside it. The files exist. You go to open them, and stop.

That month's archive is 31 daily files, compressed, a few GB each. The names are sequential; the dates inside are unknown until you open them. And nothing here tells you which file that account appears in, or where.

So you decompress them one at a time and grep. Check free space, run one, read the output, delete, next. This does not finish today.

Why it happens

Because the policy only ever specified a period.

What's written down is "retain for N years." What isn't written down is "retrievable within N hours" — and what isn't written down doesn't get designed. So retention gets automated and retrieval stays manual.

There's a second mismatch. An audit request does not arrive in the units you stored things in. You stored by host, by date, by service. The request comes as "this account," "this transaction ID," "this window" — cuts that run across your storage layout. The way you filed it is not an index into the way you're asked for it. That's the jam.

Working with general-purpose tools, and where it stops

Narrow the file set first.

ls /archive/app-202310*.log.gz

for f in /archive/*.log.gz; do
  printf '%s\t' "$f"; gzip -dc "$f" | head -1
done

Then search across them without unpacking to disk.

zgrep -h 'user_id=A1B2C3' /archive/app-202310*.log.gz > hits.txt

zgrep saves you the unpacking step in the shell, but what it does internally is decompress-and-scan, sequentially. Run it over tens of gigabytes and it costs what that costs — the point of Part 8.

Three limits.

First, -h drops the filename and omitting it prefixes every line with one. A submission needs "which file, which line," and zgrep cannot give you the original line number at all (-n exists, but it counts within the decompressed stream).

Second, each extra condition is another pass. To confirm "that account made a configuration change somewhere in October," you filter by account, then look for the change within the result. zgrep A | grep B looks like one pass, but in practice every re-narrowing means reading the compressed source from the top again.

Third, and this one hurts most in practice: grep output has no context. Matching lines alone can't answer "what happened just before this action," so you end up wanting ±20 lines. Add -C 20 and the output inflates twentyfold — and you're back to a human reading 31 files' worth.

2. Look at the real thing before you write the ingest rule

Situation

Application logs are going into the log platform. You write the parse definition.

The spec documents the log format. You build the regex to match, and load. Ingestion runs. A few days later the dashboard shows about sixty percent of the volume you expected.

Lines that failed to parse were dropped. What kind of lines they were, you can't tell — they were dropped.

Why it happens

Because a spec documents one line, from the happy path.

Real logs contain lines no spec mentions.

Stack traces. One exception spans dozens of lines. From line two on there's no timestamp, so a line-oriented parser either discards all of it or counts each line as its own event.
Lines that differ by level. DEBUG carries an extra field — the kind of difference that never reaches a spec.
The day the format changed. A library upgrade moved a delimiter. During the transition, both shapes coexist in one file.
Mixed encodings. Lines that arrived from an external system still in Shift_JIS is a real situation, not a hypothetical (Part 3).
Broken lines. Cut at a rotation boundary, or half-written when the process died.

In an audit context this matters for a specific reason: "some lines didn't make it in" is itself a finding. When completeness is the question, "sixty percent is loaded" is not an answer.

Working with general-purpose tools, and where it stops

Count the shapes in the real file before you load it.

grep -cv '^[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}' app.log

awk -F'\t' '{print NF}' app.log | sort | uniq -c | sort -rn | head

awk '{print length}' app.log | sort -n | tail -5

The second one earns its keep. If you expect eight fields and sevens and nines show up, the definition is already broken — before you load anything.

Three limits.

First, you can only count what you thought to count. Field counts, sure. "What do the seven-field lines actually look like?" needs eyes on the real lines. awk 'NF==7' will pull them, and that is one more full pass through the file, per idea you have. Tens of gigabytes, once per idea.

Second, the meaning lives at the tail of the distribution. If three lines out of a million have a different shape, those three are the ones worth reading. sort | uniq -c is excellent at ranking the common cases and no help at all at finding the rare one — you still have to go look at those three.

Third, sampling isn't enough in some of these cases. head -10000 tells you nothing about a malformed line sitting forty gigabytes in. As Part 13 put it, inspection sometimes presupposes being able to see all of it.

3. Compression ratio and retrievability draw on the same budget

Situation

You review storage. Better compression means more years on the same disk.

You set a policy: last month uncompressed, a year at normal compression, everything older at high compression in the archive. Capacity visibly improves.

Six months later, an audit asks for something two years old. That's the high-compression tier. Decompression takes a while. The scratch volume isn't big enough, so you do it in chunks. What you saved on storage has come back as retrieval labour.

Why it happens

Because the cost you cut and the cost you added land in different ledgers.

Storage is billed monthly and visible. Retrieval labour appears as one engineer's overtime in the month an audit happens. Both are costs; they never sit in the same table, so you end up optimising exactly one of them.

And the trade-off isn't binary. There are at least three axes.

Compression ratio (drives storage cost)
Decompression speed (drives retrieval time; higher ratios tend to be slower)
Whether random access is possible at all — the one that gets overlooked

That third one is the crux. Most compression formats are built as streams, so "decompress only the middle" isn't available. To read the byte at 1 GB you decompress the first 1 GB. While that holds, "pull out only the part you need" is not merely slow — it doesn't exist.

So the real trade-off isn't "compression versus searchability." It's whether the format carries an index. With one, you can raise the ratio and keep retrieval fast.

Working with general-purpose tools, and where it stops

Measure on your own data first.

for lv in 1 6 9; do
  time gzip -"$lv" -c app.log > "/tmp/t-$lv.gz"
  ls -l "/tmp/t-$lv.gz"
done

time gzip -dc /tmp/t-9.gz > /dev/null

As a general tendency, raising the level makes compression much slower while decompression changes far less. Given that you compress once and retrieve many times, that asymmetry is on your side.

The numbers move a lot with format, implementation, and the data itself. Repetitive text like logs behaves nothing like already-compressed payloads. Building that table for your own data is the actual work of this section.

Two limits.

First, block-level indexing costs operations. Split, compress per block, and keep a separate catalogue of what's where, and random access works. Building, updating, and retaining that catalogue happens in a different budget line than the capacity you saved.

Second, the person deciding compression is not the person retrieving. Platform engineering watches capacity; audit response does the pulling. A policy set for one side's convenience is inherited by the other side six months later. The "delete, keep, or compress" call from Part 4 has to be made with that split in view.

4. Will it still be readable in seven years?

Situation

A financial system. Transaction logs, retained seven years.

You accumulate them, quietly. Five years pass. An audit asks for an older year. The files are there. You open one.

You can't read it.

The encoding is different — that system wrote Shift_JIS. The delimiter differs from today's format. The in-house tool that used to read these no longer runs; its OS left support. The person who wrote it has left.

Why it happens

Because the retention period outlives everything around the data.

List what changes in seven years and the size of the problem shows.

Encoding. Somewhere in there is the year the house standard moved from Shift_JIS to UTF-8. Logs with the same name differ on either side of it.
Format. Fields get added, delimiters move, timestamp formats change.
The tool that read it. Bespoke viewers stop running after an OS upgrade. When a runtime leaves support, keeping the tool alive becomes a question of keeping a whole environment alive.
People. Nobody guarantees that, seven years on, the organisation still contains someone who knows what field 4 meant.
Media. Optical and tape formats retire along with their readers.

And the last point is the essential one. What you need in seven years isn't "to read it with the tool of the day" — it's for a human to read what's inside. A log is text. As long as it's text, it is in principle readable. What makes it unreadable is almost always something around it.

Working with general-purpose tools, and where it stops

For long-term archives, store the instructions next to the data.

cat > /archive/2019/README.txt <<'EOF'
encoding : Shift_JIS (CP932)
delimiter: TAB
fields   : 1=timestamp(JST, yyyy/MM/dd HH:mm:ss) 2=account 3=action 4=result
note     : field 5 (session_id) was added on and after 2019-08-01
EOF

file -i old.log
head -c 3 old.log | od -An -tx1        # see a BOM with your own eyes

iconv -f CP932 -t UTF-8 old.log > /tmp/old-utf8.log

That README.txt is unglamorous and has the best odds of anything here of still working in seven years. Plain text needs no tool in order to read itself.

Three limits.

First, iconv makes a copy. Leaving the original untouched is correct practice (Part 16), but it doubles the volume and adds "which of these is the original" to your bookkeeping. Hand a converted file to an auditor and you acquire the job of explaining that nothing was lost in conversion.

Second, a mixed file can't be converted in one pass. If Shift_JIS and UTF-8 coexist inside one file, iconv either stops somewhere or emits mojibake — the situation covered in Part 11.

Third, nobody has a motive to write that README at the time it must be written. The person who suffers is someone else, seven years out. Which is why this can't be an individual good habit — it has to be a step inside the procedure that creates an archive.

What the four had in common

Scene	Shape of the request	What jams	With general-purpose tools	What's left over
Dated production request	"This subject, three years ago"	Storage units ≠ request units	`zgrep` across the set	No original line numbers. Every re-narrowing rereads
Pre-ingest check	"Is anything missing?"	Real lines the spec omits	Count field-count and length distributions	You can only count what you thought to count
Retention economics	"Cut the capacity"	Saved cost and added labour, different ledgers	Measure per compression level	Without an index, random access is impossible
Long-term archive	"From seven years ago"	Retention outlives everything around it	Ship the instructions; `iconv` to read only	Conversion makes copies. Mixed files resist one pass

Four different departments, four different clocks. The right-hand column rhymes anyway, because in all four the assumptions present at storage time are absent at retrieval time.

Whoever filed it knew the split, the encoding, the meaning of each field. Whoever retrieves it knows none of that. A stranger, seven years later, standing in front of tens of compressed gigabytes with no context and no query. That is the shape of audit response.

Three conditions, then.

Readable while still compressed — as long as retrieval requires scratch space and a wait, it stays a labour cost.
The original coordinates survive narrowing — a submission needs a correspondence ("line N of the original"), not a loose fragment. If every stage renumbers, you can't build one.
It finishes on your own machine — whether audit-scope logs may go to an external service is a policy question, not a technical one. Where the answer is no, this is the first filter anything has to pass.

Back to those two weeks. That was never a retention problem. It was the time when nobody storing the logs thought about the person who would have to get them back out.

The tool I use

UwView (free), which I develop, is a viewer that displays, scrolls, and searches from the moment it opens. It doesn't load the whole file into memory, so it opens files larger than RAM. The index builds in the background; when it completes, line numbers appear.

Of the three conditions above, the free edition covers the third.

Everything runs on your own machine. The file isn't sent anywhere. Where taking audit-scope logs off-site is the contentious part, that's a precondition rather than a feature.
It never writes to the original. No splitting, no extracting, so a file pulled from an archive stays one file, unmodified.
Encoding switches while the file stays open (UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16, auto-detected). For reading, the second copy iconv makes in section 4 stops being necessary.

The first two conditions are UwView Pro territory.

Sequence search: only the places where w1 → w2 → w3 appear in that order. The audit-shaped sequences — "request → approval → execution," "login → privilege change → logout" — become the query directly (the implementation post). Honestly: each stage scans forward from the previous hit, so it costs roughly what a full-text search costs.
Drill-down search: narrow a result by another term. Section 1's "filter by account, then find the configuration change" becomes a second stage instead of a second read of the source. Tabs show term (count), and the original line numbers survive to the last stage (the drill-down post).
±N is per stage: ±1 while narrowing, ±20 on the stage you actually read. You choose the width after you know what you need, so section 1's "-C 20 inflates the output twentyfold" doesn't happen (free edition is fixed at ±1; variable ±N is Pro).
Tally (frequency ranking): counts the values a regex captures, ranked, click a row to descend to it. Close to section 2's field-count distribution, without a pass per idea. It does not compute sums or averages — its lineage is grep -oE | sort | uniq -c, not awk.
The index and the compression are saved: from the second open onward the file comes back with line numbers, instantly (0.02–0.07 s measured on a 47.73 GB text file; one specific setup, results vary). Audit work is never read-once. Follow-up questions, clarifications, writing the submission — the same file, reopened across days.
~1/9 storage that stays searchable: directly against section 3's "no index, no random access." Pro's sidecar (.uwvz) carries a per-block offset table, so you jump to the part you need while it stays compressed — no decompress-from-the-top scan the way zgrep does it (Part 8).

The sidecar also keeps an XxHash3 table per compressed block and verifies on every decompression, so bit rot during long storage, or a copy that was truncated in transit, won't be read straight past unnoticed. This is not tamper detection. XxHash3 is a non-cryptographic hash chosen for speed. What tamper resistance needs is sha256sum plus separated custody; this check exists to catch accidents (Part 16).

Where it honestly stops

UwView is a viewer. It is not a SIEM, not a log management platform, not an audit-trail system.

It doesn't write your ingest definitions (section 2). No parse validation, no loading, no automatic detection of what got dropped.
It doesn't automate retention policy (section 3). No generation management, no deletion, no inventory reports.
Writing the README.txt in section 4 is your job. There's no metadata management.
It records no access trail — who opened what, when. If your audit requires viewing logs, this tool alone cannot satisfy it.
No cross-log correlation, no alerting, no report generation, no threat-intel matching.
It handles text. Binary database dumps and disk images are out of scope.

One more constraint, stated plainly for people who touch archive storage. Pro's sidecar (.uwvz) is created as a new file beside the original. The original itself doesn't change by a single byte, but if your practice is that nothing new appears inside the archive area, copy the file to a working volume before opening it. (The sidecar keys on the original's length and mtime, so it invalidates itself automatically if the original changes.)

What this tool covers is the step before all of that: read the stored raw log, as it is, on your own machine, with the original coordinates intact, with your own eyes. If you're at the scale that needs correlation and alerting, that's a different product's job.

For reference, non-destructive differential editing (Edit Upgrade) exists as a separate licence, but every one of this article's four sections is read-only work. What you need here is the View side.

The same things, from the command line

uvp /archive/app-20231015.log 'user_id=A1B2C3' 'config.change' -C 20 -out audit-202310.txt

uvp /archive/app-20231015.log -seq 'REQUEST,APPROVE,EXECUTE' -C 5

uvp app.log -uniq '^(.{0,10})' -head 20

uvp /archive/2023/app-20231015.log.uwvz 'user_id=A1B2C3'

uvp /archive/2019/old.log.uwvz -extract -out /tmp/restore/

Exit codes are grep's — 0 found, 1 not found — plus 2 when the 1,000,000-hit cap is exceeded. Section 1's "work through 31 files" becomes one uvp line inside for f in /archive/app-202310*.log; do ... done, with the exit code telling you which files actually held something. Stated plainly, though: uvp records no access trail either. An audit that requires a log of who viewed what, and when, is still not something this tool answers on its own.

And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly).

Links

Part 1: four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
Part 3: four character-encoding traps and how to isolate them: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
Part 4: deciding whether to delete, keep, or compress: https://uvp.y42u.net/en/blog/uwview-ps04-log-retention-decision-en/
Part 8: keeping logs compressed and still searchable: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
Part 11: reading legacy encodings in 2026: https://uvp.y42u.net/en/blog/uwview-ps11-legacy-encoding-euc-utf16-en/
Part 13: four techniques for inspecting huge data: https://uvp.y42u.net/en/blog/uwview-ps13-huge-data-inspection-en/
Part 14: the traces of an attack are in the raw log: https://uvp.y42u.net/en/blog/uwview-ps14-attack-traces-raw-logs-en/
Part 16: four principles for preserving, excerpting, and proving integrity: https://uvp.y42u.net/en/blog/uwview-ps16-log-as-evidence-en/
Part 19: four situations where tail -f can't keep up: https://uvp.y42u.net/en/blog/uwview-ps19-streaming-log-follow-en/
We shipped a uvp command (measured against ripgrep): https://uvp.y42u.net/en/blog/uvp-cli-release-vs-ripgrep-en/
Drill-down search — narrowing a result by another term: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
Sequence search — finding only what appears in that order: https://uvp.y42u.net/en/blog/uvp-sequence-search-en/
Archive plus session restore, as a working flow: https://uvp.y42u.net/en/blog/uwview-archive-session-restore-workflow-en/
Source code (GitHub): https://github.com/amru195704/UwView

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Actual audit response, internal control, and retention policy must follow your organisation's rules and the applicable law and regulators. Nothing here is advice on statutory retention obligations or on what a submission must contain. Retention periods, dates, account names, filenames, and field layouts are illustrative and describe no specific regime, organisation, or case. The behaviour of gzip, zgrep, grep, awk, iconv, file, and od varies by implementation (GNU/BSD/busybox), version, and build options, as do option names and defaults — check your own man pages. Statements about compression ratio and decompression speed are general tendencies; results change with the data, the format, and the implementation. Measured figures come from one specific setup and are not a guarantee of the same result — disk type, filesystem, fragmentation, encryption, page-cache state, and concurrent processes all move them substantially. If you spot an error, a comment is welcome and I'll check and correct it.

Following a Log That Won't Stop Growing — Four Situations Where tail -f Can't Keep Up

y4u — Tue, 15 Sep 2026 16:03:56 +0000

Three thousand lines a second is unreadable.

Your eyes manage maybe a dozen lines a second. So where did that ERROR go — the one that went past at two hundred times reading speed?

A log that keeps growing is hard in ways a static log isn't. Mid-incident triage, an overnight load test, an embedded serial console, a dev box turned up to TRACE: four different situations, and the dead end has the same shape in all of them. Here they are in order.

Up front: UwView Pro saves the index and the compression, so even on a file that keeps growing you can repeat "reopen and read to the end" in 0.02–0.07 s from the second open onward (measured on a 47.73 GB text file; one specific setup, results vary), and reapply a saved search condition as-is. Note that following a file while it is being written — real-time tail — lives in the free edition of UwView (details at the end)

1. The screen only shows "now" — after the ERROR has scrolled away

Situation

You're in the middle of an incident, with tail -f on the application log.

It pours. You thought you saw an ERROR and tried to follow it, but it's already off the top. Hit Ctrl+C and nothing after that point reaches the screen at all — while the file keeps growing.

Scrolling back up gets you a few thousand lines. At 3,000 lines a second, that's a few seconds' worth.

Why it happens

tail -f is a tool for forwarding appends to the end of a file. It has no notion of going backwards.

Then human reading speed multiplies in. You can follow maybe a dozen lines a second, and a handful if you're actually extracting meaning. At 3,000 lines a second, the gap between output rate and reading rate is more than two orders of magnitude. With a gap that wide, "watch all of it" was never on the table.

Adding a pipe to narrow things down introduces a second problem. When its stdout is a pipe or a file, grep switches to block buffering, so nothing reaches you until a few kilobytes have accumulated. A few seconds of lag mid-incident is a few seconds of lag on the decision.

Working with general-purpose tools, and where it stops

Turn off the buffering, then narrow.

tail -f app.log | grep --line-buffered -E 'ERROR|FATAL'

tail -f app.log | stdbuf -oL grep -E 'ERROR' | tee triage.log

tail -n 200000 -f app.log

tail -F app.log

--line-buffered and stdbuf assume GNU coreutils and GNU grep. BSD and busybox behave differently, so check your own man pages.

Three limits.

First, you have to choose the condition before you start. Once you're running grep 'ERROR', realising you also want WARN doesn't help: the WARN lines that went by while you were watching only ERROR never appeared. The stream doesn't wait, so there's no do-over.

Second, you cannot go backwards. -n 200000 rewinds to a point at the moment you start. Deciding ten minutes in that you want to see ten minutes ago means stopping tail -f and running something else — and while it's stopped, you see nothing.

Third, there are no line numbers. tail's output doesn't carry them. Appending grep -n gives you "which line arrived on the pipe," not a coordinate in the original. Not being able to write "behaviour changes from this line of the original" in the incident report is exactly the excerpting problem from Part 16.

2. The overnight load test — picking up "yesterday's" thread first thing

Situation

A 24-hour load test, unattended overnight.

You arrive to find an 18 GB log. The monitoring graph says latency spiked some time around 3 a.m. The terminal where you'd left tail -f running shows — a dropped session. The scrollback wrapped hours ago.

That afternoon a developer asks to look at the same window from a different angle. You face the 18 GB again.

Why it happens

Three things stack up.

Terminal scrollback wraps by line count. Defaults sit in the thousands to tens of thousands. At a few hundred lines a second, that's minutes. The screen you were "watching" isn't kept.

ssh sessions drop. A network blip, a sleeping laptop, a VPN reconnect. None of it was designed to survive eight unattended hours.

And nothing records how far you got. This is the one that really costs. Picking up the next morning and re-examining in the afternoon both start from zero, because there's no record of where you were. Reopening the same 18 GB several times a day is Part 10's "ten reopens a day," transplanted into an overnight test.

Working with general-purpose tools, and where it stops

Keep the screen somewhere durable, and cut by time.

tmux new -s load -d 'tail -f run.log | tee -a session.log'

sed -n '/2026-09-13T03:/,/2026-09-13T04:/p' run.log > 3am.log

awk '$0 >= "2026-09-13T03:00" && $0 < "2026-09-13T04:00"' run.log

Three limits.

First, tee -a means storing it twice. Another log appears alongside the 18 GB — and if you run the test nightly, it appears nightly (Part 8 is about the storage side).

Second, excerpting renumbers from 1. What line of the original is line 500 of 3am.log? You need that coordinate the moment you paste it into the test report.

Third, both sed -n and awk scan from the top every time. One pass over 18 GB is minutes, and comparing the 3 a.m. hour against the 4 a.m. hour is two passes. Jumping by timestamp was Part 7's subject — but when the target is still growing, the tail moves while you scan.

3. Reading days of serial-console capture end to end

Situation

You ran an embedded board for three days and captured everything on the serial line. 2.4 GB of text.

There are several reboots in there. Some were planned; some weren't. You want to know which reboots were the abnormal ones.

Your editor hangs on it. less opens it, but the screen goes strangely coloured and the layout breaks.

Why it happens

A serial capture isn't "text for people to read." It's the byte stream that went to the terminal, verbatim.

Control characters and escape sequences, raw. Colour codes like ESC[32m (ESC is 0x1B), cursor moves, carriage returns used to overwrite a line. The bootloader and kernel were writing to a terminal, so of course they're in there.
Mixed line endings. Bare CR, CRLF, and LF in one file, varying with how picocom / minicom / screen was configured. With bare CR, what counts as "one line" stops agreeing between tools — the same accident as the factory logs in Part 18.
No timestamps, or relative ones. The board may only have milliseconds since boot. Answering "when" across three days means keeping a separate mapping to wall-clock time.
Two different kinds of mojibake. Byte-level corruption from a baud-rate mismatch, and vendor messages emitted in Shift_JIS. The second is fixed by switching encodings; the first is not recoverable (Part 3).

Working with general-purpose tools, and where it stops

See what's mixed in, then count.

cat -v capture.log | head -40

perl -pe 's/\e\[[0-9;]*[a-zA-Z]//g' capture.log > clean.log

grep -c 'Booting Linux' capture.log

tr -d '\r' < capture.log | wc -l

Three limits.

First, cleaning it produces a different file. clean.log is readable, but it isn't the original, and dropping CR changes the line count outright. The coordinate you'd give the board vendor — "from this line of the original" — is lost in the cleanup step.

Second, baud-corrupted text doesn't match a search. If the o in Booting came through as a corrupted byte, grep 'Booting Linux' won't count that reboot. The answer says four; it might really be five. No amount of cleverness in the search term fixes that. Reading it through is what fixes it.

Third, it can't answer "when." With only milliseconds since boot, grep gives you a place but not a time. Finding "the second day, late afternoon" in three days of capture comes down to reading around it and estimating.

4. TRACE produced 40 GB in an hour — designing for the flow rate itself

Situation

To catch a bug that won't reproduce, you turned the log level up to TRACE and started a run.

40 GB in an hour. The disk warning fires. And the bug hasn't shown up yet. You want several more hours, which means clearing what's already there.

Why it happens

Log level multiplies rather than adds.

What was one line per request becomes 200 at TRACE. At 100 requests a second, that's 20,000 lines a second. Extrapolating linearly from how DEBUG felt is guaranteed to miss by an order of magnitude.

And everything around it is still configured for DEBUG.

Rotation can't keep up. "100 MB × 10 generations" cycles several times an hour, so the earliest part — the part you most want — is the first to go (evidence vanishing in the rotation gap was Part 16).
Compression eats CPU. The post-rotate gzip runs on the same host as the application and contaminates the latency numbers you're measuring.
The dev box doesn't have the disk. You can't reproduce production's flow rate locally (Part 17 is that friction).

Working with general-purpose tools, and where it stops

Emit less, decide retention first, discard while streaming.

<!-- TRACE only where you suspect (logback) -->
<logger name="com.example.payment" level="TRACE"/>
<root level="INFO"/>

logrotate -f "$LOGROTATE_CONF"

tail -F app.log | grep --line-buffered 'txnId=7f3a' > trace-7f3a.log

watch -n 60 'df -h /var/log; ls -l --block-size=M /var/log/app.log'

Three limits.

First, narrowing erases the lines you assumed were irrelevant. This is the nastiest one here. You set TRACE on com.example.payment only — and the cause turns out to be contention with a different module, whose evidence was never emitted. You raised the level precisely because you didn't know what was relevant, and now you're being asked to narrow. The order is backwards (the same contradiction shows up in Part 5's printf debugging).

Second, it disappears while you wait for rotation. More generations and the disk won't hold; fewer and the old end goes. Either way, you're forced to choose before you've decided anything.

Third, without a way to open 40 GB, you end up throwing away the log you just produced. This genuinely happens. You turn on TRACE, get 40 GB, grep a handful of lines because nothing will open it, and delete the rest. Raising the output volume loses its meaning at the point where it can't be read (Part 1's "won't open" comes back around here).

What all four had in common

Situation	How it flows	General-purpose approach	What's left over
Mid-incident tailing	Thousands of lines/s, two orders above reading speed	`tail -f \	grep --line-buffered`
Overnight load test	8 unattended hours, 18 GB	`tmux` + `tee`, `sed -n` by time	Stored twice. Renumbered from 1. No coordinate to resume from
Serial console	3 days, 2.4 GB, control characters mixed in	`cat -v` / `sed` to strip ANSI / `grep -c`	Cleaning yields a different file. Corrupted terms don't match
Dev box at TRACE	40 GB/hour, growing multiplicatively	Per-logger levels / `logrotate` / filtered `tee`	Narrowing suppresses the evidence. Rotation deletes it. Unreadable means wasted

Four unrelated situations. They jam in the same place because a flowing log has three properties at once:

"Now" and "a moment ago" don't fit on one screen. A follower sticks to the end and has no reverse gear; the tools that do go backwards (less, sed -n) aren't watching the end while they do it. Pick one and you lose sight of the other.
There are no do-overs. On a static file you can change the condition and try again as often as you like. On a flowing one, the lines that pass while you're changing the condition are simply gone from view. Hence "decide the right condition first" — an impossible request.
No coordinates survive. Not in the follower's output, not at the end of the pipe, not in the cleaned-up file. You can't get back to "that line" afterwards.

With those three together, "read it as it flows" and "go back and read it later" become two separate jobs — tail -f for the first, less or sed for the second. Two tools isn't the problem. The coordinate not surviving the handoff between them is the dead end.

What's needed is to make the second job cheap.

Open everything that exists right now, even while the file grows — look at the past in another window without stopping the follower.
Make reopening cheap — the faster the flow, the more often you reread to the end. If one wait is short, the count stops mattering.
Don't rebuild the condition — apply last night's filter to this morning's file as-is.

Back to that ERROR that scrolled away. It didn't vanish. It's still in the file, with a line number. It only left the screen. What couldn't keep up was the eye; the record held.

The tool I use

UwView (free), which I develop, is a viewer that displays, scrolls, and searches huge text from the moment it opens. It never loads the whole file into memory, so it opens files larger than RAM. The index is built in the background; when it finishes, line numbers appear.

For flowing logs, here's what the free version covers.

It opens files that are being written. It opens for shared reading, so you can read while the application keeps appending. Section 1's "look at ten minutes ago in another window without stopping tail -f" works.
Real-time tail — following appends — is in the free version. It detects growth and auto-scrolls to the end. As noted below, Pro does not have it (desktop only; it doesn't work in the browser build).
How much you can read isn't decided by RAM. Section 4's 40 GB, section 2's 18 GB, and section 3's 2.4 GB all open unsplit (measured ceiling on the free version: 47.73 GB, about 890 million lines — one specific setup).
It never writes to the original, and never excerpts or splits it. No clean.log from section 3, no 3am.log from section 2, which means line numbers stay the original's.
Encoding can be switched while the file is open (UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16, auto-detected). Section 3's "only the vendor messages are Shift_JIS" needs no intermediate file.
Highlighting colours lines rather than removing them. Colour section 1's ERROR|FATAL and you can pick them out of the stream without filtering — and because nothing is removed, their surroundings stay on the same screen.

Section 2's "pick up where I left off" and section 4's 40 GB of retention are UwView Pro territory.

The index and the compression are saved: from the second open onward, the same file comes back instantly, with line numbers (0.02–0.07 s measured on a 47.73 GB text file; one setup, results vary). When the workflow is "reopen and read to the end," the number of reopens is the cost. Section 2's morning and afternoon, and section 1's repeated review, are exactly that.
Search conditions can be saved and reapplied: last night's filter goes straight onto this morning's file. Section 2's "from zero every time" disappears (the filter popup).
Drill-down search: filter a result by another term, and another. The original line numbers survive to the last step. Section 1's "started with ERROR, now I also want WARN" becomes something you can do afterwards, without racing the stream (drill-down search).
±N is independent per step: ±1 while narrowing, ±200 while reading. Section 3's "found the reboot line, now show me the 100 lines before it" works directly (the free version is fixed at ±1; a variable ±N is Pro).
Sequence search: find only the places where w1 → w2 → w3 appear in that order. Section 3's planned and unplanned reboots can be separated by the order of the messages that precede them (how it works). Honestly: each step scans the text from the previous position, so it takes about as long as a full-text search.
Tally (frequency ranking): counts per captured value, and clicking a row descends to where it occurred. It's section 3's reboot count, with the difference that you can get back to the actual lines after counting.
Stored at roughly 1/9 and still searchable: section 4's 40 GB an hour stays compressed and searchable, without zgrep's decompress-as-you-scan on every query (Part 8).
Larger files: 258.68 GB and 4.5 billion lines measured on Pro (one specific setup; results vary).

The honest limits

This article is about following a log, so the most important limit comes first.

UwView Pro does not have real-time tail. Following appends lives in the free edition of UwView. Pro manages edits as a diff file (.ewvz), which sits badly with an original that keeps changing, so the feature isn't there. If you need to follow a log while it's being written, use the free version; if you only need to reopen and read to the latest, Pro does that fine. Worth checking before you buy.
No alerting and no notifications. "Page me when FATAL appears" is monitoring's job. Section 1 assumes a human is watching.
No automated cross-log correlation. Lining section 2's test log up against the monitoring graph belongs to another tool.
No automatic stripping of control characters or ANSI escapes. Section 3's capture is displayed as it is — read through, not scrubbed.
No involvement in log levels or rotation. How much section 4 emits is the application's business.
Lines display up to the first 8,192 characters. On a file where one line gets extremely long — section 3's bare-CR capture, for instance — the rest is elided.
Text only. A capture recorded in binary has to be written out as text first.

All four items above are read-only work. Non-destructive diff editing (Edit Upgrade) exists as a separate licence, but what you need here is the View side.

The same things, from the command line

uvp app.log -E 'ERROR|FATAL' -C 5

uvp app.log -E 'ERROR|WARN|FATAL' 'txnId=7f3a' -C 5

uvp run.log '2026-09-13T03:' 'latency' -C 10 -out 3am-latency.txt.gz

uvp capture.log -seq 'reboot: Restarting system,Booting Linux' -C 10

uvp app.log.uwvz -uniq 'txnId=([0-9a-f]+)' -head 20

uvp app.log.uwvz 'txnId=7f3a' -open

Exit codes are grep's — 0 found, 1 not found — plus 2 when the 1,000,000-hit cap is exceeded. Put if uvp run.log 'FATAL'; then at the end of section 2's overnight job and the morning starts with the one thing you needed to know. Stated plainly, though: there is no tail equivalent in the CLI either. Following a log while it's being written is the free UwView's real-time tail; what uvp covers is the other side — taking the stream back, fast, in a form you can keep.

Links

Part 1: four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
Part 3: four character-encoding traps and how to isolate them: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
Part 5: four practices for living with development logs: https://uvp.y42u.net/en/blog/uwview-ps05-debug-log-practices-en/
Part 7: four ways to read a timestamp as a weapon: https://uvp.y42u.net/en/blog/uwview-ps07-timestamp-driven-triage-en/
Part 8: keeping logs compressed and still searchable: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
Part 10: four things to set up for the you at 2 a.m.: https://uvp.y42u.net/en/blog/uwview-ps10-oncall-night-preparation-en/
Part 16: four principles for preserving, excerpting, and proving integrity: https://uvp.y42u.net/en/blog/uwview-ps16-log-as-evidence-en/
Part 17: four points of friction between logs and your dev environment: https://uvp.y42u.net/en/blog/uwview-ps17-dev-env-log-friction-en/
Part 18: four fields where machines write the logs: https://uvp.y42u.net/en/blog/uwview-ps18-device-generated-logs-en/
Why search results got their own window (the filter popup): https://uvp.y42u.net/en/blog/uwview-filter-popup-jump-save-context-en/
We shipped a uvp command (measured against ripgrep): https://uvp.y42u.net/en/blog/uvp-cli-release-vs-ripgrep-en/
Drill-down search — narrowing a result by another term: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
Sequence search — finding only what appears in that order: https://uvp.y42u.net/en/blog/uvp-sequence-search-en/
Source code (GitHub): https://github.com/amru195704/UwView

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Line counts, sizes, filenames, log contents, and configuration values are illustrative and do not describe any real product or incident. The behaviour of tail, grep, sed, awk, tr, cat, stdbuf, tee, logrotate, tmux, and watch varies by implementation (GNU/BSD/busybox), version, and build options, as do option names and defaults — in particular the difference between tail -f and tail -F, and whether --line-buffered or stdbuf is available at all. Check your own man pages and the official documentation. Before changing log levels in production, confirm disk capacity, rotation settings, and performance impact, and follow your organisation's change-management process. Measured figures come from one specific setup and are not a guarantee of the same result. If you spot an error, a comment is welcome and I'll check and correct it.

Ten Thousand Sensors, 1.4 Billion Lines a Day — Four Fields Where Machines Write the Logs

y4u — Mon, 14 Sep 2026 02:57:06 +0000

1.44 billion lines. In one day.

That's what ten thousand sensors produce when each one emits 100 metrics, one per line, every minute. The fault is in one of those sensors, during one of those minutes.

Four fields, four ways of looking for it: IoT, medical devices, PLCs on a factory line, and research simulation output. Completely different worlds — and they get stuck in exactly the same place.

Unlike logs a person wrote, logs a machine writes were never meant to be read. That single fact produces all four problems.

Up front: UwView Pro opens a 258.68 GB, 4.5-billion-line log and displays and searches all of it from the moment it opens, so even when line counts multiply out as devices × metrics × frequency you can follow the original, unsplit, by eye. It saves the index and the compression, so the second open comes back instantly with line numbers, and the file stays searchable at roughly 1/9 the storage (measured on one specific setup; results vary — details at the end)

1. IoT: line count is devices × metrics × frequency

Situation

Ten thousand sensors, reporting once a minute.

They're stored long-format, one reading per line. Each device emits 100 metrics — temperature, humidity, voltage, current, and so on — as 100 separate lines per report, so a day comes to 10,000 × 1,440 × 100 = 1.44 billion lines. The lines are short, so it's only tens of gigabytes; it's the line count that goes up an order of magnitude.

The plant tells you "one unit in Building 3 was behaving strangely yesterday evening." Nobody knows the device_id.

Why it happens

Machine logs are short lines in enormous numbers. Application logs written by people are the opposite — long lines, comparatively few of them. Same phrase, "huge log," opposite shape.

Long format compounds it. Each line is just timestamp,device_id,metric,value, so reading one device's state at one moment means reading 100 lines. Pivoting them side by side is possible, but that's work you do after you know where the anomaly is.

And the way anomalies surface makes it worse. Averaging ten thousand devices buries the one faulty unit. Aggregation answers "is the fleet healthy"; it does not answer "which one."

Working with general-purpose tools, and where it stops

The obvious approach is to narrow first, then look.

# Cut roughly by time (two hours of that evening)
grep -E '2026-09-11T1[6-7]:' sensor-20260911.csv > evening.csv

# Look at one metric
grep ',voltage,' evening.csv | head -50

# Pick off outliers mechanically (column 4 is the value)
awk -F, '$4+0 > 250 || $4+0 < 180' evening.csv | head

# Which device_id shows up most
awk -F, '$4+0 > 250 {print $2}' evening.csv | sort | uniq -c | sort -rn | head

When that lands, it's fast. When it doesn't, three limits show up.

First, you have to pick the threshold before you look. The example calls 180–250 normal, but that range is something you learn from the anomaly. The order is backwards. Guess wrong and you get either zero hits or several million — neither is useful.

Second, narrowing hides the neighbours. Pull only ,voltage, and the current and temperature readings from the same instant drop out of view. Device faults very often live in the relationship between metrics, and the filter severs exactly that. Adding ±N lines of context doesn't fix it either: in long format, "the same device's other metrics" aren't necessarily adjacent lines.

Third, line numbers disappear. What line of the original is line 1 of evening.csv? When you want to tell the vendor "behaviour changes from this line," you no longer have the coordinate (Part 16 covered exactly this cost of excerpting).

2. Medical: doing the first-pass triage in-house instead of handing it to the vendor

Situation

An analyser halts about once a week for no stated reason.

You contact the vendor; they ask for the logs; you send them; the answer comes a week later. The machine keeps halting in the meantime. The log exported from the device's service mode is 12 GB.

Why it happens

Instrument logs are written on the assumption that only the people who built the instrument will read them.

Internal codes. Lines like E-1042 or SUBSYS3 STATE=7. The meanings live only in the service manual.
Several subsystems in one stream. Control, transport, optics, and communications all write to the same file in time order. The lines are chronological; the context is per-subsystem (Part 5 covered the same shape with threads).
Old encodings. Instruments from domestic manufacturers may still emit Shift_JIS, which breaks the moment a UTF-8-assuming tool opens it (Part 3).
Export restrictions. If patient or examination identifiers are mixed in, you can't simply upload the file to a cloud analysis service.

The real reason these investigations get handed off isn't difficulty. It's that nobody in the building has a tool that opens 12 GB. You can't open it, so you can't look; you can't look, so the ticket says no more than "once a week, roughly in the evening"; thin information means more round trips. That's the loop.

Working with general-purpose tools, and where it stops

This is about as far as you get on your own.

# Check the encoding before converting — never convert on a guess
file -i device.log
iconv -f CP932 -t UTF-8 device.log > device-utf8.log

# Which error codes appear, and how often
grep -oE 'E-[0-9]{4}' device-utf8.log | sort | uniq -c | sort -rn | head -20

# Cut out what surrounds the halt
grep -n -B 200 -A 20 'E-1042' device-utf8.log | head -100

Both file's detection and iconv's behaviour vary by implementation and version, so check your own man pages.

Three limits.

First, running iconv adds an intermediate file whose line numbers may not match the original. Instrument logs in particular may use CRLF endings, or pad the tail with null bytes.

Second, you can count occurrences but not order. uniq -c throws the sequence away, and instrument faults cascade — what you actually want is which subsystem broke first (Part 2 is the same problem).

Third, these logs become maintenance records. A fault investigation ends up supporting a service report. If all you kept is a processed copy, that support is weaker (Part 16).

3. Manufacturing: the PLC alarm history won't open in the vendor's own software

Situation

The line stopped. To find out why, you export the PLC's alarm history.

Out comes a 3.8 GB CSV. The manufacturer's own history viewer spins for a while and then reports that it is out of memory. The spreadsheet doesn't even get that far — the 1,048,576-row ceiling means it can't take the whole file in (Part 1).

Why it happens

Vendor software is built for the volume that machine was expected to produce.

The expectation usually breaks for one of these reasons:

Contact chatter. Micro-vibration on a contact records the same alarm dozens of times per second, swelling the day to millions of lines.
Line expansion. More points are collected than before, while the retention setting stays where it was years ago.
Log everything. Somebody enabled all I/O points "so we won't be stuck later," and the alarm history quietly became a data logger.

On top of that, most vendor viewers read the entire file into memory before displaying anything. At 3.8 GB, installed RAM becomes the ceiling — and the machine on the floor is an office laptop that can't be upgraded.

Two more wrinkles specific to the factory floor: alarm names are often Japanese in Shift_JIS, and bare CR line endings are still in service. With bare CR, what counts as "one line" changes from tool to tool, and line numbers stop agreeing.

Working with general-purpose tools, and where it stops

# Look at both ends first (column order and time range)
head -5 alarm.csv
tail -5 alarm.csv

# Collapse repeats to see which alarms, and how many
cut -d, -f3 alarm.csv | uniq -c | sort -rn | head -20

# Pull out what surrounds the stop
grep -n '2026/09/11 14:2' alarm.csv | head -40

# If nothing opens it, split it
split -l 500000 alarm.csv part_

Three limits.

First, split cuts the context. A cascade that crosses a boundary lands in two files, and each piece renumbers from 1, so the numbers no longer correspond to what the vendor viewer shows.

Second, collapsing the chatter erases the fact that there was chatter. uniq -c makes it readable, but "480 occurrences in ten seconds" is a case where the density itself is the symptom. Readability and evidential value pull in opposite directions.

Third, there's no command line on that PC. Control-system machines frequently forbid installing anything. "Copy it to the analysis PC" means starting an export-approval process (Part 17 is about exactly that friction).

4. Research: look at the raw values before you plot them

Situation

A simulation ran for 48 hours and produced a 50 GB numeric log.

You load it and plot it, and part of the curve is suspiciously flat. A bug, a genuine physical result, or an output-side problem? The only way to tell is to look at the raw numbers.

Why it happens

Numeric logs break in ways that aggregation and plotting hide.

Four common forms:

NaN and inf slipping in. If the reader silently substitutes zero or a missing value, the plot simply looks flat.
Inconsistent notation. 1.0E-05, 1.0e-05, and 0.00001 in the same file; some parsers quietly drop one form.
Column count changing mid-file. Around a restart or a checkpoint resume, the header may be written again, or a column may appear.
A truncated tail. When a job is killed at the wall-clock limit, the final line ends mid-line.

Research logs also differ from operational logs in one decisive way: reproducing them takes 48 hours. "It was broken, so I'll rerun it" is expensive, which makes it essential to check without disturbing the original (Part 13 is the same shape).

Working with general-purpose tools, and where it stops

# Did it finish cleanly? (look at the shape of the last line)
tail -3 sim-output.dat

# Where does NaN / inf start (line number of the first one)
grep -n -m 1 -E 'NaN|nan|inf' sim-output.dat

# Did the column count change partway through
awk '{print NF}' sim-output.dat | uniq -c

# Was the header written more than once
grep -c '^#' sim-output.dat

Three limits.

First, awk '{print NF}' scans all 50 GB. That's minutes, during which you learn nothing. And the answer is only "the column count changed" — seeing where it changed means scanning again.

Second, grep -n gives you the line number but not the neighbourhood. Knowing there's a NaN at line 1,203,884,112 doesn't show you the 200 lines around it; for that you write a sed -n range and scan from the top again (Part 15 is where pipeline craft starts to hurt).

Third, the compute cluster and your analysis environment are different machines. Reading it in place with less over ssh means waiting a round trip for every screen. Pulling it local means transferring 50 GB and having room for it.

What all four had in common

Field	What the log really is	Why the line count explodes	General-purpose approach	What's left over
IoT sensors	Long-format time series, 4 columns	devices × metrics × frequency	`grep` by time and metric, `awk` for outliers	Threshold must be chosen first. Filtering hides the neighbouring metrics
Medical devices	Internal codes, subsystems interleaved	Continuous operation, all subsystems in one file	`iconv` to convert, `uniq -c` to count	Order is lost. Only a processed copy survives
Manufacturing PLC	Alarm history CSV	Contact chatter, expansion, log-everything	`cut` / `uniq -c` / `split`	Splitting cuts the cascade. The density that is the symptom gets collapsed
Research simulation	Continuous numeric output	Long jobs × high output frequency	`tail` / `grep -n` / `awk` to inspect	Minutes per scan. Reading around a hit means scanning again

Four unrelated fields. They jam in the same place because machine-written logs share three properties:

Line counts multiply. Raise devices, metrics, or frequency and it multiplies against the others, moving the order of magnitude. Logs people write grow by addition; these grow by multiplication.
Anomalies are local. They don't appear in the aggregate. They appear in one device, one minute, one line. Aggregation answers "healthy or not," never "where."
The original can't be damaged. Maintenance records in medicine, quality records in manufacturing, reproducibility in research, contractual retention in IoT. In all four there are moments when a processed copy isn't enough.

With those three together, "narrow it down, then look" stops working as an order of operations. The filter depends on seeing the anomaly, and seeing the anomaly requires the filter. That's the dead end shared by all four.

What's needed is a tool that lets you reverse the order.

Open the original, whole — look before narrowing. 1.44 billion lines, a 3.8 GB CSV, a 50 GB numeric dump, all the same.
Narrow while looking — take a filtered result and filter it again, keeping the original line numbers the whole way down.
Stay on this machine — patient identifiers, quality records, research data: none of it leaves.

Back to the 1.44 billion lines. That number doesn't mean "too much to read." It means you can't throw away the other 1.44 billion until you've found the one device and the one minute.

The tool I use

Of the three requirements above, the free version already covers the first and the third.

How much you can read isn't decided by RAM. An office laptop opens files of tens of gigabytes (measured ceiling on the free version: 47.73 GB, about 890 million lines — one specific setup). Section 3's "out of memory" no longer has a premise.
It never writes to the original, and never splits or excerpts it. No split from section 3, no intermediate files from section 4 — which means line numbers stay the original's. The coordinate you give the vendor in section 1, the maintenance record in section 2, reproducibility in section 4: all of them rest on the original's line numbers.
Everything runs on your own machine. The file is never sent anywhere. For section 2's instrument logs containing patient data and section 3's export approvals, that's often a precondition rather than a nicety.
Encoding can be switched while the file is open (UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16, auto-detected). Sections 2 and 3's Shift_JIS instrument logs open without an iconv intermediate (Part 3).
Highlighting colours lines rather than removing them. Section 2's interleaved subsystems can be untangled by eye with colour per subsystem, and because nothing is removed, the density stays visible — which is exactly what section 3's chatter needs.

The second requirement — narrowing while looking — and the case of reopening the same log repeatedly are UwView Pro territory.

Drill-down search: filter a result by another term, and another. The original line numbers survive to the last step. Section 1's "narrow by time without a device_id, then by metric, then by magnitude" works without cutting the original (drill-down search).
±N is independent per step: ±1 while narrowing, ±200 while reading. Section 4's "I have the NaN's line number but have to scan again to read around it" disappears (the free version is fixed at ±1; a variable ±N is Pro).
Sequence search: find only the places where w1 → w2 → w3 appear in that order. Section 2's "which subsystem broke first" becomes a condition over the order of the codes (how it works). Honestly: each step scans the text from the previous position, so it takes about as long as a full-text search.
Tally (frequency ranking): counts per captured value, and clicking a row descends to where it occurred. It's section 2's uniq -c, with the difference that you can get back to the actual lines after counting.
The index and the compression are saved: from the second open onward, the same file comes back instantly, with line numbers (0.02–0.07 s measured on a 47.73 GB text file; one setup, results vary). That's what a line-stop investigation like section 3 needs, since the same history gets reopened for days.
Stored at roughly 1/9 and still searchable: section 1's 1.44 billion lines a day and section 4's 50 GB output stay compressed and searchable, without zgrep's decompress-as-you-scan on every query (Part 8).
Larger files: 258.68 GB and 4.5 billion lines measured on Pro (one specific setup; results vary).

The honest limits

UwView is a viewer. It is not a time-series analysis tool and not an analytics platform for data loggers.

No statistics. Means, sums, standard deviations, moving averages — section 1's outlier test and section 4's sanity checks belong to awk or your analysis environment. What this does is find the location of the anomaly fast, by eye.
No plotting. Section 4's chart belongs to another tool. The premise of this article is checking the raw values before the plot.
No pivoting from long to wide. Section 1's reshape is out of scope.
No understanding of CSV columns. Section 3's alarm history is treated as text. "Show me only the third column" isn't something it does.
No decoding of instrument codes. The meaning of section 2's E-1042 lives in the service manual. All this gives you is reading the order and context of those codes quickly.
No live tailing, no alerting, no automated cross-log correlation.
Text only. Instruments that record in a binary format have to be exported to text first.

For completeness: non-destructive diff editing (Edit Upgrade) exists as a separate licence, but all four items above are read-only work. What you need here is the View side.

Links

Part 1: four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
Part 2: four techniques for tracing causality in logs: https://uvp.y42u.net/en/blog/uwview-ps02-log-causality-tracing-en/
Part 3: four character-encoding traps and how to isolate them: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
Part 5: four practices for living with development logs: https://uvp.y42u.net/en/blog/uwview-ps05-debug-log-practices-en/
Part 8: keeping logs compressed and still searchable: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
Part 13: four techniques for inspecting huge data: https://uvp.y42u.net/en/blog/uwview-ps13-huge-data-inspection-en/
Part 15: four limits of command-line craft: https://uvp.y42u.net/en/blog/uwview-ps15-cli-craft-limits-en/
Part 16: four principles for preserving, excerpting, and proving integrity: https://uvp.y42u.net/en/blog/uwview-ps16-log-as-evidence-en/
Part 17: four points of friction between logs and your dev environment: https://uvp.y42u.net/en/blog/uwview-ps17-dev-env-log-friction-en/
Drill-down search — narrowing a result by another term: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
Sequence search — finding only what appears in that order: https://uvp.y42u.net/en/blog/uvp-sequence-search-en/
Source code (GitHub): https://github.com/amru195704/UwView

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Fault investigation on medical devices and production equipment must follow each device's manual and service documentation and the manufacturer's instructions; nothing here is intended to replace vendor maintenance, only to support first-pass triage on site. Handling of patient data, quality records, and research data must follow your organisation's policy and applicable law. Line counts, sizes, error codes, and filenames are illustrative and do not describe any real product or incident. The behaviour of grep, awk, cut, split, iconv, file, sort, and uniq varies by implementation (GNU/BSD/busybox), version, and build options, as do option names and defaults — check your own man pages and the official documentation. Measured figures come from one specific setup and are not a guarantee of the same result. If you spot an error, a comment is welcome and I'll check and correct it.

Before You Carry Production Logs Home — Four Points of Friction Between Logs and Your Dev Environment

y4u — Mon, 14 Sep 2026 02:56:40 +0000

A build log, 480,000 lines.

One line in there explains the failure. What do you do before you reach for the scrollbar?

There are four places this gets stuck. All four happen when you bring a log into your own environment to read it.

CI build logs, the dump you carry home from production, file access across a WSL boundary, and the gap between one teammate's machine and another's. In none of these four cases is anything wrong with the log. What's wrong is the path between the log and the person reading it.

Up front: UwView opens a 480,000-line build log or an 80 GB dump without splitting it, and displays and searches it from the moment it opens. It never loads the whole file into memory, so how much you can read isn't decided by how much RAM you have. Pro saves the index and the compression, so from the second open onward the same file comes back instantly, with line numbers (0.02–0.07 s measured on a 47.73 GB text file; one setup, results vary). Everything runs on your own machine — the file is never sent anywhere (details at the end)

1. The build log is too long to find the failure in

Situation

The pipeline goes red. You open the log in the browser. 480,000 lines.

Searching for ERROR gives 137 hits. The first one says dependency resolution failed, which is a result, not a cause. Scrolling on, you notice the same log appears three times. Retries.

Why it happens

CI logs aren't written for humans to read. Four things stack up.

Interleaved parallel jobs. Several jobs' output lands in one stream. The lines are in time order; the context is not (Part 5 covered the same shape with threads).
ANSI escapes. Colour control characters sit inside the text, so grep 'ERROR' sometimes misses. [31mERROR does contain ERROR, but any regex that anchors on what comes before or after falls apart.
Progress bars. Output that overwrites itself with carriage returns becomes a single line of several megabytes in a file. That "480,000 lines" figure is already a lie.
Retries. The same failure is recorded two or three times, and nothing on screen tells you which one came first.

On top of that, browser log viewers usually only hold part of the file in a virtual scroller. Ctrl+F reaches what has been loaded. "I searched and it wasn't there" is not the same as "it isn't there."

Working with general-purpose tools, and where it stops

The basic move is to fetch the raw log and clean it up locally.

# 1) Pull the raw log with the CI's own CLI, bypassing the browser viewer
#    -> build-raw.log

# 2) Strip the colour control characters (ansi2txt from colorized-logs is easy)
ansi2txt < build-raw.log > build-plain.log

# 3) Expand carriage returns into newlines (progress-bar lines unfold)
tr '\r' '\n' < build-plain.log > build-lines.log

# 4) Look at only the first failure, with context
grep -n -m 1 -B 20 -A 5 -E 'ERROR|FAIL' build-lines.log

Command names and options vary by CI service and by implementation (GNU / BSD). If ansi2txt isn't available, a sed or perl substitution does the same job — but how you write the escape character changes with the shell and the implementation, so check your own man pages.

Three limits.

First, every cleanup step produces another file. You end up with build-raw, build-plain, build-lines — and their line numbers no longer agree. When the CI UI says "line 12,345," that's a different line in the cleaned-up file.

Second, retention. Most services keep logs for a few weeks. The moment you want to check "I think it failed in the same place last month," the comparison is gone.

Third, it's still big afterwards. Expanding carriage returns multiplies the line count. Your editor stalls trying to open it — straight back to Part 1.

2. The bug only happens in production — and carrying the log home has two constraints at once

Situation

It doesn't reproduce in dev, so you ask for the production log.

The estimate comes back: 80 GB. An hour to transfer, and you have 60 GB free on the SSD. Then ops mentions that the log contains personal data.

Why it happens

Two constraints of different kinds apply simultaneously — size and confidentiality — and each fix makes the other worse.

To solve size, you narrow or you split. But the filter you'd narrow by is only knowable after you've found the cause. You're carrying the log home because you don't know, yet you're asked for the filter before you leave. That's the ordering contradiction. In practice you grep out "just that day," and later discover the answer was outside it.

To solve confidentiality, you mask — and masking is irreversible. Once email addresses are redacted, "I want to correlate two actions by the same person" is no longer possible. And a masking regex tends to fall one of two ways: it leaks, or it breaks. Leaking is a disclosure incident; breaking means the JSON no longer parses.

Working with general-purpose tools, and where it stops

# Transfer compressed (pre-compressing beats on-the-fly compression on a thin link)
gzip -c app.log > app.log.gz
# then move app.log.gz with whatever transfer you normally use

# If you narrow, keep the filter in the filename
grep -E '2026-09-1[01]' app.log > app-20260910-11.log

# Masking can't be made reversible; at least keep the original and mask a copy
sed -E 's/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+/MASKED/g' app.log > app-masked.log

Three limits.

First, storing it compressed makes searching slow. 80 GB becomes 9 GB, but zgrep decompresses as it scans, every time (Part 8).

Second, splitting destroys the coordinate. Line 1 of a split fragment is line 1, and nothing tells you which line of the original it was. Part 16's "three things an excerpt breaks" applies here unchanged.

Third, and this is the real one: whether data may leave the building is a question about the path, not the size. Keeping a file local and reading it on your own machine is a different review from uploading it to a cloud log-analysis service. With a tool that works entirely locally, the question stops being "where do we send it" and becomes only "where do we put it."

3. Reading 30 GB across a WSL boundary is ten times slower

Situation

You investigate a 30 GB log that lives on the Windows side, using grep inside WSL.

Something that takes two minutes on a Linux server hasn't finished after twenty. The CPU is idle. The disk light is calm. You can't see where the waiting happens.

Why it happens

File access that crosses an OS boundary pays a round trip per access.

In WSL2, the Windows drives you see from Linux are the Windows filesystem mounted over a protocol (9P by default). Each read becomes a round trip to the host side, so the more small reads a workload issues, the worse it does. The reverse direction — Windows tools reading files on the WSL side — has the same shape.

What matters here isn't bytes transferred but number of accesses. Sequential scanners like grep read in larger chunks and fare relatively better; anything that reads back line by line, or bounces repeatedly around the tail of a file, slows down in proportion to how often it asks. WSL1 and WSL2 behave differently, and behaviour changes between versions.

Working with general-purpose tools, and where it stops

The most effective move is not crossing the boundary at all.

# WINLOG = path to the log on the Windows side, as WSL sees it

# Copy to the Linux-side native filesystem first, then work there
cp "$WINLOG" ~/work/app.log

# Or reduce what crosses the boundary (compress it first)
gzip -c "$WINLOG" > ~/work/app.log.gz

Mount options and metadata settings in WSL's configuration file (wsl.conf) leave some room for improvement, but the effect is configuration-dependent.

Two limits.

First, the copy doubles your disk usage and takes time itself. You pay those minutes for 30 GB on every investigation.

Second, reading on the Windows side avoids the boundary — but then you have no tool. Notepad goes quiet at a few gigabytes (Part 1), and neither grep nor less ships as standard. So you go back to WSL. That round trip is the problem. Which means this isn't a performance question but a placement question: which side of the boundary holds the tool.

4. "It opened fine on my machine"

Situation

A colleague you handed a 30 GB log to says it won't open. It opened on your desktop.

The difference is 64 GB of RAM versus 16. The result: log investigation collects around the one person with the memory. While they're on leave, the investigation stops.

Why it happens

If your tool assumes it will read everything into memory, your ceiling is your RAM. When the ceiling differs per machine, the team's capability doesn't settle at the weakest machine — it becomes dependent on the one person with the strongest.

There's a second effect: when tools differ, the vocabulary stops being shared. "Look at line 12,034,541" shows a different line if the other person's tool counts \r as a line break. BOMs, UTF-16, and line-ending conventions all shift the numbering too (Part 3, Part 11). Line numbers get used as a shared language, but they aren't one unless the tools agree.

Working with general-purpose tools, and where it stops

The standard answer is that everybody sshes to a shared box and reads with less. The log never leaves, and the environment gap disappears.

Three limits.

First, latency. Every repaint waits for a round trip, which is poor for the kind of investigation you do with your eyes. Remote work makes it heavier still.

Second, nothing persists. Colours, bookmarks, search terms — close less and they're gone. The unit of handover becomes a verbal explanation (the same problem behind the resumption point in Part 13).

Third, everyone queues on one machine. Two people scanning tens of gigabytes at once fight over that box's I/O.

So the conclusion: levelling the environment doesn't mean levelling the RAM — it means agreeing on a tool that doesn't depend on RAM. If a 16 GB laptop opens it, everyone sees the same line numbers, and the work stops pooling on one person.

What the four had in common

Friction	Where it stalls	Direct cause	General-purpose approach	What's left over
CI build log	The browser viewer	Interleaving, ANSI, progress bars, retries	Fetch the raw log, clean with `ansi2txt` / `tr`	Line numbers shift with each pass; still huge afterwards
Carrying production home	Transfer and review	Size and confidentiality at once	Compressed transfer, `grep` to narrow, masking	The filter is only knowable afterwards; masking is irreversible
Reading across WSL	The OS boundary	One round trip per access	`cp` to the native filesystem first	Double the disk, plus copy time; no tool on the Windows side
Team environment gap	The reader's machine	The ceiling is set by RAM	`ssh` to a shared box, read with `less`	Latency; nothing persists; everyone on one machine

All four push you into the same choice: transform the log before reading it, or travel to where the log lives. Transform it and the line numbers change. Travel to it and nothing you learn persists.

Which makes the requirements three.

How much you can read isn't set by RAM. Satisfy this and section 4's gap simply stops existing — and sections 1 and 2 no longer need "shrink it so it fits."
You can read the original as it is. Opened without splitting, masking, or cleanup, so section 2's irreversible decisions can be postponed.
It completes locally. Section 2's review question changes from "where do we send it" to "where do we keep it," and section 3 reduces to where the file sits.

Back to the 480,000 lines. That log was never too long to read. It was a log you had to transform before reading — which cost you the only coordinate it had.

The tool I use

UwView (free), which I develop, displays, scrolls, and searches huge text from the moment it opens. It doesn't load the whole file into memory, so files larger than RAM open fine. The index builds in the background, and line numbers appear when it finishes.

Of the three requirements above, the free version covers the first two outright.

How much you can read isn't set by RAM. A 16 GB laptop opens files of tens of gigabytes (47.73 GB / about 890 million lines measured on the free version; one specific setup). The precondition for section 4's "it opened on my machine" disappears.
It never writes to the original, and never splits or extracts. Section 1's cleanup intermediates aren't needed, so the line numbers still match the CI UI. Section 2's "just narrow it for now" can wait.
The same tool runs on Windows, macOS, and Linux. That makes section 3's "which side holds the tool" a choice. Read a Windows-side log with a Windows-side tool and no round trips cross the boundary.
All processing is local. The file is never transmitted anywhere. In section 2's review, that is sometimes the precondition rather than a feature.
Encoding switches while the file stays open (UTF-8 / Shift-JIS(CP932) / EUC-JP / UTF-16, auto-detected). No iconv intermediate (Part 3).
Highlighting colours lines rather than removing them. Section 1's interleaving can be untangled by eye, colouring per job name (Part 5).

The third requirement — and every case where you reopen the same file — is UwView Pro's territory.

It saves the index and the compression: from the second open onward the file comes back instantly, with line numbers (0.02–0.07 s measured on a 47.73 GB text file; one setup, results vary). The dump you carried home in section 2 gets reopened the day you write the report, the day review sends it back, and the day something similar recurs — the same tens of gigabytes, across days.
~1/9 storage, still searchable: aimed straight at section 2's "it won't fit on my SSD." Search goes through the compressed cache, so there's no zgrep-style decompress-as-you-scan on every query (Part 8).
Drill-down search: narrow a result by another term, then another, and the original line numbers survive to the last stage. Against section 2's "the filter is only knowable afterwards," this answers by narrowing the view instead of narrowing the file (implementation write-up).
±N is independent per stage: ±1 while narrowing, ±20 on the stage you read. Section 1's "discover -B 20 was too small, then run grep again" disappears (free version is fixed ±1; variable ±N is Pro).
Sequence search: match only where w1 → w2 → w3 appear in that order. In section 1's log, where a retry recorded the same failure three times, the ordering of the first occurrence becomes the query (implementation write-up). One honest note: each stage scans the body from the previous stage's position, so it takes about as long as a full-text search.
Search terms and colour rules can be saved and handed over: aimed at section 4's "close less and it's gone" (archive plus session restore).

The honest limits

UwView is a viewer. It is not CI log infrastructure and not a log-aggregation service.

It doesn't strip section 1's ANSI escapes or fold progress bars. If preprocessing is needed, sed and tr are your job.
It doesn't fetch logs from a CI API either. You still need the CI's own CLI or equivalent.
It doesn't mask anything. Handling personal data follows your organisation's policy.
Section 3's I/O penalty cannot be removed by a tool. It's solved by placement — which side of the boundary the file sits on, and which side's tool reads it.
No live tailing, no alerting, no automated cross-log correlation.
Text only. Binary dumps and core dumps are out of scope.

For completeness: non-destructive diff editing (Edit Upgrade) exists as a separate licence, but all four items above are read-only work. What you need here is the View side.

Links

Part 1: four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
Part 3: four character-encoding traps and how to isolate them: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
Part 5: four practices for living with development logs: https://uvp.y42u.net/en/blog/uwview-ps05-debug-log-practices-en/
Part 8: keeping logs compressed and still searchable: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
Part 11: reading legacy encodings in 2026: https://uvp.y42u.net/en/blog/uwview-ps11-legacy-encoding-euc-utf16-en/
Part 13: four techniques for inspecting huge data: https://uvp.y42u.net/en/blog/uwview-ps13-huge-data-inspection-en/
Part 15: four limits of command-line craft: https://uvp.y42u.net/en/blog/uwview-ps15-cli-craft-limits-en/
Part 16: four principles for preserving, excerpting, and proving integrity: https://uvp.y42u.net/en/blog/uwview-ps16-log-as-evidence-en/
Drill-down search — narrowing a result by another term: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
Sequence search — finding only what appears in that order: https://uvp.y42u.net/en/blog/uvp-sequence-search-en/
Archive plus session restore, as a working flow: https://uvp.y42u.net/en/blog/uwview-archive-session-restore-workflow-en/
Source code (GitHub): https://github.com/amru195704/UwView

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Moving production data, masking personal information, and retaining logs must follow your organisation's policy and applicable law. Line counts, sizes, and filenames are illustrative and do not describe any real case. The behaviour of sed, tr, grep, gzip, and cp varies by implementation (GNU/BSD/busybox), version, and build options, and CI service CLIs and retention periods change with the vendor's specifications. WSL file-access performance varies substantially with version (WSL1/WSL2), Windows build, mount settings, and whether antivirus is active, so the description here explains a tendency rather than guaranteeing any figure. Check option names and defaults against your own man pages and the official documentation. Measured figures come from one specific setup and are not a guarantee of the same result. If you spot an error, a comment is welcome and I'll check and correct it.

Logs as Evidence — Four Principles for Preserving, Excerpting, and Proving Integrity

y4u — Mon, 14 Sep 2026 02:56:13 +0000

The reboot cost thirty-eight minutes of log.

Fixing it was right. What should you have taken with you first?

Loss has a shape. There are four of them.

Earlier parts of this series were about how to read. This one is about what comes before that — keeping the thing you'll read in a form you can later show to someone else. Reboots, rotation, the excerpt you paste into a report, and the question of how you demonstrate that a file wasn't altered. None of it is glamorous. All of it, done wrong, removes the investigation itself from the table.

Up front: UwView never writes to the original. It displays and searches from the moment it opens, and it never splits or extracts, so a preserved log stays one file, unmodified. Pro's drill-down search stacks the narrowing and keeps the original line numbers all the way to the last stage, so you can cite a passage in the form a report needs — "line N of the original." Everything runs on your own machine; the file is never sent anywhere (measured on one setup; results vary — details at the end)

This article assumes the preserved log is already in your hands, or is about to be. Containment, notification, and any judgement about legal admissibility belong to your organisation's policy and the relevant authorities.

1. Before "a reboot fixed it" — sorting what dies from what survives

Situation

Two in the morning. The service stops responding. You don't know why, but a reboot brings it back.

The next morning someone asks what caused it. You can't say. You go looking for the logs, and the window you care about isn't there — or it's there with a few minutes missing from the middle.

Why it happens

A reboot is a destructive operation. Not uniformly, though — what dies and what survives divides cleanly.

Almost certainly gone:

The logging buffer inside the process. Most logging libraries don't write per line; they batch. On a hard kill (power loss, a panic, a forced stop signal), whatever hadn't reached the disk is gone. That's the last few seconds to few tens of seconds — which is to say, the part nearest the cause.
The kernel ring buffer. What dmesg reads gets replaced on reboot unless you're persisting it.
In-memory scratch space. Directories backed by tmpfs empty out. If journald is Storage=volatile — including the common case where auto finds no /var/log/journal — the journal also lives in the in-memory runtime area and disappears with the reboot.
Volatile state. Socket tables, the process tree, open file descriptors, files that were deleted but are still held open. Not logs, but frequently more eloquent than logs.

Surviving: whatever finished landing in /var/log/*, a Storage=persistent journal, already-rotated archives.

So what a reboot takes is precisely the side nearest the cause.

Working with general-purpose tools, and where it stops

There are things you can do in under a minute, before the reboot.

# Have one destination directory ready (e.g. preserve-20260910T031500Z)
D=preserve-20260910T031500Z

dmesg -T                       > "$D/dmesg.txt"
journalctl -b -o short-precise > "$D/journal-thisboot.txt"
ps auxww                       > "$D/ps.txt"
ss -tanp                       > "$D/ss.txt"
cp -a /var/log/app/current.log "$D/"          # copy — not move, not a slice

If your system has a command that lists open file descriptors, capture that output too. And if you have to stop the application, try the clean stop first — given a termination signal, most implementations flush what's left before they exit. A forced stop is the last resort, for when that doesn't work.

The point is that it's cp and not mv, and that nothing is being truncated with tail. At this stage, never shrink anything to a size you can read. You can redo the shrinking as many times as you like later. You cannot undo it once you've asked for "the part before that."

Three limits.

First, taking everything produces something you can't open. A journalctl -b that runs to tens of gigabytes is not unusual. Preserved, yes; readable, no — which walks straight back into Part 1.

Second, where it lands. On the same disk, you eat the free space of a box that's already in trouble. On another disk, you spend the minutes you least want to spend.

Third — and in practice this is the one that decides the outcome — whether you can remember the procedure at 2 a.m. A procedure that depends on the memory of someone whose hands are shaking is not a procedure. Which is why, as in Part 10, this belongs in a script: one preserve.sh, somewhere on your path.

2. The rotation gap — the moment evidence disappears is already on the schedule

Situation

The incident was at 03:24. The next day you go to read the log.

app.log exists, but it starts at 03:25. The 03:24 lines should be in app.log.1.gz. You open it and those minutes are simply not there — or, with rotate 8, the file that held them was deleted days ago.

Why it happens

Rotation swaps a file out, and every swap has a seam. The size of the seam depends on the method.

Two logrotate modes, treated separately:

create (the default). The old file is mv'd, a new one is created, and postrotate tells the application to reopen (typically by sending it a HUP signal). If the reopen fails, the application keeps writing to the moved inode. The directory shows an empty app.log while the writes pour into app.log.1 — a classic "logging just stopped" incident.
copytruncate. The contents are copied, then the original is truncated. Nothing has to be told anything, which is why people like it. But lines written between the copy and the truncate land in neither file. Copying several gigabytes takes seconds. At a few thousand lines per second, that's thousands to tens of thousands of lines.

And the deletion schedule is written down in the config. rotate 8 with daily means eight days; maxage 30 means thirty. Being told "we retain three years" while the machine says eight days is a gap that shows up in real audits — the Part 4 decision about delete, keep, or compress, arriving as a consequence.

Working with general-purpose tools, and where it stops

First, see what will happen before it happens.

logrotate -d app.conf                    # dry run: report, don't act
grep -rn 'rotate\|maxage\|copytruncate\|create' logrotate.d/

Read the config paths as whatever your distribution uses — usually a logrotate.d directory plus the top-level config file above it. When it last ran is recorded in logrotate's own state file (logrotate/status).

Then put the copy before the swap.

/var/log/app/*.log {
    daily
    rotate 8
    dateext
    dateformat -%Y%m%d
    delaycompress
    sharedscripts
    prerotate
        cp -a /var/log/app/app.log /archive/app-20260910T031500Z.log
    endscript
    postrotate
        # tell the app to reopen (a service reload, or a HUP signal)
    endscript
}

In practice the timestamp in the staged filename comes from expanding a date command; the example above hard-codes one.

dateext looks like a cosmetic setting and isn't. Under the numbered scheme, app.log.1 shifts to app.log.2 and the same name refers to different contents over time, so "which day is this app.log.3 from" becomes unanswerable. Put the date in the name and the name becomes a coordinate.

Three limits.

First, the prerotate copy lengthens the rotation itself. Copying 50 GB takes minutes, during which disk I/O saturates and production slows. Land that on top of the nightly batch window and your preservation has manufactured a second incident.

Second, you've doubled the space. The instinct is to compress, but searching compressed logs with zgrep is slow — Part 8 covers why: it's a sequential scan with decompression on the way through.

Third, and heaviest once you think of the file as evidence: the moment you cp, it is a different file. cp -a preserves mtime and atime, but the destination inode is new and its ctime is now. And if you offer mtime as proof that "this was written at 03:24," touch sets mtime to anything you like. File attributes prove nothing on their own. Which leads to section 4.

3. The "±10 lines" you paste into a report — excerpting without touching the original

Situation

You've found the line. "Put the relevant log in the report." Ten lines either side is enough.

You cut it with sed -n and paste. In review: "what was it doing just before that?" Back to sed. Then: "start twenty lines earlier." Back to sed. By the third round you've lost track of which fragment came from which line number.

Why it happens

Excerpting breaks three things at once.

The coordinate. Line 1 of the excerpt is line 1, not line 12,034,541.
The timing of the width decision. How much context you need is knowable only after reading. Excerpting demands the number first.
Identity. The moment you cut, it isn't the original. The hash won't match, and the fragment alone can't demonstrate that it came from the original at all.

The third one bites when the report leaves the building. "Lines 12,034,541–12,034,561 of the original" is a claim a reader without the original cannot check.

Working with general-purpose tools, and where it stops

You can keep the coordinate.

# Original line numbers survive, with context
grep -n -C 10 'OutOfMemoryError' huge.log > slice.txt

# When you already know the line
awk 'NR>=12034531 && NR<=12034551 {printf "%d: %s\n", NR, $0}' huge.log

# Record where you looked rather than what was there
grep -n 'OutOfMemoryError' huge.log | cut -d: -f1 > hit-lines.txt

The first is the practical winner: original line numbers, plus context.

The limits stay.

First, -C has to be chosen before you type it. You learn that ten wasn't enough only after reading, and -C 20 reads the file from the top again. Under the I/O-bound ceiling from Part 15, those re-reads accumulate one round trip at a time.

Second, lots of hits make it useless. A hundred hits at 21 lines each is 2,100 lines interleaved with -- separators. Not something you paste into a report. Cap it with -m 1 and you've lost the fact that there were others.

Third, prefixing the line number changes the text. 12034541: [ERROR] ... reads well, but that string doesn't exist in the original — searching for it finds nothing. You bought the coordinate by contaminating the line.

And underneath all three: a fragment becomes evidence only in reference to an original that still exists. What the report actually needs isn't the fragment, it's the correspondence — there is an original, and this is line N of it. Holding that correspondence together is the next section.

4. Demonstrating "this wasn't altered" — what a checksum covers

Situation

You hashed the preserved log.

sha256sum evidence.log > evidence.log.sha256

That felt like the end of it. Then an auditor asks when, and by whom, that hash was computed — and there's no answer in the file.

Why it happens

A checksum proves that two byte sequences are the same. That is the entire claim. It does not establish:

When it was taken. The mtime of the .sha256 is writable. So is any date you typed inside it.
Who took it. A hash has no author.
That the hash itself is authentic. If evidence.log and evidence.log.sha256 sit in the same directory, anyone who can rewrite the log can rewrite the hash. A hash stored beside the file is a lock with the key left in it.
That the contents are true. If the application logged a lie, the hash matches the lie perfectly.

So sha256sum on its own says exactly one thing: nothing has broken, in my hands, since the moment I ran this. Tamper resistance doesn't come from the strength of the hash function. It comes from where you put the hash.

Working with general-purpose tools, and where it stops

Two moves: separate the storage, and add time and authorship.

# 1) Take them together — a manifest, not one file at a time
find /archive -type f -name '*.log*' | sort | xargs sha256sum > manifest-20260910T031500Z.sha256

# 2) Verification is one line
sha256sum -c manifest-20260910T031500Z.sha256

# 3) For compressed logs, hash the *contents*
gzip -dc app.log.gz | sha256sum

Where filenames may contain spaces, use the null-terminated options of find and xargs. And to add the "who", attach a detached signature to the manifest (GnuPG, or whatever key management your organisation already runs). A hash has no author, so this is the field nothing else fills in.

The third is the one people miss. gzip writes the original filename and mtime into its header, so compressing identical content twice produces two different .gz hashes. Compression level and implementation (gzip / pigz / a given zlib) change it too. Record only the .gz hash in your ledger and, the first time anyone recompresses, you get a mismatch that means nothing and a scare that costs a day. Record the hash of the contents. (gzip -n drops the name and mtime and stabilises the .gz side, but hashing the contents is the safer habit.)

Separation can be bought at whatever price you can afford: ship it to another host, write it to append-only storage, commit it somewhere with real history, print it and seal it, or attach a third-party timestamp (RFC 3161). What they share is one property — the authority to rewrite the log and the authority to rewrite the hash are held by different people, in different places.

Two limits.

First, all of it requires ongoing operation. Doing it once and finding six months later that nobody kept it up is the most common ending.

Second, and this one is structural: hash late and you can say correspondingly little. A hash taken three days after the incident attests only to the three-days-later-onward state. Which puts the right moment for hashing in the same place as section 1's "before the reboot." Preservation and integrity aren't two jobs. They're two motions inside the same sixty seconds.

What all four had in common

Principle	When the evidence is lost	Direct cause	General-purpose response	What remains
Preserve	Reboot, hard kill	Unflushed buffers and volatile storage	Dump before rebooting; `cp`, never `mv`	Too big to read afterwards; nobody remembers the steps at 2 a.m.
Stage	The rotation seam	The `copytruncate` gap; the `rotate N` deadline	`logrotate -d` first; `cp` in `prerotate`	Double the I/O and space; `cp` changes the inode, so attributes prove nothing
Excerpt	The moment it's pasted into a report	Coordinate, width, and identity lost together	`grep -n -C`; `awk` with `NR` prefixed	Width chosen up front; a fragment alone can't be verified
Prove integrity	When the hash wasn't stored separately	A hash only asserts byte equality	Manifest + separated storage + signature	Taken too late; `.gz` hashes shift on recompression

The four look like four unrelated jobs, owned by different people at different hours. The reason the right-hand column rhymes is that all four lose something other than what they touched, at the instant they touch the original.

mv loses the location. truncate loses the seam. Excerpting loses the coordinate. Recompressing loses the hash. Pushed far enough, there's only one principle: handle the original as the original, and take what you need without changing it.

Three conditions follow.

Never write to it, split it, or cut pieces out of it. A tool that has to shrink a file before it can show it to you is the wrong tool here — the shrink triggers all three of section 3's losses at once.
Narrow while keeping the original coordinate. The number has to survive to the end, in the form a report needs: "line 12,034,541." Stacking a second filter must not renumber anything.
Assume you'll reopen the same file many times. A preserved log is not read once. It's read the day you write the report, the day review sends it back, and the day the auditor asks. The same tens of gigabytes, reopened across days.

Back to the thirty-eight minutes. The reboot didn't really take them. What took them was not having decided, beforehand, what the sixty seconds before a reboot are for.

The tool I use

UwView (free), which I develop, displays, scrolls, and searches huge text from the moment it opens. It doesn't load the whole file into memory, so files larger than RAM open fine. The index is built in the background and line numbers appear when it finishes.

Of the three conditions above, the free version covers the first outright.

It never writes to the original. Nothing is split, nothing is extracted, so the log you cp'd in section 1 stays one file, unmodified. Because "shrink it to read it" stops being necessary, section 3's three losses stop having an occasion to happen.
Highlighting colours lines rather than removing them. You sort by colour instead of excluding, so "the thing I needed was inside what I filtered out" doesn't arise.
All processing is local. The file is never transmitted anywhere. On preserved logs, where whether data may leave the building is itself the question, that's often a precondition rather than a feature.
Encoding switches while the file stays open (UTF-8 / Shift-JIS(CP932) / EUC-JP / UTF-16, auto-detected). No iconv in the pipe, so no second copy on disk (Part 3).

Conditions two and three are what UwView Pro adds.

Drill-down search: narrow a result by another term, then another. Tabs carry term (count), and the original line numbers survive to the last stage — so section 3's "the coordinate dies at stage two" doesn't happen, and the excerpt you paste can carry "line N of the original." A right-click history lists, in order, which term matched at which line (implementation write-up).
±N is independent per stage: ±1 while narrowing, ±20 on the stage you actually read — the width is changeable after the fact. Section 3's "discover that -C 10 was too small, then run grep again" disappears (free version is fixed ±1; variable ±N is Pro).
Sequence search: match only where w1 → w2 → w3 appear in that order. Section 1's "clean stop attempted → failed → hard kill" becomes the query itself (implementation write-up). One honest note: each stage scans the body from the previous stage's position, so it takes about as long as a full-text search.
It saves the index and the compression: from the second open onward the file opens instantly, with line numbers (0.02–0.07 s measured on a 47.73 GB text file; one setup, results vary). That lands directly on the third condition — a preserved log reopened across days.
~1/9 storage, still searchable: aimed straight at section 2's doubled disk usage. Search goes through the compressed cache, so there's no zgrep-style decompress-as-you-scan on every query (Part 8).

There's one feature that touches section 4. The sidecar Pro builds (.uwvz) carries an XxHash3 table per compressed block and verifies on every decompression. It also checks the offset table for sanity on open (monotonic, final offset equal to the real file length), so bit rot in storage or a copy that was cut short won't slip past you unnoticed.

This is not tamper detection, though. XxHash3 is a non-cryptographic hash chosen for speed; it isn't designed to resist deliberate rewriting. What section 4 needs is sha256sum and separated storage. This check exists to catch accidents — bit rot, a bad transfer, a truncated file. Don't conflate the two.

The honest limits

UwView is a viewer. It is not a SIEM and not a forensic suite.

It doesn't take section 1's dumps. Writing preserve.sh is your job.
It doesn't configure logrotate and doesn't run section 2's staging.
It doesn't compute hashes, doesn't sign, and doesn't record chain-of-custody (who opened what, when). It is not a place to keep an evidence ledger.
No automated cross-log correlation, no threat-intel matching, no alerting, no report generation.
Disk images and memory dumps are out of scope entirely.

One more constraint, stated plainly because it matters to people handling evidence: Pro's sidecar (.uwvz) is created as a new file next to the original. The original itself doesn't change by a single byte, but if your procedure is that nothing may be added to the preservation directory, copy the original into a separate working area before opening it. (The sidecar carries the original's length and last-modified time as its validation key, so if the original ever changes, the sidecar invalidates itself.)

What this tool covers is the step before all of that: reading a preserved raw log as it is, locally, with the original coordinates intact, using your own eyes. If your scale calls for correlation and alerting, that's a different product's job.

For completeness: non-destructive diff editing (Edit Upgrade) exists as a separate licence, but every one of the four principles above is read-only work. What you need here is the View side.

Links

Part 1: four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
Part 3: four character-encoding traps and how to isolate them: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
Part 4: deciding between delete, keep, and compress: https://uvp.y42u.net/en/blog/uwview-ps04-log-retention-decision-en/
Part 6: picking traces out of tens of millions of auth-log lines: https://uvp.y42u.net/en/blog/uwview-ps06-intrusion-triage-auth-logs-en/
Part 7: four ways to read a timestamp as a weapon: https://uvp.y42u.net/en/blog/uwview-ps07-timestamp-driven-triage-en/
Part 8: keeping logs compressed and still searchable: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
Part 10: four things to set up for the you of 2 a.m.: https://uvp.y42u.net/en/blog/uwview-ps10-oncall-night-preparation-en/
Part 13: four techniques for inspecting huge data: https://uvp.y42u.net/en/blog/uwview-ps13-huge-data-inspection-en/
Part 14: the traces are in the raw log: https://uvp.y42u.net/en/blog/uwview-ps14-attack-traces-raw-logs-en/
Part 15: four limits of command-line craft: https://uvp.y42u.net/en/blog/uwview-ps15-cli-craft-limits-en/
Drill-down search — narrowing a result by another term: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
Sequence search — finding only what appears in that order: https://uvp.y42u.net/en/blog/uvp-sequence-search-en/
Why search results moved into a separate window: https://uvp.y42u.net/en/blog/uwview-filter-popup-jump-save-context-en/
Archive plus session restore, as a working flow: https://uvp.y42u.net/en/blog/uwview-archive-session-restore-workflow-en/
Source code (GitHub): https://github.com/amru195704/UwView

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Actual incident response, evidence preservation, and audit work must follow your organisation's policy and applicable law and regulatory guidance. Nothing here is advice about legal admissibility. Times, line numbers, and filenames are illustrative and do not describe any real case. The behaviour of logrotate, journalctl, sha256sum, gzip, and grep varies by implementation (GNU/BSD/busybox), version, build options, and distribution defaults — check option names and defaults against your own man pages. Measured figures come from one specific setup and are not a guarantee of the same result. Disk type, filesystem, fragmentation, encryption, page-cache state, and concurrent load all change the outcome substantially. If you spot an error, a comment is welcome and I'll check and correct it.

"less Is Enough" — Until It Isn't: Four Limits of Command-Line Craft

y4u — Fri, 11 Sep 2026 09:56:39 +0000

You press G. Ninety seconds pass.

less was supposed to be the light tool. So what was being read during those ninety seconds?

The answer isn't about the quality of the tool. It's about one thing: where the line breaks are.

Every part of this series has been about where general-purpose tools stop. This one goes at the four that are trusted most — less, Vim, pipe craft, and grep. None of them is broken. They're all excellent. And past a certain size, all four hit the same wall. The goal here is to draw that wall as a line you can see.

Up front: UwView scrolls to the end of a huge file from the moment it opens; the index builds in the background and line numbers appear when it finishes. Pro saves the index and the compression, so a 47.73 GB log you've opened once reopens instantly, with line numbers, from the second open onward (0.02–0.07 s measured on one setup; results vary — details at the end)

Except where noted, every timing and transfer rate below is environment-dependent. The numbers are worked examples and measurements meant to show the shape of the problem, not a promise about your machine.

1. `less` was enough — until line jumps and Japanese search

Situation

less huge.log opens instantly. So far, so good.

It falls apart on the next keystroke. G to jump to the end: it stalls. -N for line numbers: it stalls harder. 12000000g to reach a specific line: stalls again. And in a Japanese log, typing /エラー returns nothing at all.

"less is enough" was a sound judgement. On a few megabytes, nothing is lighter.

Why it happens

less is fast because it doesn't read, and slow the moment you ask something it can't answer without reading.

less treats a file as a sequence of lines, but at open time it knows exactly one thing: the first screenful. Where the file's lines begin and end is determined by newline bytes, and you cannot know where those are without reading them. There is no way to answer "at which byte does line 12,000,000 start?" by seeking. So:

G (go to end) = read every byte to the end, counting newlines
-N (show line numbers) = count newlines up to the current position
12000000g (jump to a line) = find the 12,000,000th newline from the top

All three are the same operation: read everything, count newlines. On a 50 GB file, that's 50 GB read. The ninety seconds is what that costs. less isn't being inefficient.

The Japanese wall is a different problem. less does not transcode. It only decides whether bytes are displayable (LESSCHARSET configures that decision, not a conversion). If your terminal is UTF-8 and the file is CP932, the UTF-8 bytes you typed for /エラー simply do not appear in the file's CP932 bytes. The string isn't absent; it exists as a different byte sequence — the topic of Part 3 and Part 11, showing up here as a pager keystroke.

What general-purpose tools do, and where they stop

There are standard moves.

less -n huge.log            # stop computing line numbers → G gets light (no line numbers)
LESSCHARSET=utf-8 less huge.log
iconv -f CP932 -t UTF-8 huge.log | less    # convert, then pipe in
tail -c 200M huge.log | less               # just look at the tail

The first one works. It is also a trade: -n buys speed by giving up line numbers, so you cannot have both. On any job where the incident report needs "line 12,034,551," you end up back on -N.

The third is the one that quietly costs the most. The moment you write iconv | less, what less receives is standard input, which cannot be seeked. You can still scroll back — because less is writing what flowed past into a temp file. A second 50 GB copy grows in /tmp. And to move forward you still have to push every byte through the converter, so G becomes "read everything, converting as you go." The mapping to the original line numbers survives (converting doesn't change the line count), but displaying those numbers costs exactly what it cost before.

tail -c is a genuine escape hatch, but the moment you slice, it stops being the original: the first line may begin mid-character and render as garbage, and the line numbering restarts at 1.

Three limits. One, "see the end," "show line numbers," and "jump to a line" all demand a full read. Two, the result of that full read is discarded the instant you quit — open it again tomorrow and it's another ninety seconds. Three, there is no way to resolve an encoding mismatch without inserting a conversion, and inserting one costs you both seeking and disk.

2. `:e huge.log` freezes — what Vim can configure, and what it can't

Situation

You type vim huge.log, hit Enter, and nothing comes back. For a stretch, Ctrl-C doesn't even land. Eventually there's a swap-file warning, or a message about memory.

The reason people reach for Vim over less is usually: search, read the surrounding lines in place, and mark something. The goal is investigation, not editing.

Why it happens

A Vim buffer is designed to hold the entire file as an array of lines.

That isn't a flaw; it's what an editor has to be. Inserting, deleting, and replacing any line requires holding the sequence as a structure. So :e reads the whole file, and while reading, builds several other things:

A swapfile (.swp) — another copy on disk, for crash recovery
Undo history — and another copy on disk if undofile is on
Syntax highlighting state — deciding whether a line is inside a comment means scanning back through earlier lines; this is what makes very long lines and very large files collapse
Fold computation — foldmethod=syntax walks everything

So :e huge.log starts reading 50 GB, writing something like 50 GB back to disk, and running a parser over every line, all at once. Ninety seconds isn't in the running.

What general-purpose tools do, and where they stop

Vim can be told to do considerably less.

vim -u NONE -N huge.log     # no vimrc, no plugins

:syntax off
:set noswapfile
:set noundofile
:set nofoldenable
:set lazyredraw
:set synmaxcol=200          " give up highlighting past column 200
:set viminfo=

The LargeFile-style plugins essentially apply exactly this automatically. The effect is real and clearly noticeable.

One thing does not shrink: the number of bytes read.

Every setting above makes the work after reading cheaper; none of them touches the premise that :e reads all of it. Vim has no partial-load model — no way to keep most of the file on disk and page in only the region you're looking at. Which means opening a 48 GB file in Vim on a 16 GB machine is not a configuration problem. It's a structural one.

So in practice, people slice.

sed -n '12000000,12000500p' huge.log > slice.txt && vim slice.txt
awk 'NR>=12000000 && NR<=12000500 {print NR": "$0}' huge.log > slice.txt   # keeps line numbers

Three limits. One, you must choose the range before you can look at it. You find out 500 lines wasn't enough only after opening, and each retry sends sed back to read from the top of the file. Two, a plain slice loses the original line numbers — the second form keeps them via NR, but now the numbers are text inside the line, so you can't paste it straight into a report. Three, slice.txt files accumulate. By the end of an investigation there are twenty unrelated fragments in /tmp, and you can no longer say which range each one was.

None of this is an argument against Vim. If the job is editing, Vim is the answer. The problem is opening Vim when you only wanted to look, and paying the full cost of an editor for it.

3. Pipe craft vs. a viewer — where the line actually is

Situation

One line of tail, head, sed, awk, sort, and uniq handles most things. It genuinely has, for years.

It snags in one specific place: when you don't yet know what to look for. You have the error message. You don't have the condition that produces it. So you can't decide what to hand grep.

You head -100 and stare. You cut a column with awk. You look at a distribution with sort | uniq -c. Then back again. Ten round trips. Twenty.

Why it happens

A pipe demands that you decide the question first.

To write awk 'NR>=a && NR<=b' you must already know a and b. To write grep 'X' you must already know X. But the first half of an investigation is precisely the part where a, b, and X are decided by looking. You're asked for a question at the stage where you don't have one yet.

The structure of a pipeline has a few more consequences:

Intermediate state doesn't survive. After grep A | grep B | grep C, how many lines A matched and how many B matched is gone. You see the final count without knowing which condition did the work.
Line numbers die after stage one. grep -n counts within the block that reached it, not in the original file.
Every retry starts from the beginning. Change one character and the pipeline re-reads the file from the top.

The flip side is equally clear. A pipeline is unbeatable at repeating the same question. A one-liner goes into cron, into CI, and over ssh unchanged. When the log lives on a remote server you can't pull down, a pipeline is the only option (a constraint that connects directly to the "where do you keep it" question from Part 4).

What general-purpose tools do, and where they stop

The line, in one sentence:

If the shape of the answer is fixed, use a pipe. If you're still looking for the shape of the answer, use a viewer.

Concretely:

Nature of the work	Better fit
Cutting columns, computing, routine transformation	Pipe
Running it daily, repeatedly, automatically	Pipe
Staying inside a remote server	Pipe
Exploring what to look for	Viewer
Counting what's common inside a narrowed result	Either (see below)
Reading around a hit to make a judgement	Viewer
Narrowing while changing the condition ten times	Viewer
Citing a line with its position in the original	Viewer

In practice you use both, usually as "narrow with a pipe, then read in a viewer." There's a trap in that flow, though: narrowing already cost you one full read, and it's easy not to notice. And if you drop the narrowed result into another file, Part 9 applies — that file is no longer the original.

Two limits. One, full reads pile up in proportion to the number of exploratory round trips. Ninety seconds each, twenty trips, half an hour. Two, the history of those trips dies with the terminal. Tomorrow morning starts with remembering which conditions you already ruled out.

To be honest about it: this is not a case for replacing pipes with a viewer. Work that needs no interaction belongs in a pipe. What gets replaced is only the interactive first half.

4. `grep` isn't slow — you're I/O-bound

Situation

grep felt slow, so you switched to rg (ripgrep). Almost no difference.

Or the reverse: you ran the same command twice and the second run was suspiciously fast. Convinced you'd made a mistake, you measured again.

Why it happens

The bottleneck isn't the CPU. It's the disk.

Fixed-string search is very cheap on a modern CPU — Boyer-Moore-family algorithms with SIMD, running close to memory bandwidth. Which means the CPU is sitting there waiting for the disk to deliver bytes.

The floor is a division problem:

Medium	Rough sequential read	Just to read 50 GB (theoretical floor)
HDD	100–200 MB/s	~4–8 min
SATA SSD	~500 MB/s	~100 s
NVMe SSD	2–7 GB/s	~7–25 s

This table is calculated from typical sequential-read figures, not measured. Real numbers move with the filesystem, fragmentation, encryption, competing processes, and whether the file is on the far side of NFS/SMB.

Searching 50 GB on an HDD cannot beat four minutes, no matter how clever the search program is. grep and rg performing the same is exactly what you'd expect. The difference shows up only where the work is CPU-bound: complex regular expressions, large numbers of small files (rg parallelises across them), matching that involves Unicode normalisation.

The fast second run has the same explanation. Whatever the first run pulled in landed in the kernel's page cache, so the second read came from RAM, not disk — dramatically faster, up to the amount RAM can hold. Which is why a benchmark number means nothing unless it says whether the cache was warm. That's the reason every measurement in this series carries its conditions.

What general-purpose tools do, and where they stop

First, find out which side is the bottleneck.

time grep -c 'ERROR' huge.log
# real 4m12s / user 0m21s / sys 0m38s  → the CPU is idle = I/O-bound

# measure the sequential ceiling on this machine (bypassing the cache)
dd if=huge.log of=/dev/null bs=1M count=20000 iflag=direct

# watch I/O wait while it runs
iostat -x 1        # %util pinned at 100 means saturated
vmstat 1           # a large wa column says the same

If user + sys is tiny next to real, the CPU is waiting. Swapping the search program at that point is buying a faster car to sit in the same traffic jam.

Once you know you're I/O-bound, there are exactly three things you can do:

Put it on faster media (HDD → SSD — reliable, but you often don't get to choose where the file lives)
Read in parallel (works across many files; on one huge file it plateaus depending on the medium)
Read fewer bytes

The third is the real one, and it splits in two. An index — record where things are, so next time you don't read everything. And compression — if I/O is the bottleneck, reading a smaller compressed byte stream and expanding it on the CPU can be faster than reading the raw file, because the CPU is the resource you have spare. That arithmetic is why "search it while it stays compressed" in Part 8 makes sense.

One limit, and it's a heavy one. Neither grep nor rg carries anything forward. The structure of the 50 GB you just spent four minutes reading vanishes when the command exits. Change the condition and it's four minutes again. The page cache helps only as far as RAM reaches, and 50 GB doesn't fit.

What all four had in common

Tool	What breaks it	Direct cause	The workaround	What's left over
`less`	`G`, `-N`, line jumps, Japanese search	Line boundaries can't be known without reading	`-n`, or insert `iconv`	Speed and line numbers are mutually exclusive; converting costs seeking and disk
Vim	The moment you `:e`	The buffer assumes the whole file	`syntax off`, `noswapfile`, slicing	Bytes read doesn't drop; slicing destroys the original coordinates
Pipes	The moment you start exploring	Demands the question up front	`tee` the intermediate stages	Full read per round trip; per-stage counts and history don't survive
`grep`	The moment the file exceeds RAM	I/O-bound; the CPU is waiting	Faster media, parallelism	The scan isn't carried forward

Four tools, four different design philosophies, four different failure modes. The rightmost column is the same shape in all four rows because all four rest on the same premise: read the whole file, every time.

On a small file that premise is free. A 0.1-second full read is indistinguishable from no read at all, so the design is correct. It only breaks when that 0.1 second becomes ninety. Nothing about the operation got harder. It just stopped being worth it. This series keeps arriving at the same sentence, but this time the tools themselves are right — which makes the cause harder to see. You assume you're holding it wrong and go looking for a setting.

Three things are needed:

Don't throw away the line boundaries you already computed. The ninety seconds in less, the :e, the pipeline round trips — all of them are counting the same newlines over again. Build it once, save it, and the second time there's nothing to count.
Read fewer bytes. Under an I/O bottleneck, this is the only real speedup. Searching without decompressing means less to read.
Keep the original's coordinates while you narrow. A slice, an iconv, a sed -n — each one stops being the original the moment it passes through. On any job that ends in "line number, please," that's what bites last.

Back to the ninety seconds. That wasn't less being slow. It was newlines being counted — and the count was thrown away the moment you pressed q.

The tool I use

UwView (free), which I develop, displays, scrolls, and searches huge text from the moment it opens. It doesn't load the file into memory, so files larger than RAM open fine. The index builds in the background and line numbers appear when it completes (most viewers show only the head until indexing finishes). Nothing is split and nothing is extracted, so the original stays one file, unmodified. Section 1's G — moving to the end — works from the first second.

Switch encodings without reopening: UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16 are detected automatically, and when detection is wrong you switch in place without rebuilding the index. Section 1's "inserting iconv costs you seeking and disk" doesn't arise.
No editing engine: the swapfile, undo history, and syntax parsing that make Vim's :e expensive simply don't exist here. Section 2's list of settings is unnecessary. (For work that genuinely needs editing, Edit Upgrade is a separate licence.)
Context width is changed afterwards: you don't have to decide how many surrounding lines you need before opening. Section 2's "re-run sed after discovering 500 lines wasn't enough" disappears. (Free gives ±1 line; adjustable ±N is Pro.)
Search results open in a separate popup you can jump from into the original (write-up) — the replacement for section 3's return trip to the terminal.

And UwView Pro is where those three requirements live.

It saves the index and the compression: a file you've opened once reopens instantly, with line numbers, from the second open onward (0.02–0.07 s measured on a 47.73 GB text file; one setup, results vary). Section 1's "ninety seconds discarded on q" and section 4's "the scan isn't carried forward" are the same problem, solved by the same feature.
Search through the compressed cache: exactly section 4's arithmetic — under an I/O bottleneck, reading fewer bytes is the real speedup. Storage at roughly 1/9 while still searchable helps both the disk and the wait (keeping logs compressed and still searchable).
Drill-down search: narrow a result by another term, then another. Tabs carry term (count), which is section 3's "per-stage counts don't survive." Stages after the first search only the previous window, so there's no wait, and backing out is one click on an earlier tab. The original line numbers are preserved all the way down (write-up).
Resume tomorrow: tabs and conditions persist, so section 3's "close the terminal and the history goes with it" doesn't happen (archive × session restore workflow).

One row from section 3's table has crossed the line: frequency tallying. As of v1.5.0, drill-down search includes Tally — the equivalent of grep ... | grep -oE '...' | sort | uniq -c | sort -rn. Apply a regular expression to the hits, take the first capture group as the key, and get a ranked frequency table on top of whatever stage you've narrowed to, with the hit count, the number of distinct values, the elapsed time, and CSV export (uniq -c, in a GUI). It's one step past section 3's complaint that per-stage counts don't survive.

It stops at a frequency ranking, though — no sums, no averages, no charts. That's an analysis tool's job.

With that said, honestly: UwView is not a replacement for a shell. It won't cut arbitrary columns and compute on them, won't join files, and won't run on a schedule in cron or CI. When the work has to stay inside a remote server, a pipeline is the only answer. Section 3's line moved by exactly one row, and otherwise stands: this tool is mainly the right-hand column — exploring, context, narrowing, citing. Routine transformation and automation still belong to awk.

One more honest note. A complete first index takes real time. Section 4's I/O bottleneck applies to UwView the same as to anything else; reading 50 GB once doesn't become free. The difference is whether that result is saved and reused. If you only ever open the file once, less is enough — that judgement is still correct today. It stops being correct on the day you start reopening the same file.

Links

Part 1: four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
Part 3: four encoding traps and how to isolate them: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
Part 4: deciding between delete, keep, and compress: https://uvp.y42u.net/en/blog/uwview-ps04-log-retention-decision-en/
Part 8: keeping logs compressed and still searchable: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
Part 9: reading structured data raw: https://uvp.y42u.net/en/blog/uwview-ps09-read-raw-structured-data-en/
Part 10: four things to set up for the you at 2 a.m.: https://uvp.y42u.net/en/blog/uwview-ps10-oncall-night-preparation-en/
Part 11: reading legacy encodings in 2026: https://uvp.y42u.net/en/blog/uwview-ps11-legacy-encoding-euc-utf16-en/
Part 14: the traces are in the raw log: https://uvp.y42u.net/en/blog/uwview-ps14-attack-traces-raw-logs-en/
3GB/10GB/48GB × HDD/SSD/internal SSD — a nine-cell benchmark: https://uvp.y42u.net/en/blog/uwview-pro-benchmark-3sizes-en/
An honest comparison with klogg (Ver1.1.1, complete edition): https://uvp.y42u.net/en/blog/uwview-klogg-feature-comparison-v111-en/
Drill-down search — narrowing a result by another term: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
uniq -c joins the GUI (Tally in drill-down search): https://uvp.y42u.net/en/blog/uvp-tally-uniq-c-drilldown-en/
Why search results open in a separate window: https://uvp.y42u.net/en/blog/uwview-filter-popup-jump-save-context-en/
Archive × session restore workflow: https://uvp.y42u.net/en/blog/uwview-archive-session-restore-workflow-en/
Source code (GitHub): https://github.com/amru195704/UwView

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. The behaviour of less, Vim, grep, and awk varies by implementation (GNU/BSD/busybox), version, build options, and locale settings — always confirm option names and effects against your own man pages. Except where explicitly marked as measured, the transfer rates and durations here are calculated from typical performance figures, not observed. Figures marked as measured come from one specific setup and are not a guarantee of the same result. Disk type, filesystem, fragmentation, encryption, page-cache state, and competing processes all move the numbers substantially. If you spot an error, a comment is welcome and I'll check and correct it.

The Traces Are in the Raw Log — Four Views on Lateral Movement, SQLi, and After-the-Fact Investigation

y4u — Fri, 11 Sep 2026 09:54:55 +0000

The encryption took eleven minutes.

What matters is the six days before it. Which log do you read first, and in what order?

The trace is never in one line. It's in the arrangement.

Part 6 covered what to look at in the hours right after a suspicion is raised. This one is about what comes after: showing, in a defensible way, what already happened. Tracing lateral movement, investigating a ransomware event after the fact, finding the SQLi that got past the WAF, and inventorying personal data. Different goals, different people — and all four operating under the same constraints: don't touch the original, don't let the data leave, and be able to cite your evidence.

Up front: UwView Pro's sequence search matches only where `w1 → w2 → w3` appear in that order, so an attack pattern like `auth success → privilege escalation → connection to another host` becomes the query itself. Drill-down search stacks the narrowing and shows the count at each stage, so you can change a condition and see what it did. Everything runs on your own machine — the file is never sent anywhere (measurements are from one setup and vary; details at the end)

This article assumes you already have preserved logs in hand and are deciding how to read them. Containment, preservation, notification, and reporting procedures themselves belong to your organisation's policy and the relevant authorities.

1. Lateral movement — the identifier changes hands mid-trail

Situation

You know patient zero. The question from above is always the same: "How far did it spread?"

On disk you have the VPN gateway log, the authentication server log, the jump host's sshd log, and the file server's audit log. Four products, four formats. One attacker's behaviour, split across four files.

Why it happens

Lateral movement isn't an anomaly inside one machine. It exists only in the space between logs.

Worse, the identifier you're chasing changes hands. A VPN session ID gets you in; an authentication ticket is issued behind it; that ticket opens an SMB session to a share. Three different strings, and there is no column that joins them. What joins them is a fact: at the same moment, from the same source, the next identifier came up.

So the test isn't a term — it's an order. VPN success → ticket issued → connection to another host → share enumeration. In that order, it's lateral movement. Out of order — share enumeration before the ticket was issued — it's someone doing their job. The clock skew between servers that part 7 covered can flip that conclusion outright.

What general tools do, and where they stop

You collect across files, then order by time.

grep -h -F 'S-1a9f3c' vpn.log auth.log sshd.log fileserver.log   # the ID you know
# merge the window across files into one timeline (field positions vary)
sort -m -k1,2 <(grep '2026-08-2[0-6]' vpn.log) <(grep '2026-08-2[0-6]' auth.log) | less
# harvest identifiers that came up inside that window
awk '$1 >= "2026-08-24T02:10" && $1 <= "2026-08-24T02:40"' auth.log | grep -o 'TGT=[A-Za-z0-9]*' | sort -u

The third command is the real work. Search the known ID → read the window → harvest a new ID → search that. One round trip per hop in the chain.

Three limits. First, every round trip re-reads the originals from the top. Five hops means five passes, and the hypotheses that don't pan out add more. At tens of seconds each that's tolerable in isolation; an after-the-fact scope determination isn't finished until the chain is exhausted.

Second, you can't watch four files at once. sort -m gives you something readable, but the moment you merge, you lose which line came from which file. What you actually wanted was four files open, moving through the same window in all of them.

Third, grep cannot express order. grep -E 'VPN|TGT|SMB' returns lines containing any of the three; it never looks at the arrangement. You can write a state machine in awk, but on a trail where the identifier keeps changing, that means a throwaway script for every hunch — which is not the pace this work runs at.

2. Ransomware, after the fact — the interesting part is just before

Situation

The encryption is done. Restore is running from backup. In parallel, a second assignment lands: "When did they get in? What did they take?"

Production is isolated. What you have is a preserved copy of the file access and audit logs. Tens of gigabytes, most of it still .gz.

Why it happens

The encryption phase is short, and the records that matter are in the quiet days before it.

Encryption produces an enormous volume of writes, so on a timeline it's a dense black band. Only a small slice of the incident is blacked out. What you want to read is before that — the exploration, the bulk reads, the archive being assembled, the outbound transfer. That part looks almost exactly like a normal week.

And after-the-fact work carries two constraints that outage triage doesn't. One is preservation: no operation that writes to or rewrites the original. The other is where the data may live. Uploading logs from a compromised system to an external analysis service is often off the table for contractual or policy reasons. Running entirely on the local machine is frequently a requirement, not a preference.

What general tools do, and where they stop

Search without unpacking to disk.

zgrep -h 'READ\|COPY' audit-2026-08-*.log.gz | head -100
# narrow to the 24 hours before encryption began
zcat audit-2026-08-24.log.gz | awk '$1 >= "2026-08-24T00:00" && $1 <= "2026-08-25T03:12"' | less
zcat audit-*.gz | awk '{print $5}' | sort | uniq -c | sort -rn | head -20   # who touched the most

zgrep is the right instinct: nothing is written to disk, which is also the correct posture for preservation.

Three limits. First, every change of mind decompresses again. zgrep runs the decompression per search, and after-the-fact investigation is nothing but changing your mind. The cost multiplies by the number of hypotheses (part 8 went into this trade in detail).

Second, you can't size "just before" in advance. Whether 24 hours is enough or you need six days is something you learn by looking. awk's range wants the answer before you open, and every revision is another full pass.

Third, the test is, again, an order. bulk read → archive created → outbound connection in that sequence is exfiltration. The same three terms in a scrambled order are usually a backup job.

3. The SQLi that got past — what a WAF log doesn't hold

Situation

You have a WAF, so you have a record of what it blocked. What you actually need to examine is the requests it didn't block — and those exist only in the raw access log of the web server.

Why it happens

The attacker's whole job is to not match your pattern.

URL encoding, double encoding, comments splitting keywords apart (UNION/**/SELECT), mixed case, injected control characters. The same payload has dozens of appearances in the log. So the regex is never written once — it's written and revised.

And the moment you write it, the opposite problem arrives: false positives. A site search with or in the query string. An article title containing union. select is an ordinary English word. Most hits are harmless, and confirming that they're harmless means reading each one in context.

There's also a distinction that matters legally: an attempt is not a success. Whether it worked shows in another field on the same line (status code, response size) and in the lines that follow — an unnaturally regular burst of requests from the same address.

What general tools do, and where they stop

Cast wide, then narrow.

grep -Ei "union[[:space:]/*]+select|'[[:space:]]*or[[:space:]]*'1'?=|sleep\(|benchmark\(" access.log | head -50
# decode first, twice, to catch double encoding
perl -pe 's/%([0-9A-Fa-f]{2})/chr(hex($1))/ge' access.log | grep -Ei 'union.*select' | head
# did that address get a 200 back?
grep '203.0.113.77' access.log | awk '{print $9, $10}' | sort | uniq -c

The decode step works. Multiple encoding layers give way to simply running the substitution more than once.

Three limits. First, once decoded, it isn't the original any more. What flows down the pipe is transformed text with no line numbers attached to it. When it's time to quote a specific request in a report, you go looking for where it actually was — the same problem part 9 described as "the moment you transform it, it stops being the original."

Second, tuning a regex means re-reading everything, repeatedly. Cutting false positives is ten or twenty rounds of adding and removing conditions, each one a full pass over gigabytes. And the count at each stage isn't kept, so you can't tell which condition did the work. A visible drop — 94,979 → 184 → 54 — answers that instantly, but a pipeline throws its intermediate state away.

Third, "did it succeed" can't be written as one grep. The status code is on the same line; the follow-on behaviour is on other lines. No single pipeline sees both, so you end up shuttling between the hit list and the scene.

4. Inventorying personal data — one person is not one key

Situation

An erasure request arrives. Or there's a suspected breach and you need to identify who was affected. The ask is: "produce every trace of this person." There's a deadline.

The logs span years, tens of gigabytes, several systems.

Why it happens

In a log, one human being appears as several unrelated strings.

A member ID, an email address, the email address they used before they changed it, a phone number, a session ID, whatever IP they happened to hold that day. Search any one of them and the lines written with the others stay hidden. And the awkward part: the keys grow as you work. A line found by email contains a session ID you didn't know about; searching that turns up another identifier.

The nature of the job matches the full inspection in part 13: to say "nothing," you have to go through all of it. Answering "we found no records" requires being able to state which range you checked under which conditions.

And one more constraint: the subject matter is personal data. Uploading it somewhere to search it is largely off the table.

What general tools do, and where they stop

Build a key list and apply it in one pass.

grep -h -F -f keys.txt access.log auth.log app.log | tee hits.txt | wc -l
# harvest identifiers you didn't know about yet
grep -o -E 'sid=[A-Za-z0-9]{16}|[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+' hits.txt | sort -u
# keep filename and line number, for the report
grep -n -H -F -f keys.txt *.log > evidence.txt

grep -F -f is the workhorse here: dozens of keys, matched as fixed strings, in one pass. And -n -H in the third command is basic hygiene for anyone who has to write the report.

Two limits. First, every new key restarts everything. One identifier discovered means appending to keys.txt and running the whole corpus again — and identifier chains tend to branch rather than terminate.

Second, what you looked at doesn't persist. Shell history dies with the terminal. Assembling "which conditions, which range" the day before the deadline, from memory, is how people end up redoing the whole thing. On this job the search conditions are part of the deliverable, and general-purpose tools have nowhere to keep them.

What all four had in common

View	What you're looking for	Constraint	Where general tools stop
Lateral movement	A chain of identifiers that changes hands	Follow four files at once	Every round trip re-reads; order can't be expressed
Ransomware, after the fact	The window just before encryption	Preservation; stay local	Re-decompresses per query; the window can't be sized up front
SQLi that got past	Attempts, and whether any succeeded	Citations need original coordinates	Decoding drops the coordinates; per-stage counts vanish
Personal data inventory	Several keys pointing at one person	Can't leave the machine; deadline	Every new key restarts; conditions aren't kept

The second column has nothing in common. Neither does the third. The fourth lines up anyway, because all four are asking for the same three operations.

Search by order. VPN success → ticket → another host; bulk read → archive → outbound. The trace of an attack is in the arrangement of terms, not the set of them.
Stack the narrowing and see the count at each stage. Neither regex tuning nor key expansion finishes in one pass. When the drop is visible, the condition that did the work is obvious.
Keep the original's coordinates, and stay on the local machine. A citation needs a filename and a line number; the data can't leave. Work on the original, not on a transformed copy.

All three are things anyone does without thinking on a small file. At tens of gigabytes, each round trip demands a full re-read, decoding erases the coordinates, and the conditions die with the terminal. The operations aren't hard; size is what stops them being worth their cost. This series keeps arriving at that sentence — after the fact, it just comes with a deadline and a higher price for being wrong attached.

Back to those eleven minutes. Any tool will find them. What no tool hands you is the six days before.

The tool I use

UwView (free), which I develop, displays, scrolls, and searches huge text from the moment it opens. The index is built in the background and line numbers appear when it completes. Nothing is split and nothing is extracted, so a preserved log stays one file, unmodified. It never writes to the original — the first principle of forensics. Highlights add colour without removing lines, which is exactly section 3's "set the harmless hits aside without deleting them." All processing happens on your machine; the file is never transmitted anywhere.

The three operations above are what UwView Pro adds.

Sequence search: match only where w1 → w2 → w3 appear in that order. Section 1's VPN success → ticket issued → another host and section 2's bulk read → archive → outbound become the query itself. A right-click "history" shows the path that actually matched — which line held which term — and you can jump to any of them. One honest note: each stage searches the body starting from the previous stage's position, so it takes about as long as a full-text search. Implementation write-up.
Drill-down search: narrow a result by another term, then another. Tabs show term (count), so section 3's regex tuning and section 4's key expansion run with the count visible at every stage. Stages after the first search only the previous stage's window, so there's no wait, and backing out means clicking the earlier tab. Implementation write-up.

In practice both look like this. The tabs across the top read Tokyo (10,967) → Ariake (7) → 有明 (4) — with counts — which is section 3's "how much did each stage remove." The right-click History (drill-down path) reads Line 320,574,475: Tokyo → Line 320,574,475: Ariake — term and original line number, in order — which is section 1's "which term caught this, and where did it come from." (That example is 892 million lines of OSM data; logs behave the same way. Screenshots are in the original article.)

Set against a grep pipe, what survives is different.

| | grep A file \| grep B \| grep C | Drill-down + history |
|---|---|---|
| Per-stage counts | Gone — only the final result | Tabs carry term (count) |
| Original line numbers | Lost from stage 2 on (grep -n counts within the block that reached it) | Still 320,574,475 — and you can jump there |
| Which term caught the row | Unknowable | Listed in History, term and line number, in order |
| Changing a condition | Rewrite the pipe, re-read the file from the top | Click the tab of the stage you want back |
| Stopping for the day | Close the terminal and the conditions and the path go with it | Tabs and history persist; resume there tomorrow |

Section 4's "what you looked at doesn't persist" is rows 1 and 5 of that table. On work where the search conditions are part of the deliverable — an erasure-request answer, an after-the-fact report — that difference lands directly on the number of days and on how hard the report is to write.

±N is independent per stage: ±1 while narrowing, ±10 on the stage where you actually read — and changing N doesn't re-read the original. That's section 2's "you can't size the window up front."
It saves the index and the compression: from the second open onward the file opens instantly, with line numbers (0.02–0.07 s measured on a 47.73 GB text file; one setup, results vary). Compressed-cache search and ~1/9 storage come with it, so retention-mandated logs cost less disk (all OS, one-time or monthly). On section 2's kind of work — the same preserved tens of gigabytes, reopened for days — that difference shows up in the schedule.

One honest set of limits: UwView is a viewer. It is not a SIEM and not a forensic suite. No automated cross-log correlation, no threat-intel matching, no alerting, no report generation. It will not write section 3's regex for you, it does not normalise the timestamps in section 1, and it does not join files automatically — putting them side by side is still your eyes doing the work. Disk images and memory dumps are out of scope entirely. What it covers is the step before all of that: looking at a preserved raw log, as it is, locally, with your own eyes. If your scale calls for correlation and alerting, that's a different product's job.

For completeness: non-destructive diff editing (Edit Upgrade) exists as a separate licence, but every one of the four views above is read-only work — what you need here is the View side.

Links

Part 1: four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
Part 2: four ways to trace causality in logs: https://uvp.y42u.net/en/blog/uwview-ps02-log-causality-tracing-en/
Part 4: deciding between delete, keep, and compress: https://uvp.y42u.net/en/blog/uwview-ps04-log-retention-decision-en/
Part 6: picking traces out of tens of millions of auth-log lines: https://uvp.y42u.net/en/blog/uwview-ps06-intrusion-triage-auth-logs-en/
Part 7: four ways to read a timestamp as a weapon: https://uvp.y42u.net/en/blog/uwview-ps07-timestamp-driven-triage-en/
Part 8: keeping logs compressed and still searchable: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
Part 9: reading structured data raw: https://uvp.y42u.net/en/blog/uwview-ps09-read-raw-structured-data-en/
Part 13: four techniques for inspecting huge data: https://uvp.y42u.net/en/blog/uwview-ps13-huge-data-inspection-en/
Sequence search — finding only what appears in that order: https://uvp.y42u.net/en/blog/uvp-sequence-search-en/
Drill-down search — narrowing a result by another term: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
Chasing 5xx in an nginx access log: https://uvp.y42u.net/en/blog/uwview-access-log-5xx-workflow-en/
Source code (GitHub): https://github.com/amru195704/UwView

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Actual incident response, evidence preservation, and personal-data handling must follow your organisation's policy and applicable law and regulatory guidance. Times, counts, and identifiers shown here are illustrative and do not describe any real incident. All IP addresses are documentation-range examples. Log formats, field positions, and timestamp conventions vary widely by OS, middleware, product, and configuration. Command examples may need adjustment for your environment (GNU/BSD differences, shell, awk / grep / zgrep implementations). Measured figures come from one specific setup and are not a guarantee of the same result. If you spot an error, a comment is welcome and I'll check and correct it.

DEV Community: y4u

We Lose to klogg on Opening. uvf Wins the Whole Run — Re-measured at 50 GB

First — what uvf … -open is

The loss, first

And yet the whole run turns over

258 GB behaves the same way

Why one pass is enough

The 3 GB case narrows

The debt we owe

The side that keeps an index

Which to use

How it was measured (you can repeat it)

ripgrep Is 2.9x Faster With no-mmap — Measured Against the Default mmap Read on One 51 GB File, on Three OSes

The numbers first

Why we ended up measuring this

How it was measured

It was neither the disk nor the CPU

Why (this part is a guess)

What to actually do

About our own tool, briefly

Trying it yourself

I Thought We Lost at 3 GB — Turns Out We'd Won (As Long as the Cache Is Cold)

Up front

First, drop the cache and measure again

At 10 GB there is no warm state at all

① A day you fix and save — when the job writes output, bytes read matter directly

② A day you use something rg does not have

③ A day the logs are kept as gz / zip — stop expanding them every time

④ A day you do not yet know what you are looking for — this is the real point

Summary — four situations where you win at 3 GB

The tools

Conditions

Links

Picking the One Place to Beat ripgrep — Read 1/9 of the Bytes, Then Hand the Result From the CLI to the GUI

Up front

The premise: fight fair and you lose

The one move left: read fewer bytes

The table of losses

Which question pays for the index

The by-product: the CLI and the GUI became one surface

What was done, in short

The tools

Conditions and commands

Links

The Morning the Monthly Batch Died — Four Realities of Running a Data Pipeline

1. All that's left is a line number

Situation

Why it happens

General tools, and where they stop

2. The odd rows in a 50 GB corpus

Situation

Why it happens

General tools, and where they stop

3. Reopening the same 31 GB every month

Situation

Why it happens

General tools, and where they stop

4. Look at the original before you transform it

Situation

Why it happens

General tools, and where they stop

What the four had in common

The tool I use

The same things, from the command line

Links

"Please Produce the Logs from Three Years Ago" — Four Ways to Survive an Audit by Design

1. The retention period is met. The retrieval was never designed.

Situation

Why it happens

Working with general-purpose tools, and where it stops

2. Look at the real thing before you write the ingest rule

Situation

Why it happens

Working with general-purpose tools, and where it stops

3. Compression ratio and retrievability draw on the same budget

Situation

Why it happens

Working with general-purpose tools, and where it stops

4. Will it still be readable in seven years?

Situation

First — what `uvf … -open` is