We Lose to klogg on Opening. uvf Wins the Whole Run — Re-measured at 50 GB

If all you do is open a 51 GB file, klogg is faster. 52.55 s against 100.6 s. That one we lose.

Run it through to the search, though, and it turns over: klogg takes 108.14 s, uvf … -open takes 53.69 s.
About 2×. The reason is not speed. It is how many times the file gets read.

First — what uvf … -open is

uvf … -open runs through this whole article, so here is what it means.

UwView has two faces: a window (GUI) and a command line (uvf). Same tool — the window is for
opening and reading a huge file, the command is for finding things in it.
-open is the option that joins the two.

uvf <file> <pattern> -open

Type that and uvf reads the file once, searches it, hands the byte offsets of the matching lines
straight to the window, and the UwView window comes up. The window does not search again.
A line-numbered list of hits is there from the first frame, and clicking any of them puts you at that
spot in the file.

In other words, it removes the round trip where you grep to find roughly where something is, then
open the same file again in a separate viewer and search all over again.
Every uvf … -open figure in this article is that one command, timed from pressing enter to the
list of hits being on screen.

• uvf is free (it ships with the free edition of UwView) and runs standalone on Windows, macOS and Linux
• -open has existed since v1.6.0, but v1.6.4 is where it became a single pass and v1.6.5 is where the wait for the index disappeared. Everything measured here is v1.6.5

The loss, first

The same data at 3 GB, 10 GB and 51.25 GB (OpenStreetMap Japan expanded to XML, plus splits of it),
just opening it, and just searching for one term.

Opening

	3 GB	10 GB	50 GB
klogg	3.65 s	10.98 s	52.55 s
UwView (free)	5.27 s	19.62 s	100.6 s

Searching (東京)

	3 GB	10 GB	50 GB
klogg	0.56 s	11.75 s	55.59 s
UwView (free)	0.20 s	16.72 s	88.9 s

Hit counts were 11,274 / 11,393 / 94,979 and matched between the two at all three sizes.
From 10 GB up, klogg wins both opening and searching.

Convert the seconds to read throughput and it is obvious what is happening.

	3 GB	10 GB	50 GB
klogg, opening	792 MB/s	891 MB/s	930 MB/s
klogg, searching	(cached)	832 MB/s	879 MB/s
UwView free, opening	549 MB/s	498 MB/s	486 MB/s
UwView free, searching	(cached)	585 MB/s	550 MB/s

This drive reads at about 950 MB/s raw. klogg saturates it. Our free GUI runs at half.
That is not a bug — it matches the figures we have published all along (473–491 MB/s to open,
575–581 MB/s to search). It is how the thing is built, and it is a debt we owe. More on that below.

And yet the whole run turns over

What you actually want is not "open it". It is "find it and read it." So we timed that.

Time to find it and read it	3 GB	10 GB	50 GB
UwView free GUI (open + search)	5.47 s	36.34 s	189.50 s
klogg (open + search)	4.21 s	22.73 s	108.14 s
uvf … -open (v1.6.5)	3.28 s	10.41 s	53.69 s
ratio (vs klogg / vs free GUI)	1.28 / 1.67	2.18 / 3.49	2.01 / 3.53

klogg reads the file once to open it and again to search it. The UwView free GUI also reads it twice,
and at half the speed — which is how you get to 3.5×.

uvf file 'pattern' -open reads it once.

$ time uvf osm/japan-latest.osm 東京 -open
11.32s user  7.31s system  34% cpu  53.690 total

51.25 GB ÷ 53.69 s = 910 MB/s — essentially this drive's raw read speed (≈ 950 MB/s).
CPU sits at 34%; the rest is waiting on the disk, so the medium is the entire bottleneck.
Inside those 53.69 seconds it searches, builds the index, hands everything to the window and puts a
line-numbered list of hits on screen. The window itself then appears in under a second.

At 10 GB it is 10.405 s (940 MB/s); at 3 GB, 3.279 s (882 MB/s). At every size, it costs one pass over the file.

For reference, the uvf command measured on the search alone runs at 871 / 933 / 962 MB/s.
-open at 882 / 940 / 910 sits in the same band, dropping about 5% only at 50 GB — that is the window starting up.

258 GB behaves the same way

We checked the largest file too — the whole United States from OpenStreetMap expanded to XML,
258.68 GB and 4.5 billion lines, in one unsplit file.

$ sudo purge
$ time uvf osmUs/us-260726.osm 'New York' -open
56.24s user  40.60s system  36% cpu  4:25.21 total

265.21 s (4 min 25.2 s), 930 MB/s — the medium's speed again.

The first run gave 250.68 s (984 MB/s). That was above the drive's ~950 MB/s raw read, so we put it down to
cache left by a run we had stopped partway, and re-took it after sudo purge. The difference is 5.5%.
We are publishing the slower number — the one without cache.

Now line that up against how long each of them takes just to finish opening the file.

258.68 GB	Time	Throughput
klogg, to finish opening	258 s	956 MB/s
uvf … -open (find it, all the way to the screen)	265.21 s	930 MB/s
UwView Pro, to finish opening (building the .uwvz)	328 s	752 MB/s
UwView free GUI, to finish opening	530.1 s	465 MB/s

In about the time klogg needs merely to finish opening the file, we have already searched it and put the hits
on screen (258 s against 265.21 s — 7 seconds, 2.8%, apart). klogg starts searching from there.

Doing the same thing in the free GUI costs 8 min 50.1 s to open plus 7 min 20.0 s to search — 16 min 10 s, or 3.66×.

(Both free-GUI figures were re-taken the same day and came out the same — the earlier run gave 532.6 s / 449.4 s, 0.5% and 2.1% apart. That is 465 / 561 MB/s, the same band as 486 / 550 at 50 GB.)

And of course, if you keep coming back to the same file, that is Pro. Once the .uwvz exists, a search takes
34.8 s and opening it again is instant. One pass, or many — that is the dividing line.

Why one pass is enough

When uvf first shipped in v1.6.0, -open passed only the pattern to the window, and the window
searched again on its own. The CLI read everything, then the window read everything.

• v1.6.4 — the CLI hands over the byte offsets of the lines it found. The window stops re-searching
• v1.6.5 — it also collects the index markers while searching and hands those over too. The window assembles the index instead of re-reading the file, and since the line numbers travel with it, the results list has line numbers from the first frame

It is reading the whole file anyway, so counting one more thing costs almost nothing. And indeed the
timings above are near-identical to the search-only figures for uvf (3.32 s / 10.48 s / 50.82 s).
The handoff adds essentially zero.

The 3 GB case narrows

Look at the ratios again: 1.28× against klogg at 3 GB, against 2.01× at 50 GB.

The reason is simple — 3 GB fits in memory (32 GB here). klogg and the free GUI still read it twice,
but the second read comes from cache, so reading twice costs almost nothing (klogg's 0.56 s search at
3 GB, the free GUI's 0.20 s).

From 10 GB up it does not fit, so the second read really does hit the disk, and the gap opens.
This is the same thing we have been saying for months: 3 GB is a draw; the difference starts at 10 GB.

The free GUI's 0.20 s at 3 GB is near the limit of what a person can time by hand. Even if it were
really 1.0 s, the total is 6.27 s — 1.9× — and the conclusion does not move.

The debt we owe

This measurement made our own weak spot plain.

The free GUI opens at 486 MB/s. klogg opens at 930 MB/s. Very nearly a factor of two.
The uvf command was rebuilt as a single pass in v1.6.3 and reaches 962 MB/s, so we already have proof
that this machine and this disk can do it. If the GUI's index build is brought to the same way of reading,
opening 50 GB should go from 100.6 s to somewhere near 52 s.

klogg saturating the medium is, for us, a measured target. We will close it in a coming release.

The side that keeps an index

Everything above is the free edition. The paid uvp converts the file once into a .uwvz
(about one ninth of the original, with a line index) and never touches the original again.

Search	klogg	uvp (with .uwvz)	Difference
10 GB	11.75 s	1.80 s	6.5×
50 GB	55.59 s	6.34 s	8.8×

Opening it again	klogg	uvp	Difference
10 GB	10.98 s (every time)	0.01–0.07 s	150–1,100×
50 GB	52.55 s (every time)	0.01–0.07 s	750–5,000×

klogg rebuilds its index every time you open the file. That is the dividing line between
"look at it once" and "come back to it again and again".

Which to use

• Looking once → klogg is enough. It opens fast, it is free, and it is open source
• Find it, then read it → uvf … -open. One pass over the file gets you all the way to the window. Free
• Coming back to the same file → UwView Pro. Opening again takes 0.01–0.07 s, searching takes 6.34 s

How it was measured (you can repeat it)

• File: OpenStreetMap Japan XML, one file of 51,254,526,392 bytes (the 10 GB and 3 GB files are splits of the same data)
• Term: 東京, fixed string. Hit counts confirmed at 11,274 / 11,393 / 94,979 — all three matched
• sudo purge before every run, then a 10 s settle; everything measured cold
• Mac M4 / 32 GB / external USB SSD; klogg 24.11.0.1685; UwView free GUI and uvf v1.6.5
• The GUI runs were timed by hand (open = until the index finishes and you can move anywhere in the file; search = until the hit count settles)

uvf is free and runs on Windows, macOS and Linux as a single binary from
GitHub Releases.
If you want an index so you can come back to the same file, that is UwView Pro.