y4u

Posted on Sep 16 Originally published at uvp.y42u.net

Picking the One Place to Beat ripgrep — Read 1/9 of the Bytes, Then Hand the Result From the CLI to the GUI

#ripgrep #cli #performance #showdev

You cannot out-engineer ripgrep.

ripgrep finishes a 258 GB, 4.5-billion-line file in 278 seconds. wc -l finishes the same file in 268. The gap is 4%, and for those 278 seconds the CPU is mostly asleep. When I saw that number, I knew there was no algorithmic path to catching up. So where do you fight?

Up front

I ran ripgrep and my own index-based uvp against a single 258 GB, 4.5-billion-line file under the same rules (Apple M4 / 32 GB / external SSD, sudo purge before every run, /usr/bin/time -p real, all figures second run = index present).

ripgrep runs at 1.04× the time it takes to read the file once. It does not saturate a single core (user/real = 0.04). There is no way to beat that on efficiency
There is exactly one place to win: the number of bytes read. Index plus compression takes 258.68 GB down to 28.6 GB (1/9.04). Read 1/9 the bytes, and the time follows: 278.35 s → 35.51 s
The first question is always a loss (building the index takes 333.67 s). Break-even is at question 1.37. At 3 GB, ripgrep wins no matter how many questions you ask
One by-product. The moment you hold an index, the CLI result can be handed straight to the GUI and narrowed again there as many times as you like. A drill-down stage costs 0.24 s (50 GB, v1.6.2), which makes that round trip practical

I once wrote about picking a single axis against commercial editors instead of counting features. This is the CLI edition. The opponent is ripgrep, and the losses come first.

The premise: fight fair and you lose

Before any comparison, the physical floor.

$ /usr/bin/time -p wc -l us-260726.osm
4509830821
real 268.73

268.73 s = 963 MB/s. That is the time to read 258 GB once. Against that, ripgrep:

Q	Term	Hits	`rg -c`
1	New York	100,492	275.80
2	Brooklyn	30,871	278.35
3	Central Park	14,306	276.75
4	Statue of Liberty	88	293.85
5	coffee_shop	36,506	346.21
		median	278.35

1.04× the physical floor. Same for 88 hits or 100,000. ripgrep is a tool that "finishes when it has finished reading", and the reading speed is set by the disk.

The CPU numbers make it sharper.

rg -c 'New York'   real 275.80   user 11.55   sys 51.60

Of 275 seconds, 11 were spent on the CPU. On a 10-core M4, that is 4% of one core. It does not need more — the disk cannot keep up.

That is the starting point. In an I/O-bound world, a faster CPU changes nothing, and neither does parallelism. Polish the algorithm all you like; the 268.73-second wall does not move.

(A side note: this is also why "written in Rust, therefore fast" is not a general rule. amber, which parallelises within a single file, takes 357.69 s on the same file — slower than ripgrep. Burning 25–36× the CPU time does not help when there is one disk. Measurements are in the 258 GB article.)

The one move left: read fewer bytes

If the disk is the ceiling, the only lever is how much you read.

uvp builds an index and a compressed copy first.

$ /usr/bin/time -p uvp convert us-260726.osm
real 333.67

333.67 s — 1.24× the physical floor — to read 258 GB once and produce a .uwvz. The result is 28,608,409,953 bytes = 1/9.04 of the original. Every later search reads only those 28.6 GB. The original 258 GB can be deleted.

	Bytes read	Floor at 963 MB/s	Measured (median)	Ratio to floor
`rg -c`	258.68 GB	268.73 s	278.35 s	1.04×
`uvp`	28.6 GB	29.72 s	35.51 s	1.20×

Both are pinned to their own I/O floor. The only difference is how many bytes each must read. And per byte, ripgrep is the more efficient one (1.04 versus 1.20). What wins is not the algorithm; it is the prepayment.

The CPU picture flips too.

rg   real 275.80   user  11.55   → user/real = 0.04 (waiting on I/O)
uvp  real  34.70   user 168.63   → user/real = 4.86 (~5 cores decompressing and searching)

The moment you read 1/9 the bytes, the disk has slack and the CPU becomes the bottleneck. The bound flips — that is what this approach really is.

The table of losses

Pick one axis and you lose on the others. All of them, up front.

Situation	Result	Why
First question	ripgrep 278.4 s vs uvp 369.3 s (1/1.33)	You pay 333.67 s for the index. A consistent 1.2–1.3× loss at every size (3 GB 1/1.18, 10 GB 1/1.31, 50 GB 1/1.29)
3 GB	ripgrep faster from the second question on too (1/2.3)	The file sits in the OS cache and rg scans it in 0.33 s. uvp always pays 0.6–0.7 s of startup plus index load. Absolute gap: 0.4 s
`-head N` (first few only)	`rg \	head` 0.05 s vs uvp 0.32 s
Efficiency per byte	rg 1.04× vs uvp 1.20×	As above. Decompression costs
Drill-down with context (`-C N`)	~8.5 s per stage (50 GB)	It looks outside the previous hit lines, so it re-reads the `.uwvz`. At ±0 it is 0.24 s

3 GB is ripgrep's ground, and I concede it. The gap is 0.4 s, so conceding costs nothing. The index starts to matter once the file no longer fits in memory.

Which question pays for the index

CLI(N) = N × 278.35            (rg, median of 5 questions)
uvp(N) = 333.67 + N × 35.58    (index build + mean .uwvz search)

N* = 333.67 ÷ (278.35 − 35.58) = 1.37

Questions N	ripgrep	uvp	rg ÷ uvp
1	278.4	369.3	1/1.33 (ripgrep wins)
2	556.7	404.8	1.38×
5	1,391.8	511.6	2.72×
20	5,567.0	1,045.4	5.33×

It flips at the second question. The formula applies to any index-based approach (T_index ÷ (T_scan − T_search)). 10 GB and 50 GB show the same shape: from the second run on, 5.5–6.9× at 10 GB and 7.3–10.2× at 50 GB.

That is the whole "speed" story. Honestly, on its own it would barely merit an article. Of course the second search is faster once you have built an index.

The by-product: the CLI and the GUI became one surface

The prepayment bought something I had not planned for.

Put -open at the end of a uvp command and the CLI's result is handed straight to the GUI.

uvp us-260726.osm.uwvz 'New York' -C 0 -grep hospital -v -grep school -v -grep Brooklyn -open

Step by step:

The CLI searches the 258 GB file (via its 28.6 GB index), narrows through three stages to 35 lines — 34.88 s
Those 35 lines are handed to the GUI through a temporary file. The GUI does not search again (v1.6.2; until then only the conditions were passed and the GUI re-ran the search, so on a huge file you waited for the same search twice)
The GUI opens. The index already exists, so reopening does not read 258 GB
Click a row in the list to jump to it in the text. Show ±N lines of context. Narrow again with another term — that costs 0.24 s per stage (v1.6.2)

Step 4 is the point. In the interest of honesty: there was a period when only the drill-down was oddly slow in the comparison against rg. Single searches ran 7–9× faster, yet stacked drill-down stages managed only 3–4×. It turned out one code path was re-reading per stage; v1.6.2 fixes it. At 50 GB, three stages went from 29.46 s to 8.10 s — 0.24 s per stage. Three stages on 258 GB now take 34.88 s, the same as a single search (34.85 s). The cause and how it was isolated will get their own article.

With that, the following workflow finally holds together:

Narrow in a script, look at it as a person, and narrow again while looking.
Not waiting 278 s for every re-narrowing (CLI only), not re-reading 258 GB every time the GUI opens (GUI only) — build the index once, and let both the CLI and the GUI use it.

A grep pipeline is "one pass no matter how many stages", and that is the right design for a CLI. But change the condition and try again and it is another full pass. The GUI is quick to change conditions but slow to open. Put an index in between and both of those "slow"s disappear.

Because the CLI and the GUI call the same functions (drill-down, tally, sequence), the count the CLI prints and the count the GUI shows always agree. Verification: 81 command combinations plus 12 gz/zip inputs plus every 258 GB case were checked byte-for-byte against rg / sed, with zero defects on the product side.

What was done, in short

ripgrep cannot be beaten on efficiency. 1.04× the physical floor, 4% CPU. It is finished
The only place to win was "bytes read". Read 1/9 and you get 1/8 the time — no more, no less
The first question is always a loss. Break-even at question 1.37. At 3 GB you lose every question. That ground is conceded
As a by-product of the prepayment, the CLI and the GUI share one index and pass results between them
The one place I picked is not "fast" but "inside a single investigation, moving between CLI and GUI without waiting"

The tools

uvp ships with UwView Pro v1.6.2 (Windows, macOS and Linux, one licence for all three; one-time or monthly; 14-day free trial, and uvp works during the trial). The free UwView ships uvf, which has no index — search and -open only, and no speed to speak of.

All figures are my own measurements on my own machines, not an independent benchmark. ripgrep is among the fastest there is for searching across many files, and this article does not contest that. It is only about asking a gigantic single file many questions. If you spot a configuration issue, tell me; I will check and correct.

Conditions and commands

Apple M4 / 32 GB / macOS 26.3.1 / external SSD (963 MB/s)
ripgrep 15.2.0 / uvp 1.6.1–1.6.2 / sudo purge before every run, 150 s (258 GB) or 60 s (50 GB) cooldown between runs
OpenStreetMap USA XML 258,679,440,228 bytes / 4,509,830,821 lines; OpenStreetMap Japan 51,254,526,392 bytes

/usr/bin/time -p wc -l us-260726.osm                       # physical floor
LC_ALL=C rg -c -F 'New York' us-260726.osm                 # rg
uvp convert us-260726.osm                                  # build the .uwvz index
uvp us-260726.osm.uwvz 'New York' --no-line-number         # uvp (second run)
uvp us-260726.osm.uwvz 'New York' -C 0 -grep hospital -v -grep school -v -grep Brooklyn -open

DEV Community