DEV Community

y4u
y4u

Posted on

258 GB, 4.5 Billion Lines, Five Questions: ripgrep Wins at N=1 and Loses at N=2

#cli #performance #showdev #tooling

I ran a whole job — five searches, a four-stage drill-down, a tally, a replace-all, and a save — against one unsplit 258 GB file, on the CLI and in a GUI, and timed everything.

CLI: 51 m 17 s. GUI: 15 m 35 s — or 12 m 29 s finishing with a .uwvz save, as you would in real use. But with a single question, ripgrep wins. Here's where the crossover is.

The file

us-260726.osm — OpenStreetMap, whole United States, expanded to XML, not split.

  • 258,679,440,228 bytes (258.68 GB)
  • 4,509,830,821 lines
  • Apple M4 / 32 GB / macOS 26.3.1 / external SSD

Rules: LC_ALL=C for grep, -c only (printing hits measures your terminal, not the search), sudo purge before each run, 150 s cooldown between runs, times are real from /usr/bin/time -p.

The floor 

$ /usr/bin/time -p wc -l us-260726.osm
4509830821
real 268.49
Enter fullscreen mode Exit fullscreen mode

268.49 s = 963 MB/s. Nothing beats one pass. Read everything below as a distance from this.

Single search

Pattern Hits BSD grep GNU grep ripgrep BSD grep (UTF-8)
New York 100,492 342.19 — 276.77 363.54
Statue of Liberty 88 373.34 — 277.06 416.78
k="highway" 60,471,216 474.40 273.66 275.04 503.43

Two findings:

  1. ripgrep is constant time. 88 hits or 60.47 M hits: 275–277 s, ±2 s. It uses 11–16 s of CPU. That's the floor +2%.
  2. macOS stock BSD grep is 1.73× slower than GNU grep. 474.40 vs 273.66 s, with 383.53 vs 66.54 s of CPU — a 5.8× efficiency gap. brew install grep and use ggrep.

Correction to my earlier post: on a 48 GB file I measured grep 64.64 s beating ripgrep 71.49 s. That storage ran at 0.41 GB/s, so everything was I/O-bound and bunched up. At 963 MB/s, CPU efficiency shows. "grep is faster on one huge file" only held on slow storage.

Five questions

New York / Brooklyn / Central Park / Statue of Liberty / coffee_shop.

Tool Total Per question
BSD grep 1,789.22 357.84
ripgrep 1,389.55 277.91
UwView Pro (GUI) 161.90 32.38

Hits: 100,492 / 30,871 / 14,306 / 88 / 36,506 — identical on both sides. 8.58×.

Drill-down and tally 

grep 'New York' f | grep -v hospital | grep -v school | grep -c Brooklyn
→ 35 hits / 343.52 s
Enter fullscreen mode Exit fullscreen mode

GUI: 35.2 s total, and the shape is the point —

stage 1  New York          32.2 s    → 100,492
stage 2  exclude hospital   under 1 s → 100,491
stage 3  exclude school     under 1 s → 100,484
stage 4  Brooklyn           under 1 s →      35
Enter fullscreen mode Exit fullscreen mode

Stages 2–4 were too fast for my stopwatch — the count changes as the click lands — because they only look at the previous stage's 100 k lines. A pipeline has to stream the whole file once. The 35.2 s counts each unmeasurable stage as a full second, so 9.76× is an upper bound and the real gap is wider. (Context set to ±0 so the line-stream semantics match; the final 35 matched exactly.)

Tally (| sort | uniq -c | sort -rn | head -20): CLI 344.38 s, GUI 34.0 s (10.13×; 32.0 s of that is the search, the tally itself was again too fast to measure). Top values matched to the tenth, and top-20 + 2,583 "other" = 100,492 — same as the hit-line count, because OSM XML puts one tag per line.

Replace and write 

sed 's/New York/NYC/g' f > /dev/null   → 741.48 s
sed 's/New York/NYC/g' f > out.osm     → 999.38 s
Enter fullscreen mode Exit fullscreen mode

Replacing costs 2.76× reading. 680 s of CPU; the disk idles.

GNU sed (gsed 4.10): 896.39 s — BSD is faster here. GNU grep was 1.73× faster, GNU sed is slower. "GNU is faster" is not a general rule.

GUI side: replace-all 62.0 s (100,992 occurrences) — 11.96× faster than sed's transform, because the original is never rewritten; edits accumulate as a diff and get written out once. Save .uwvz (28.61 GB) 138.1 s; save plain text 323.9 s.

The whole job

You save either plain text or .uwvz, never both, so the total comes in two rows.

Step CLI (rg) GUI Ratio
Open + index — 317.8 GUI only, once
5 searches 1,389.55 161.9 8.58×
Drill-down ×4 343.52 35.2 (max) 9.76×
Tally 344.38 34.0 (max) 10.13×
Replace (in the save row) 62.0 —
Save as text (258.68 GB) 999.38 323.9 2.59× incl. replace
Save as .uwvz (28.61 GB) no equivalent 138.1 —
Total (exporting plain text) 3,076.83 s = 51 m 17 s 934.8 s = 15 m 35 s 3.29×
Total (finishing with .uwvz) 3,076.83 s = 51 m 17 s 749.0 s = 12 m 29 s 4.11×

3.72× and 4.64× against BSD grep. "(max)" counts each unmeasurable step as a full second, so the GUI totals are upper bounds — the real gap can only be wider.

The first row produces the same artifact as the CLI. The second is how you actually finish: 28.61 GB that reopens without rebuilding the index. The artifacts differ — .uwvz is not plain text, so if another tool has to read it you need the export, which is the 3.29× row. If you are the one coming back, the .uwvz row is the real number: next session skips the 317.8 s index, while the CLI starts tomorrow's first question at 277.91 s again.

The crossover 

CLI(N) = N × 277.91
GUI(N) = 317.8 + N × 32.38
N*     = 317.8 / (277.91 − 32.38) = 1.29
Enter fullscreen mode Exit fullscreen mode
N ripgrep BSD grep GUI Ratio (rg ÷ GUI)
1 277.9 357.8 350.2 0.79× (rg wins)
2 555.8 715.7 382.6 1.45×
3 833.7 1,073.5 415.0 2.01×
5 1,389.6 1,789.2 479.7 2.90×
10 2,779.1 3,578.4 641.6 4.33×
20 5,558.2 7,156.8 965.4 5.76×

At N=1, ripgrep wins (0.79×). The index doesn't pay for itself. From N=2 it flips — 1.45× at two, 2.90× at five, 5.76× at twenty.

Did both sides do the same work?

Six cross-checks, all matching:

  1. Total lines 4,509,830,821
  2. Five hit counts 100,492 / 30,871 / 14,306 / 88 / 36,506
  3. Drill-down final 35
  4. Tally top 10 and total 100,492
  5. Replacements 100,992
  6. Text output 258,678,935,268 bytes

#6 is the one that matters. The GUI's non-destructive edit, written out, was not one byte different from sed's output.

5 is the fun one: grep -c counts 100,492 lines, sed replaced 100,992 occurrences. The 500 difference is lines with more than one "New York".

What the numbers don't show

  • Time to first visible text: 0 s. The index takes 317.8 s, but you can read the tail while it builds. grep is silent until it's done.
  • Going back from drill-down stage 4 to stage 1: too fast to measure. On the CLI you rebuild the pipeline and pay another 343 s.
  • Close mid-session and reopen to resume: both too fast to measure. Nothing to flush — the original was never touched.

Caveats

  • All measured on my machine. It may be a config issue on my side — if you know better, tell me and I'll correct it.
  • I develop one of the two things being compared (UwView Pro). Discount accordingly.
  • GUI numbers include stopwatch timing of on-screen operations; CLI numbers are /usr/bin/time -p.
  • "Too fast to measure" means exactly that — under my stopwatch's resolution. Totals and ratios count each such step as a full second, so they are upper bounds; the real gap can only be wider.
  • The CLI drill-down and tally numbers are BSD grep; GNU grep would likely be faster, unmeasured.
  • One-off searches, scripts, cron, piping onward — the CLI is the right answer. This isn't a replacement.
  • A GUI helps for interactive investigation, when you don't yet know what to throw away.

Which CLI command maps to what

If you want the per-command mapping rather than the whole-job numbers — grep -i / -E / -w / -v, sort | uniq -c | sort -rn, sed's replace-all, and the parts that honestly do not map (awk arithmetic, tail -f, pipes and scripting) — that's a separate cheat sheet: https://uvp.y42u.net/en/blog/uvp-cli-to-gui-map-en/

UwView Pro — Windows/macOS/Linux, one license for all three, 14-day free trial (editing features included).

Top comments (0)