"Just use rg" is good advice — most of the time. ripgrep is Rust-powered, multithreaded, .gitignore-aware, and for scanning thousands of files in a repository it is deservedly the default.
But what about one huge file? When the target is a single 51 GB file, does rg's design still pay off? I measured it.
Setup
- Target: OpenStreetMap Japan,
japan-latest.osm— a single XML file, 51.25 GB / 892 million lines - Search term:
東京("Tokyo" — a realistically frequent hit) - Environment: Mac with an external USB drive (measured physical bandwidth: 0.41 GB/s) — an I/O-bound setup
- Contenders: ripgrep (rg), grep, and UwView Pro — my own viewer for huge files. Yes, I built one of the contenders; keep that in mind and discount accordingly.
Results
| Tool | First run | Repeat runs | Notes |
|---|---|---|---|
| ripgrep (rg) | 71.49 s | ~same every time | reads all 51 GB, every time |
| grep | 64.64 s | ~same every time | plain stream processing |
| UwView Pro | 54.7 s (while building its index) | 14 s (searching the compressed cache) | builds a 5.3 GB sidecar cache on first pass |
In this environment, rg was slower than grep on a single huge file. And from the second search onward, the tool that keeps a cache is roughly an order of magnitude faster than either.
Why this happens
This is not a defect in ripgrep. It's a mismatch of specialties.
- rg's parallelism works across files. With one file, there is little for those threads to do
- In an I/O-bound environment (0.41 GB/s USB), every tool's floor is "the time it takes to read 51 GB." Disk dominates; regex-engine cleverness barely matters
- grep beat rg here simply because plain stream processing carries less overhead, and under these conditions overhead was the only variable left
UwView Pro's slight first-run edge comes from building its line-offset index and compressed sidecar cache while reading. The real difference appears on the second search: read 51 GB again, or read a 5.3 GB cache — that design choice is the whole ~9× gap.
Full disclosure
- These numbers are from an I/O-bound setup (external USB). On a fast internal SSD the read floor drops, and the ranking could change
- rg's home turf is multi-file search. For repository-wide scans I know of no faster general-purpose tool
- UwView Pro is my own product, and these are my own measurements on my own machine — not an independent benchmark
- Even UwView Pro's 54.7 s first run cannot escape reading the file once at storage speed. What changes is that you can view and search while it reads — and everything after that first pass
Takeaway — "just use rg" depends on the job
- Scanning many files across a repo → ripgrep
- A simple filter in a pipeline → grep is fine
- Returning to the same huge single file again and again during an investigation → a tool with an index and cache is an order of magnitude faster
None of this is surprising once stated, but I hadn't seen numbers for the huge-single-file case, so here they are.
How the cache and index work (sidecar compression, line offsets, drill-down search) is documented here:
Top comments (0)