I Expanded the Entire US OpenStreetMap to a Single 258 GB XML File — 4,509,830,821 Lines, and What Broke

A few days ago I benchmarked the native and browser builds of my viewer against a 3.16 GB, 100-million-line XML file carved out of Japan. The whole time I kept thinking the same thing: at 100 million lines, this thing still isn't trying.

UwView is built around one claim — "open any huge file, instantly readable and searchable." If you're going to say that out loud, you owe it to yourself to find out where it breaks. So I went and got the OpenStreetMap data for the entire United States.

Making the test file

Genuinely huge files are surprisingly hard to come by. Logs get rotated. Dumps get chunked. An unsplit, near-terabyte single file is something you basically have to manufacture yourself.

So I took the Geofabrik extract for the whole US (PBF, about 11 GB) and expanded it to XML with osmium cat. About 15 minutes later:

us-260726.osm
258,679,440,228 bytes  =  258.68 GB (240.9 GiB)

ls -lh reports 241G — same file, GiB spelling. One XML document. No splitting.

The numbers

Measured on real macOS hardware with UwView Pro, stopwatch-timed from the UI at second-level precision.

Item	Measured	Conditions
First open (until indexing completes)	5 min 28 s (328 s)	full scan of 258.68 GB ≈ 789 MB/s
Second open onward	instant	`.uwvz` sidecar (Pro feature)
Total lines	4,509,830,821	determined once indexing finished
Search "New York"	34.8 s / 100,492 hits	after indexing
Search "Boston"	~34 s	after indexing
`.uwvz` sidecar size	28.61 GB (28,608,409,551 B)	~11% of the original

The line count broke `int`

4,509,830,821  >  2,147,483,647

This is the part I want to put in front of anyone who has ever written a text viewer, a log parser, or an indexer. 4.5 billion lines does not fit in a signed 32-bit integer. Not the offsets — those obviously need 64-bit at this size — the line numbers themselves.

If line numbers are int anywhere in the pipeline, this file doesn't produce an error. It produces a negative line number, silently, somewhere past line 2,147,483,647, and every jump and every search result after that point is garbage. UwView carries line numbers as long throughout, which until this test was a design decision I had never actually needed. Now I have the receipt.

You can read the file while the index is still building

This is the screen I most want to show, and the reason I bothered building the file at all. (Screenshots are in the full write-up.)

The tab reads us-260726.osm 10%, the status bar shows an indexing progress bar with a cancel button — and the body is already displaying real <relation> and <member type="way" ref="..."/> elements, and it scrolls.

That's page-mode instant open: the file is never loaded, only the visible window is read and painted. So at 258.68 GB content appears the moment you pick the file. Indexing runs behind it, and line numbers appear when it finishes.

Most viewers show you the head of the file and nothing else until indexing completes. For 5 minutes and 28 seconds, do you stare at page one, or do you start reading? That gap matters most precisely when you are in trouble at 3 a.m.

The line-number gutter is empty at this stage. Line numbers arrive after indexing — by design, and I'd rather say it than hide it.

Search time is bound by data scanned, not hits found

The most interesting result was search.

"New York" returned 100,492 hits in 34.8 seconds, listed in the filter results window with line numbers and ±1 line of context — things like <tag k="operator" v="New York City Transit Authority"/>. Very much OSM.

"Boston" took about 34 seconds. Wildly different hit counts, essentially identical time.

So full-text search here is bound by how much data it scans, not how many matches it finds. That's a quietly useful property in practice: you never have to guess whether a rare word will be fast or a common word slow. Knowing that any word costs about 34 seconds is enough to decide whether you have time to go make coffee.

258.68 GB ÷ 34 s works out to about 7.6 GB/s, but that is a nominal figure, and I want to be explicit about it: Pro's search reads through the saved compressed cache (28.61 GB), so the bytes actually touched are far fewer. This is not a claim that an SSD hit 7.6 GB/s.

The sidecar is 11% of the original

On first open, UwView Pro writes the index plus compressed data into a .uwvz sidecar. Here it landed at 28.61 GB — about 11% of 258.68 GB, roughly 1/9.

That 28.61 GB is what turns 5 min 28 s into zero on every subsequent open. And 11% is also a storage story: instead of keeping the 258.68 GB XML around, keep the sidecar and open and search that directly when you need it.

Honest notes

All figures are from one machine, one file, one run each. Different hardware and storage will give different numbers.
The 5 min 28 s first open cannot be shortened. 258.68 GB has to be read through once. The claim was never "fastest first open" — it's that you can read during it, and every open after that is instant with line numbers.
Search needs the index. Browsing works while indexing runs; full-text search waits for it to finish.
The instant second open is a Pro feature. Free UwView doesn't persist a sidecar, so it rebuilds the index in the background each time — browsing and searching from the first open work the same.
"≈789 MB/s" and "≈7.6 GB/s" are derived (size ÷ elapsed), not storage benchmarks.

Summary

Testing at a size nobody meets in real work wasn't about bragging. It was about finding out whether the design claims survive contact with it:

At 10% indexed, the whole file was already browsable and scrollable
4,509,830,821 lines, and jumping to the end was instantaneous
Search cost ~34 s regardless of the word, and 100k+ hits came back with context
A 28.61 GB sidecar (~11%) made every reopen instant
And the line count itself proved why long was never optional

Near a terabyte, I didn't have to change how I say it.

UwView is free and cross-platform (Windows / macOS / Linux, plus a WASM build). If you work with big logs or dumps, throw it at the largest file you own — and if it breaks, please tell me. That's the useful part.