DEV Community

y4u
y4u

Posted on

I Opened 51GB / 890M Lines of Real Data — Then Calculated How Far It Could Theoretically Go (~9.2 Quintillion Lines)

#showdev #performance #opensource #uwview

When I released my large-file text viewer UwView on GitHub, I claimed it "handles up to roughly 200 million lines at high speed." But until then, my measurements were all on synthetic data — files containing nothing but sequential line numbers. So first I put it to the test on real data an order of magnitude bigger, and then I worked out how far it could go theoretically, straight from my own code's data types and structures.

For the test I chose an XML dump of the entire OpenStreetMap (OSM) Japan dataset. The result: a file of 51GB and roughly 890 million lines — more than four times the "200 million lines" the product advertises.

Repository: amru195704/UwView (PolyForm Internal Use License 1.0.0)

How I Built the Test Data — Turning OSM Japan into a 51GB XML File

OSM's Japan data is distributed by Geofabrik in binary .osm.pbf format (about 2.3GB), so I used osmium-tool to expand it into XML text (.osm).

# Download the pbf (2.3GB, binary)
curl -L -o japan-latest.osm.pbf \
  https://download.geofabrik.de/asia/japan-latest.osm.pbf

# Convert to XML (text) with osmium → 51GB, 890 million lines
osmium cat japan-latest.osm.pbf -o japan-latest.osm
Enter fullscreen mode Exit fullscreen mode

The resulting file is UTF-8 text — an endless stream of <node> elements carrying latitude/longitude.

<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="osmium/1.19.1">
  <bounds minlat="20.08228" minlon="122.5607" maxlat="45.815403" maxlon="154.4709"/>
  <node id="31236558" version="5" timestamp="2020-06-27T07:40:04Z" lat="35.635073" lon="139.768101"/>
  ...
Enter fullscreen mode Exit fullscreen mode

Counting with wc -l gives 892,239,125 lines — squarely in "no editor will ever open this" territory.

Measured Results (51GB, 892,239,125 lines, Apple Silicon Mac, external SSD)

Measured using the benchmark harness bundled with UwView (UwView.Bench).

Item Measured value
File size 51,254,526,392 bytes (about 48 GiB)
Open + encoding detection 12 ms
Page-mode display (first 50 lines + 50 lines at the 50% position) 3 ms
Index build (one sequential read) 172.8 seconds (283 MB/s)
Total line count 892,239,125 (exact match with wc -l)
Checkpoint count (index) 3,485,310 entries ≒ 26.6 MB
Managed heap growth 33.3 MB
GetLine random × 1000 average 1.28 ms / p99 3.0 ms / max 5.7 ms
First-line fetch <?xml version='1.0' encoding='UTF-8'?> (2.9 ms)
Jump to the last line (line 892,239,125) </osm> in 0.006 ms
Encoding switch 3.4 ms (no index rebuild)
WorkingSet 12,083 MB (includes mmap'd pages; reclaimable by the OS)

Clearing the "200 Million Line Wall" Without Issues

What I most wanted to verify was whether things would hold up beyond the advertised 200 million lines. The short answer: no problems at all.

  • Accurate total line count: the 892,239,125 that UwView counted matched wc -l exactly, without a single line off.
  • Instant jump to the end: jumping to the last line (line 892,239,125, </osm>) took 0.006 ms. Once the index exists, jumping anywhere in the file is instantaneous.
  • Memory footprint stays flat: for a 51GB file, what's actually resident is roughly 26.6MB of index plus about 33MB of managed heap. The file itself is never loaded into memory.

UwView carries line numbers as a long (64-bit integer), but 890 million comfortably fits even within the int limit (about 2.14 billion). By design, line count is never going to be a bottleneck here.

Note that the "WorkingSet 12GB" figure looks large, but it's the cache of file pages touched via mmap — non-resident memory that the OS reclaims on demand under memory pressure. It is not memory the app itself has allocated.

The Difference from Synthetic Data — an Honest Look

Let's line up the earlier synthetic 200-million-line data against this run's real 890-million-line data.

Item Synthetic 200M lines (5.1GB) Real data 890M lines (51GB)
Index build speed 538 MB/s 283 MB/s
GetLine random (average) 0.005 ms 1.28 ms
Jump to last line 0.003 ms 0.006 ms
Index size 6.0 MB 26.6 MB

I'll be upfront that GetLine is slower than on synthetic data (0.005 ms → 1.28 ms). Three reasons:

  1. Random reads on the external SSD are the limiting factor (an internal SSD would be faster).
  2. Real XML has widely varying line lengths, so cache efficiency is worse than with synthetic data.
  3. 51GB doesn't fit entirely in memory (WorkingSet 12GB), so many reads end up hitting actual disk I/O.

Even so, it's still solidly in the single-digit-millisecond range, and scrolling and line-jumping feel plenty practical. The 283 MB/s index-build figure reflects a single sequential read of the full 51GB, and that number is entirely determined by storage speed.

Visible the Instant You Open It — the Strength of Page Mode

Another point worth emphasizing: you can start browsing without waiting for the index build (about 3 minutes). Opening plus encoding detection took 12 ms, and the first full screen rendered in 3 ms. Even for a 51GB file, the moment you open it — much like the instant less opens a huge file — the content is immediately visible in "byte-offset-based page mode," and once the index finishes building in the background, it quietly gets promoted to "line-number-based line mode."

There's no need to brace yourself for "opening a huge file" — the content is visible the instant you double-click. That's the experience UwView most wants to convey.

So How Far Can It Go, Theoretically?

With 890 million lines opening without trouble, I got curious: what's the theoretical upper limit on the number of lines? I worked through my own code's data types and structures to calculate the ceiling, step by step.

The short answer first: the type limit is about 9.2 quintillion lines

UwView tracks both line numbers and byte offsets as signed 64-bit integers (.NET's long) throughout. So the ceiling imposed by the data type itself is:

long.MaxValue = 9,223,372,036,854,775,807  (≈ 9.2×10^18 lines = about 9.2 quintillion lines)
Enter fullscreen mode Exit fullscreen mode

That's already a number with no real-world relevance, but the story doesn't end there. In practice, several walls appear well before you'd ever reach it.

Wall #1: File size

Since byte offsets are also long, the largest file UwView can handle is roughly 8 EiB (exbibytes). And because the smallest possible line is just a single newline byte (\n), the "upper limit on line count" and the "upper limit on file size" both land in the same 8 EiB neighborhood. Of course, in the real world the file system's own maximum file size kicks in first (ext4 caps at 16TB per file; APFS and NTFS allow several EB, and so on).

Wall #2: The index structure

To keep memory usage low, UwView uses a sparse index that records a byte position only once every N lines (default N=256). These checkpoints are held in a List<long> — backed internally by an array — but .NET arrays are capped at about 2.14 billion elements (Int32.MaxValue).

So at the default setting, the index array hits its ceiling at:

2.14 billion × 256 ≈ 5.5×10^11 lines (about 550 billion lines)
Enter fullscreen mode Exit fullscreen mode

This can be relaxed by increasing blockLines (N), but the index's memory footprint becomes the binding constraint before that. Index size is roughly "line count ÷ 256 × 8 bytes"; in this test, 890 million lines produced a 26.6MB index. Extrapolating out to 550 billion lines would require about 17GB of RAM for the index alone — and in practice, this is roughly where the real-world limit sits.

Wall #3: The real wall is "capacity"

To sum up, there are three layers:

Stage Limit Does it actually bind?
Limit imposed by the data type ~9.2×10^18 lines (about 9.2 quintillion lines) Effectively unreachable
Effective limit imposed by the structure ~550 billion lines by default (extendable via settings) Index RAM binds first
The real-world practical wall Storage capacity The realistic upper limit

The single biggest factor in the 890-million-line test was exactly this: file capacity binds before line count does. The 51GB file didn't fully fit in cache, and random reads dropped to millisecond-scale — but nothing broke. That's the natural consequence of a design that never keeps the whole file resident in memory (mmap), renders only the visible lines, and looks up line positions through a small sparse index.

So what's the actual real-world limit?

Reaching the type-imposed ceiling of 9.2×10^18 lines would require an 8 EiB-class file — far beyond anything a person could realistically provision. In practice, "the largest file you can actually get your hands on" is the real limit of the viewer.

That said, on a practical note: with as many as 890 million lines, building the index takes about three minutes and searches don't return results quickly. Something around 200 million lines is probably the comfortable sweet spot. The "up to 200 million lines" figure is only a rough guideline — in both the real-world test and the theoretical calculation, storage capacity becomes the limiting factor before line count does.

If anyone out there has an extremely large text file (hundreds of GB to TB scale) lying around, I'd genuinely love to hear where the limit actually lands when you open it in UwView.

Summary

  • Tested UwView on a real 51GB / 890-million-line dataset (OSM Japan as XML). It held up well beyond four times the advertised "200 million lines," with the total count matching wc -l exactly and a jump to the last line taking just 0.006 ms.
  • Resident memory is roughly 26.6MB of index plus 33MB of heap; the file itself is never resident. Browsing works in page mode from the instant it opens (12 ms / 3 ms) while the index builds in the background (about 3 minutes, 283 MB/s — storage-bound).
  • Theoretically, the type limit is about 9.2 quintillion lines (long) but effectively unreachable. Before that, the sparse index's array-element limit (about 550 billion lines by default) and index memory (about 17GB) bind first.
  • Past the index wall, what ultimately matters is the simple reality of storage capacity. The practical sweet spot is around 200 million lines.
  • The real-data conclusion — file capacity binds before line count — holds up exactly the same way in the theoretical calculation.

Related articles

Sources

Premium Edition: UwView Pro (on sale now, Windows/macOS/Linux)

A commercial edition that lets you "instantly view and search huge files anytime, save the index so that from the second time on it opens instantly with line numbers, search up to about 9× faster, and store logs at about 1/9 the size while still opening them directly." One-time purchase $129 / $9 per month.
→ https://uvp.y42u.net/en/pro-en/
(With UwView / UwView Pro, you can view and search the entire file the moment you open it — even on the very first open. The index is built in the background, and line numbers appear once it's complete. UwView Pro saves the index and compressed cache, so from the second open onward it opens instantly with line numbers already in place. Other viewers show only the head until indexing finishes, whereas UwView lets you view and search the whole file instantly from the first open.)

From the developer: a list of my apps, Kindle books and open-source projects is on GitHub: amru195704.

A note
The information in this article is provided for reference purposes only, and its accuracy or completeness is not guaranteed. If you notice any errors or inaccuracies, please let us know in the comments and we will review and correct them.

Top comments (0)