You double-click a file. Your app freezes. The spinning cursor appears. Ten seconds pass. Twenty. Eventually one of two things happens — either the data finally appears, or the app crashes with an out-of-memory error.
If you have ever built or used a tool that works with large files, you have been on both sides of this experience. And if you have ever tried to fix it, you have probably reached for the same solutions everyone reaches for: load less data, use a faster language, buy more RAM.
None of those are the real fix. The real fix is a change in how you think about the problem.
The Naive Approach and Why It Fails
When most developers build a tool that reads a file, they follow the same mental model:
- Open the file
- Read all of it into memory
- Process it
- Show the result This works fine for small files. For a 1 MB config file or a 10 MB dataset, reading everything upfront is fast and convenient. But the approach has a fundamental flaw that only reveals itself at scale: it couples the size of the file to the amount of memory your program consumes and the time the user has to wait before seeing anything.
A 14 GB CSV file means 14 GB of data has to be read, parsed, and held in memory before the user sees a single row. On a machine with 16 GB of RAM, this either works barely (leaving almost nothing for the rest of the system) or triggers heavy disk swapping that makes everything grind to a halt. On a machine with 8 GB of RAM, it simply crashes.
But here is the thing that makes this approach not just slow but fundamentally wrong: the user never needs all 28 million rows at once.
A monitor is typically 1080 to 1440 pixels tall. A row in a data viewer is typically 30 to 40 pixels tall. That means a user can see at most 40 to 50 rows at any given moment. Out of 28 million.
We are loading 28 million rows to show 50. That is the problem.
The Insight: Knowing Where Data Is Versus Having the Data
The key mental shift is this: there is a difference between knowing where something is and having it in hand.
Imagine a library with 10,000 books. You walk in and ask the librarian for a specific passage. The librarian does not read every book into memory before answering you. They use a catalogue a small, fast index that tells them exactly which shelf, which book, which page. Then they go directly to that location and retrieve just what you need.
Your program can work the same way.
Instead of reading the entire file into memory, you make a single fast pass through it to build an index. The index does not contain the data, it contains the location of the data. Specifically, for every row in the file, you record the byte position where that row starts.
For a 28 million row file, this index is a list of 28 million numbers. Each number is 8 bytes (a 64-bit integer is large enough to address any position in any file you will encounter in practice). That is 225 megabytes — large, but manageable, and completely independent of how much actual data is in the file.
Once you have this index, you can jump to any row in the file instantly, without reading anything in between.
Building the Index
The index-building pass has one job: scan the file from start to finish, and every time you find the start of a new row, record its byte position.
The tricky part is doing this correctly. A naive implementation might just scan for newline characters but real-world files have edge cases. Fields can contain quoted newlines. A CSV field like "line 1\nline 2" contains a newline inside the field itself, and a naive scanner would treat that as a row boundary when it is not.
The index builder needs to understand the file format well enough to distinguish real row boundaries from newlines inside quoted fields. This is not complex it just requires tracking whether you are currently inside a quoted field as you scan.
The pass is fast. It does not parse field values, does not allocate strings, does not do anything except count bytes and record positions. On modern hardware with an SSD, you can scan several gigabytes per second. A 14 GB file takes 20 to 30 seconds to index. Not instant, but the window stays responsive throughout, you can show a progress indicator, and crucially, you only pay this cost once per file open.
On-Demand Reading
Once you have the index, reading any row is a two-step operation:
- Look up the byte position of that row in the index, this is an array lookup, essentially free
- Seek to that position in the file and parse just that one row When the user opens the file, you show them the first 50 rows. You read only those 50 rows. As they scroll down, you read the next batch. As they scroll up, you read the previous batch. At any given moment, you have only a small window of parsed rows in memory — typically the visible rows plus a small buffer above and below to make scrolling feel smooth.
Memory usage is now roughly:
index size + (visible rows × row size)
For our 28 million row file, that is about 225 MB for the index plus a few hundred kilobytes for the visible rows. The 14 GB file itself is never fully in memory. Total working set: well under 1 GB, regardless of file size.
The user experience is transformed. The file "opens" the moment the index is built. Scrolling is instant. Jumping to row 14,000,000 takes the same time as jumping to row 2,look up the position, seek, parse.
A Tool That Makes This Efficient: Memory Mapping
One technique that makes on-demand reading significantly faster in practice is memory mapping, a feature available in virtually every modern operating system.
When you memory-map a file, the OS maps the file's bytes directly into your program's address space. The file appears as an array of bytes you can access by index, without any explicit read calls. The OS handles paging behind the scenes, loading pages from disk only when you actually access them.
This is a natural fit for the lazy loading pattern. Your index stores byte positions. Accessing a row is as simple as indexing into the mapped array at that position. The OS efficiently handles the underlying disk reads, using its own cache so that recently accessed regions are served from RAM on subsequent accesses.
Memory mapping is not specific to any programming language or platform. The concept exists everywhere, it is how text editors handle large files, how databases handle their data pages, and how operating systems load executable code.
The Tradeoffs — Being Honest
This pattern is not free. There are real tradeoffs to understand before applying it.
Index build time is not zero. For a 14 GB file, the initial scan takes 20 to 30 seconds. If your users are opening the same large file repeatedly, you can cache the index to disk so subsequent opens are instant. If files change frequently, caching becomes harder to manage.
Operations that touch every row still take time. Sorting a column requires extracting that column's value from every row to build the sort key. Searching for a string requires checking every row. These operations still do a full pass, they just do it more efficiently because they only extract what they need rather than materializing the entire dataset. On a 14 GB file, a sort or search takes seconds rather than the instant response users might expect from in-memory data.
Random access has higher per-row overhead than sequential access. When data is in memory, accessing row 1,000,000 is a simple array index. When it is on disk, it requires a seek, which is fast on SSDs but not free. For workloads that access rows in random order at high frequency, in-memory storage is still faster. Lazy loading is optimized for the common case of a user looking at a small window of data and occasionally jumping around.
This pattern is best suited for read-heavy workloads. If users need to edit cells, you need to track modifications on top of the on-disk representation, which adds complexity. For pure viewing and analysis, the pattern is a natural fit.
The Mental Model Shift
The change required is not a new algorithm or a clever library. It is a shift in how you think about the relationship between files, memory, and what you show the user.
The old model:
File → Memory → Screen
Load everything, then show it.
The new model:
File → Index → Screen (fetching on demand)
Know where everything is, then fetch only what you need to show.
This shift has compounding benefits. A tool built on the new model does not just handle large files, it handles files of any size with the same performance characteristics. A 100 MB file and a 100 GB file feel identical to the user. The index for a 100 GB file is larger, but it is still orders of magnitude smaller than the file itself. The on-demand reads are the same either way.
The constraint you are designing around is no longer "how much memory can I allocate" but "how fast can I seek and read a small amount of data" and that constraint is much more forgiving on modern hardware.
Applying This to Your Own Tools
If you are building any tool that reads large files — a log viewer, a data explorer, a CSV editor, a file differ, a code search tool — the lazy loading pattern is worth considering from the start. It is significantly harder to retrofit into an existing tool than to design around from the beginning.
The core implementation has three parts:
The index builder — a single sequential pass through the file that records row or record boundaries. This is the only place where you read the entire file, and you only do it once.
The random access reader — a function that takes a row number, looks up its byte position in the index, seeks to that position, and parses just that row. This function should be fast and stateless — given the same row number, it always returns the same result.
The viewport manager — the layer between your UI and the reader that decides which rows to fetch based on what the user can currently see. It maintains a small cache of recently read rows and fetches new ones as the user scrolls.
These three pieces together give you a tool that opens any size file in the time it takes to build the index, uses memory proportional to what is visible rather than what exists, and stays responsive throughout.
The technique is not new. It is not exotic. It is how high-performance tools have always handled data at scale. The reason most applications do not use it is not that it is hard — it is that the naive approach works well enough at small sizes, and by the time files get large enough to matter, the architecture is already baked in.
Start with the right model, and large files stop being a problem worth solving. They become just another input your tool handles without complaint.
If you want to see this pattern in action, I built Columnar — a desktop CSV viewer that uses exactly this approach to open files with tens of millions of rows. The source is available at https://github.com/amitkroutthedev/Columnar. If you like the project please star the project such that more and more people find the project and if you like my work follow me on Github
Drop a comment if you've hit this problem before or solved it a different way — I'd love to hear your approach
Happy Coding!
Top comments (0)