Introduction
I am building an editor called Nople. It is an editor for reading and checking code, and now and then fixing it, and its features are kept to what that needs.
One of Nople's features is how fast it opens large files. That is also what "faster than Zed" in the title means. Even a 10 GB log opens without a wait. Once it is open, you can scroll, search and edit right away.
This is a 10 GB log open in Nople. It has 131 million lines.
This article explains the trick that lets it open files faster than Zed. The trick involves how gowaku, the UI framework I wrote and used for Nople, does its work. So the first half is about Nople and the second half is about gowaku.
How fast is it?
Before I explain the trick, here is how fast it actually is. I opened the same file in Nople, VS Code and Zed and measured each.
I measured on a Windows 11 laptop (Core i7-12700H, 16 GB RAM, NVMe SSD). The file is a log with about 80 bytes per line. VS Code and Zed were started with no settings and no extensions. I measured the time from launching the editor until the file's contents show in the window. Memory is the peak during that time.
| Nople | VS Code 1.133.0 | Zed 1.27.0 | |
|---|---|---|---|
| 1 GB: until the contents show | 0.31 s | 8.0 s | 2.1 s |
| 1 GB: memory | 271 MB | 3.0 GB | 2.7 GB |
| 10 GB: until the contents show | 0.35 s | 117 s | does not open |
| 10 GB: memory | 392 MB | 5.8 GB | n/a |
VS Code automatically turns off highlighting, wrapping and similar features when it opens a large file. On the 10 GB file, Zed showed "File is too large to load" and did not open it.
I measured the times by capturing the window on screen. The scripts I used and the raw results are in Nople's tools/compare.
What happens if you read the whole file
If an editor reads the whole file before displaying it, just reading 10 GB takes about 10 seconds, even on a fast SSD. If it keeps what it read as one string per line, memory use goes past the size of the file.
But the editor only ever displays the few dozen lines that fit on screen. The other 100 million lines are not needed until you scroll to them. So it should be enough to keep none of the contents and read each line only when it is needed.
The rest of the article follows this idea in order, from opening the file to displaying it.
1. Opening: keep no contents, only where lines start
When a file is opened, only the first 1 MB is read on the spot, and the newline positions in it are recorded. At that point the open call returns, and the editor shows the first screen. Another goroutine reads the rest.
What gets recorded is where a line starts, once every 64 lines. Two positions are kept. One is the byte offset in the file. The other is the offset in the text, where every line break counts as one byte. In a CRLF file, the two drift apart. From here on I call this list of positions the index.
An entry is 16 bytes. One entry per line would take 1.6 GB for 100 million lines. One per 64 lines takes only 25 MB.
To display a line, the editor reads the 64-line block that contains it from the file and splits it at newlines. At 80 bytes per line, that is about 5 KB. Up to 512 blocks are cached, so as long as you scroll around the same area, nothing is read from disk.
One goroutine handles drawing and input. From here on I call it the UI goroutine. The indexing goroutine collects the line positions it finds and passes them in batches to the UI goroutine. When the UI goroutine receives a batch, all it does is tell the editor how far the lines now go. As indexing proceeds, the scrollbar thumb gets shorter. While the index is being built, you can still scroll, search and edit.
The next table shows opening and indexing measured separately.
| 1 GB (13.1 million lines) | 10 GB (131 million lines) | |
|---|---|---|
| Open and show the first line (Linux on WSL 2, cold cache) | 6.1 ms | 6.9 ms |
| Index all lines (Windows, warm cache) | 0.30 s | 8.3 s |
| Memory used by the index | 4.4 MB | 36 MB |
The first row is the time for opening the file alone. It does not include starting the editor or drawing. The measurement conditions and the steps to reproduce them are in Nople's docs/PERFORMANCE.md and gowaku's docs/BENCH.md.
2. Editing: keep only the lines you touched in memory
With no contents in memory, there is nothing to edit. From here on, I'm talking about gowaku's editor widget, which Nople uses. gowaku's editor holds the document in a structure called a piece table. The document is a sequence of "pieces", and there are two kinds:
- Source pieces: a range such as "lines 1,000 to 50,000,000 of the file". They hold no content.
- Memory pieces: the actual lines inserted by edits.
Right after opening, the document is a single source piece. Rewrite line 5,000,000, and it becomes three pieces:
So typing one character in a 10 GB file adds only one line's worth of memory.
3. Saving: write out a snapshot on another goroutine
To save, the pieces are written out from first to last. Source pieces are read from the original file and written as they are, and memory pieces are written from memory.
The output goes to a temporary file created in the same directory as the original. The original is replaced only after writing finishes, so if the process crashes partway through, no half-written file is left behind.
In Nople's measurements, writing takes about 1 second for 1 GB and about 50 seconds for 10 GB. If the UI goroutine did the writing, the window would freeze for that whole time. So the write also goes to another goroutine.
While the writing goroutine reads the document, though, the UI goroutine reads the same document to draw and scroll. So when a save starts, the UI goroutine takes a snapshot of the document. It copies only the sequence of pieces and the list of lines held by the memory pieces. It does not read a single line from the source, so it finishes at once. The writing goroutine reads only this snapshot, so it never collides with drawing.
While saving, Nople does not accept edits. If you could edit during a save, the file being written and the contents of the document would no longer match. The progress shows in the tab and the status bar.
When the save is done, the document's source is switched to the file that was just written. The undo history, the selection and the scroll position stay as they were.
4. Searching: run in the background, show matches as they are found
Searching 10 GB means reading the whole file, no matter what. If the UI goroutine did the reading, the screen would stop for the duration.
So search also runs on another goroutine. As with saving, it takes a copy of the piece sequence when the search starts. This copy is cheap, because a source piece holds only a range.
The search goroutine reads the file itself. It passes the match positions it finds to the UI a little at a time. The match count in the search bar goes up each time results arrive.
The hard part is edits made while a search is running. The search works through the old copy, so the line numbers of the matches it finds drift away from the current document. But restarting the search on every edit would mean rereading 10 GB on every keystroke.
I solved this by recording the edits made since the search started. When results arrive, that record is used to convert their line numbers into line numbers in the current document. The edited lines themselves are searched again on the spot by the UI goroutine. Lines are short, so this takes no time at all.
I have not measured search speed on 10 GB. What I measured was gowaku's editor widget searching a 443 MB file with 5 million lines. A case-insensitive search had all matches in 0.73 s. The longest frame during that time was 8 ms.
5. When another program modifies the file: take in appends, re-apply edits
Here the story goes back to Nople. Logs keep growing while you have them open. Once a second, Nople checks the file's size and modification time. If something was appended at the end, only that part is added to the index. As with tail -f, new lines appear at the bottom.
The tricky case is when the file is rewritten while you are in the middle of editing it. Then Nople re-applies your edits to the new file.
To do that, it remembers, for each edited line, the original line and the two lines before and after it. It looks for the same sequence in the new file, starting near the original position, and applies the edit where it finds it. This is the same idea as the patch command applying a diff. If no place can be found for an edit, Nople tells the user and lets them choose what to do.
6. Displaying: why 100 million lines don't slow it down
So far this has been about how the file is read and held. But knowing line positions right away is useless if the display side cannot cope with 100 million lines. The display side is gowaku's job.
Build only the visible lines
gowaku's editor lays out and draws only the lines that are visible on screen. Whether there are 100 lines or 100 million, a frame deals with a few dozen. When there is no wrapping and every line has the same height, where a given line sits on screen comes down to one multiplication.
Keep the scroll position in float64
There is one problem that only shows up at the scale of 100 million lines. Many UI libraries store coordinates as float32. float32 can represent every 1-pixel step only up to about 16.78 million pixels. With 20-pixel lines, the steps get coarser past 840,000 lines.
When gowaku still stored the scroll position as float32, this is what actually happened:
| What was displayed | Smallest step near the end | What happened |
|---|---|---|
| Table, 10 million rows (32 px/row) | 32 px (1 row) | An 11 px scroll did not move it at all |
| Table, 100 million rows | 256 px | Even a 40 px scroll did not move it |
| Editor, 20 million lines (20 px/line) | 32 px | A 7 px scroll did not move it |
Small touchpad movements were rounded away, and the wheel jumped a line at a time.
The fix is simple. Only the scroll position and the top of each line are float64. Both are absolute positions within the whole content. To get an on-screen position, the scroll position is subtracted from the top of the line, and the result is converted to float32. On-screen coordinates are a few thousand pixels, so float32 is enough there. Now scrolling moves 1 pixel at a time, even at the end of 100 million lines.
Draw no frame when nothing happens
When there is no input and no animation, gowaku draws no frames. The event loop blocks in the OS call that waits for the next event. No timer is running either. You can leave a 10 GB log open and it will not use any CPU.
The UI goroutine is woken only when the indexing or search goroutine delivers results. Once awake, it rebuilds only what changed, draws one frame, and goes back to waiting.
The division of work between Nople and gowaku
So far the story has switched back and forth between Nople and gowaku. Here I sort out what the application, Nople, and the library, gowaku, are each responsible for.
Everything that deals with the file is on the Nople side. When to watch and how to handle conflicts are for the application to decide.
That is why gowaku's editor never opens a file itself. What it takes is a "line source" that satisfies this interface:
type LineSource interface {
Len() int // number of lines known so far
Line(i int) string // line i
Offset(i int) int64 // byte offset where line i starts
LineAt(off int64) int // the reverse
Reader() LineReader // for reading from other goroutines
}
Nople implements this interface with its file index and cache, and passes it to gowaku's editor.
Because the work is split this way, a line source does not have to be a file. Database query results or logs coming over the network can be displayed and edited in the same editor.
gowaku's sample examples/bigfile is a simple read-only line source. It is short, about 250 lines, so it makes a good starting point for writing your own.
Limitations
This approach has limits. Nople currently treats any file over 8 MB as a "large file" and opens it with the method described here. Large files have these limits:
- They are read as UTF-8. Files of normal size are also checked for Shift_JIS, but large files are not.
- Git change markers and the outline are not shown. Both need the whole file.
- Case-insensitive regex search is slow: 12 seconds on 443 MB. The screen does not freeze, but you have to wait for all the matches to come in.
- An edit that replaced a range of more than 4,096 lines at once cannot be re-applied when another program changes the file.
Summary
Here is what it took to open a 10 GB file without waiting:
- Keep none of the file's contents, and remember only one line position per 64 lines.
- Keep only edited lines in memory, and combine them with the original file when saving.
- Run indexing, search and saving on other goroutines, so the screen never stops.
- Build only the visible lines on screen. Keep the scroll position in float64. Draw nothing when nothing happens.
If you ever need to open a large log, give Nople a try. The editor widget lives in gowaku, so you can also build the same widget into your own application.
The next article will cover gowaku's internals: recomputing only what changed, calling the GPU from pure Go, and Japanese input and typesetting.
I am Japanese and not fluent in English, so I wrote this article with the help of AI. I checked the content and the numbers myself.





Top comments (0)