We’ve all been there. You need to check a production log. You download the server.log, double-click it in VS Code, and... freeze.
VS Code is built on Electron. Loading a multi-gigabyte text file into the DOM is a death sentence for RAM and UI responsiveness. The standard solution is to close the editor and go back to less or tail in the terminal.
But I wanted the best of both worlds: the raw speed of CLI tools and the comfort of the VS Code UI (regex search, copy-paste, highlighting).
So, I built a custom extension with a Rust sidecar to solve this. Here is a deep dive into how it works under the hood.
The Architecture: Sidecar Pattern
The extension consists of two parts communicating via Stdin/Stdout (JSON IPC):
Frontend (TypeScript/VS Code Webview): Handles the UI, virtual scrolling, and rendering.
Backend (Rust): Handles file I/O, indexing, and searching.
The goal was simple: VS Code never holds the full file in memory.
🦀 The Backend: Rust & Memory-Mapping
The core logic resides in a binary called log-core.
1. Memory-Mapped I/O
Instead of reading the file into a buffer, I used memmap2. This maps the file on the disk directly into the process's virtual address space.
Benefit: The OS handles paging. Opening a 10GB file takes almost zero RAM allocation for the content itself.
Speed: Accessing a byte at offset 1,000,000 is as fast as accessing an array index.
2. The Line Index
When a file opens, the backend performs a single O(n) pass to build a Vec of line offsets.
offsets[0] = 0
offsets[1] = (position of first \n) + 1
...and so on.
This allows the backend to implement read_lines(start_line, count) efficiently. It calculates the byte range from the index, slices the memory-mapped file, and converts it using String::from_utf8_lossy (handling potential encoding issues gracefully).
3. Search & Regex
Search happens entirely in Rust. I use the regex crate for patterns or standard string matching for plain text. To prevent locking up the CPU on massive files, the search creates a stream of results with a hard limit (e.g., stopping after ~10k matches to keep the UI responsive).
⚡ The Frontend: Virtualization & Limits
The frontend is a VS Code Webview. The biggest challenge here isn't just "showing text," it's browser limits.
1. The 10 Million Pixel Problem
Browsers have a hard limit on the height of a DOM element (often around 10-30M pixels). A 10GB log file could easily exceed 100M pixels in height.
- Solution: I implemented a coordinate scaling factor. The scrollbar you see is "fake" (virtualized). We calculate a virtualHeight that fits within browser limits, and then map the scroll position back to the realLineNumber.
2. Virtual Scrolling & Buffering
The Webview maintains a state map: loadedLines: Map.
We only render the visible lines + a small buffer (BUFFER_LINES) into the DOM.
Missing ranges are calculated and requested from the Rust backend in chunks (CHUNK_SIZE).
Cache Eviction: As you scroll away, lines far from the viewport are removed from the map to keep memory usage flat.
3. The "Filter Paradox"
Implementing filtering (e.g., "Show only ERROR") was tricky.
If I filter a 1M line file and find 500 errors, I need to show a continuous list of 500 lines, BUT I still need to know their original line numbers for debugging.
Logic: The frontend builds a mapping: ViewIndex (0..499) → ActualLine (e.g., 504, 1200, 9000).
UI: The main view scrolls based on the ViewIndex, but the Gutter (line numbers) renders the ActualLine.
This means Ctrl+G (Go to Line) has to solve a reverse lookup: "Where is line 1200 in the filtered view?"
🔄 Follow Mode (Tail -f)
I wanted to replace tail -f.
Polling: The frontend triggers a refreshFile command every 500ms.
Backend: Rust checks the file size. If it grew, it re-maps the file (cheap operation) and scans only the new bytes for new line offsets.
Frontend: If the user is at the bottom (or in "Follow Mode"), the view auto-scrolls to the new lines. If the user manually scrolls up, Follow Mode pauses automatically.
📦 Build & Cross-Compilation
Since this relies on a native binary, I couldn't just publish JS code.
I set up a build pipeline using GitHub Actions to cross-compile the Rust binary for:
darwin-x64 & darwin-arm64 (Apple Silicon)
linux-x64
win32-x64
The VS Code Marketplace supports platform-specific builds. When a user installs the extension, VS Code automatically fetches the correct VSIX for their OS. The resulting package is surprisingly small (~0.8MB), with the compressed binary taking up most of that.
Conclusion
This project started as a way to fix my own frustration, but it turned into a great lesson on how much performance you can squeeze out of VS Code when you offload heavy I/O to a system language like Rust.
If you deal with massive logs, CSVs, or data dumps, give it a try.
👉 VS Code Marketplace: [https://marketplace.visualstudio.com/items?itemName=molchanovartem1994.log-analyzer-pro]
👉 GitHub Repo: [https://gitlab.com/molchanov.artem.1994/log-analyzer-pro]
Let me know if you have any questions about the memmap implementation or the Webview message passing!
Top comments (0)