DEV Community

Huge Files, No Waiting Series' Articles

Back to y4u's Series
I benchmarked my log viewer against klogg at 3/10/48 GB — the second open is the real win

I benchmarked my log viewer against klogg at 3/10/48 GB — the second open is the real win

Comments
4 min read
I Opened 51GB / 890M Lines of Real Data — Then Calculated How Far It Could Theoretically Go (~9.2 Quintillion Lines)

I Opened 51GB / 890M Lines of Real Data — Then Calculated How Far It Could Theoretically Go (~9.2 Quintillion Lines)

Comments
8 min read
How much slower is the browser build, really? — UwView native (UVF) vs WASM, measured on the same file
Cover image for How much slower is the browser build, really? — UwView native (UVF) vs WASM, measured on the same file

How much slower is the browser build, really? — UwView native (UVF) vs WASM, measured on the same file

Comments
3 min read
Why We Split Search Results Into a Separate Window — Jump, Save, and Context Lines You Can Use \"While Looking\" With the Filter Popup
Cover image for Why We Split Search Results Into a Separate Window — Jump, Save, and Context Lines You Can Use \"While Looking\" With the Filter Popup

Why We Split Search Results Into a Separate Window — Jump, Save, and Context Lines You Can Use \"While Looking\" With the Filter Popup

Comments
4 min read
I Expanded the Entire US OpenStreetMap to a Single 258 GB XML File — 4,509,830,821 Lines, and What Broke
Cover image for I Expanded the Entire US OpenStreetMap to a Single 258 GB XML File — 4,509,830,821 Lines, and What Broke

I Expanded the Entire US OpenStreetMap to a Single 258 GB XML File — 4,509,830,821 Lines, and What Broke

Comments
5 min read
Shelve an Investigated Log at 1/9th the Size, Reopen It Tomorrow Right Where You Left Off

Shelve an Investigated Log at 1/9th the Size, Reopen It Tomorrow Right Where You Left Off

Comments
4 min read
I Rebuilt My 30-Year-Old Text Viewer in C#/Avalonia — It Matched the Industry-Standard klogg. Then I Redesigned It for Today's Hardware.

I Rebuilt My 30-Year-Old Text Viewer in C#/Avalonia — It Matched the Industry-Standard klogg. Then I Redesigned It for Today's Hardware.

Comments
6 min read
Reading Huge Logs by Color — a Highlighter That Doesn't Stutter at 100 Million Lines
Cover image for Reading Huge Logs by Color — a Highlighter That Doesn't Stutter at 100 Million Lines

Reading Huge Logs by Color — a Highlighter That Doesn't Stutter at 100 Million Lines

Comments
4 min read
How Many Search Threads Is the Right Number? I Measured 0.89 Billion Lines at 1-16 Threads
Cover image for How Many Search Threads Is the Right Number? I Measured 0.89 Billion Lines at 1-16 Threads

How Many Search Threads Is the Right Number? I Measured 0.89 Billion Lines at 1-16 Threads

Comments
3 min read
My Fanless Mac Skewed My Benchmark by 54% — Thermal Throttling and Round-Robin Measurement
Cover image for My Fanless Mac Skewed My Benchmark by 54% — Thermal Throttling and Round-Robin Measurement

My Fanless Mac Skewed My Benchmark by 54% — Thermal Throttling and Round-Robin Measurement

Comments
4 min read
"File Too Large to Open" — Investigating a 50 GB Log Without Splitting It
Cover image for "File Too Large to Open" — Investigating a 50 GB Log Without Splitting It

"File Too Large to Open" — Investigating a 50 GB Log Without Splitting It

Comments
2 min read
You Found the ERROR. The Cause Is in the Lines Before It

You Found the ERROR. The Cause Is in the Lines Before It

Comments
3 min read
I Benchmarked My Free Viewer Against EmEditor and klogg — Honestly

I Benchmarked My Free Viewer Against EmEditor and klogg — Honestly

Comments
2 min read
How to Open a 50GB Log File — and Reopen It in 0.05 Seconds. A klogg Alternative, Benchmarked
Cover image for How to Open a 50GB Log File — and Reopen It in 0.05 Seconds. A klogg Alternative, Benchmarked

How to Open a 50GB Log File — and Reopen It in 0.05 Seconds. A klogg Alternative, Benchmarked

3
Comments 2
7 min read
This File Is Too Large to Open — Four Go-To Tools That Sink Under Huge Files, and What Comes Next

This File Is Too Large to Open — Four Go-To Tools That Sink Under Huge Files, and What Comes Next

Comments
8 min read
The Day printf Debugging Hit 10GB — Four Ways to Live With Development Logs

The Day printf Debugging Hit 10GB — Four Ways to Live With Development Logs

Comments
10 min read
You Found the ERROR. You Still Can't Find the Cause — Four Ways to Trace Causality in Logs

You Found the ERROR. You Still Can't Find the Cause — Four Ways to Trace Causality in Logs

Comments
9 min read
Where Finished Logs Go — Deciding Between Delete, Keep, and Compress

Where Finished Logs Go — Deciding Between Delete, Keep, and Compress

Comments
9 min read
Is ripgrep Really the Fastest on a 51 GB Single File? I Measured rg, grep, and My Own Viewer

Is ripgrep Really the Fastest on a 51 GB Single File? I Measured rg, grep, and My Own Viewer

Comments
2 min read
Chasing an Outage by the Clock — Four Ways to Use Timestamps as Evidence

Chasing an Outage by the Clock — Four Ways to Use Timestamps as Evidence

Comments
14 min read
Eight Minutes for zgrep, Every Single Time — Keeping Logs Compressed and Still Searchable

Eight Minutes for zgrep, Every Single Time — Keeping Logs Compressed and Still Searchable

Comments
12 min read
Look at It Before You Feed It to jq — Four Times You Need the Raw Bytes

Look at It Before You Feed It to jq — Four Times You Need the Raw Bytes

Comments
11 min read
"Opens a Huge Log in Your Browser" Means Three Different Things

"Opens a Huge Log in Your Browser" Means Three Different Things

Comments
3 min read
Making the On-Call Night Lighter — Four Things You Set Up for Your 2 A.M. Self

Making the On-Call Night Lighter — Four Things You Set Up for Your 2 A.M. Self

Comments
11 min read
EUC-JP, UTF-16, and a Suspicion of EBCDIC — Reading Legacy Encodings in 2026

EUC-JP, UTF-16, and a Suspicion of EBCDIC — Reading Legacy Encodings in 2026

Comments
12 min read
"I Ordered It and It Never Arrived" — Four Support-Desk Log Investigations

"I Ordered It and It Never Arrived" — Four Support-Desk Log Investigations

Comments
13 min read
EmEditor Beat My Editor 7 at 3 GB. Then We Opened a 50 GB File on a 16 GB Laptop.

EmEditor Beat My Editor 7 at 3 GB. Then We Opened a 50 GB File on a 16 GB Laptop.

Comments
3 min read
The Night the Log Wouldn't Read — Four Character-Encoding Traps and How to Tell Them Apart

The Night the Log Wouldn't Read — Four Character-Encoding Traps and How to Tell Them Apart

Comments
9 min read
[AI Grades S–C] I Fed 10 of My Own Benchmarks to Gemini and ChatGPT: EmEditor at 3 GB, grep for One-Offs, My Tool Past 10 GB

[AI Grades S–C] I Fed 10 of My Own Benchmarks to Gemini and ChatGPT: EmEditor at 3 GB, grep for One-Offs, My Tool Past 10 GB

Comments
3 min read
[In Development] grep -v and grep -w, Now in a GUI: NOT Filters and Whole-Word Match for Huge-File Search

[In Development] grep -v and grep -w, Now in a GUI: NOT Filters and Whole-Word Match for Huge-File Search

Comments
2 min read
[In Development] grep -v and grep -w, Now in a GUI: NOT Filters and Whole-Word Match for Huge-File Search

[In Development] grep -v and grep -w, Now in a GUI: NOT Filters and Whole-Word Match for Huge-File Search

Comments
2 min read
Just Search and Replace: EmEditor, klogg, grep, 010 Editor, UltraEdit at 3/10/48 GB — Windows and Mac, Kept Separate

Just Search and Replace: EmEditor, klogg, grep, 010 Editor, UltraEdit at 3/10/48 GB — Windows and Mac, Kept Separate

Comments
3 min read
sort | uniq -c | sort -rn, Now a GUI Button: Ranking 10,967 Hits' Top Values in 22 ms

sort | uniq -c | sort -rn, Now a GUI Button: Ranking 10,967 Hits' Top Values in 22 ms

Comments
2 min read
Follow-up Tests: What I Learned by Re-testing the "Handles Huge Files" Claim

Follow-up Tests: What I Learned by Re-testing the "Handles Huge Files" Claim

Comments
3 min read
grep, uniq -c and sed in a GUI — and the Parts That Don't Map (a CLI Cheat Sheet)

grep, uniq -c and sed in a GUI — and the Parts That Don't Map (a CLI Cheat Sheet)

Comments
3 min read
258 GB, 4.5 Billion Lines, Five Questions: ripgrep Wins at N=1 and Loses at N=2

258 GB, 4.5 Billion Lines, Five Questions: ripgrep Wins at N=1 and Loses at N=2

Comments
6 min read
Only the Line Number Survived — Four Techniques for Inspecting Huge Data

Only the Line Number Survived — Four Techniques for Inspecting Huge Data

Comments
14 min read
The Traces Are in the Raw Log — Four Views on Lateral Movement, SQLi, and After-the-Fact Investigation

The Traces Are in the Raw Log — Four Views on Lateral Movement, SQLi, and After-the-Fact Investigation

Comments
14 min read
"less Is Enough" — Until It Isn't: Four Limits of Command-Line Craft

"less Is Enough" — Until It Isn't: Four Limits of Command-Line Craft

Comments
16 min read
Logs as Evidence — Four Principles for Preserving, Excerpting, and Proving Integrity

Logs as Evidence — Four Principles for Preserving, Excerpting, and Proving Integrity

Comments
16 min read
Before You Carry Production Logs Home — Four Points of Friction Between Logs and Your Dev Environment

Before You Carry Production Logs Home — Four Points of Friction Between Logs and Your Dev Environment

Comments
13 min read
Ten Thousand Sensors, 1.4 Billion Lines a Day — Four Fields Where Machines Write the Logs

Ten Thousand Sensors, 1.4 Billion Lines a Day — Four Fields Where Machines Write the Logs

Comments
14 min read
Following a Log That Won't Stop Growing — Four Situations Where tail -f Can't Keep Up

Following a Log That Won't Stop Growing — Four Situations Where tail -f Can't Keep Up

Comments
16 min read
"Please Produce the Logs from Three Years Ago" — Four Ways to Survive an Audit by Design

"Please Produce the Logs from Three Years Ago" — Four Ways to Survive an Audit by Design

Comments
16 min read
The Morning the Monthly Batch Died — Four Realities of Running a Data Pipeline

The Morning the Monthly Batch Died — Four Realities of Running a Data Pipeline

Comments
14 min read
Picking the One Place to Beat ripgrep — Read 1/9 of the Bytes, Then Hand the Result From the CLI to the GUI

Picking the One Place to Beat ripgrep — Read 1/9 of the Bytes, Then Hand the Result From the CLI to the GUI

Comments
9 min read
I Thought We Lost at 3 GB — Turns Out We'd Won (As Long as the Cache Is Cold)

I Thought We Lost at 3 GB — Turns Out We'd Won (As Long as the Cache Is Cold)

Comments
10 min read
ripgrep Is 2.9x Faster With no-mmap — Measured Against the Default mmap Read on One 51 GB File, on Three OSes

ripgrep Is 2.9x Faster With no-mmap — Measured Against the Default mmap Read on One 51 GB File, on Three OSes

Comments
5 min read
We Lose to klogg on Opening. uvf Wins the Whole Run — Re-measured at 50 GB

We Lose to klogg on Opening. uvf Wins the Whole Run — Re-measured at 50 GB

Comments
8 min read