Why File.ReadAllLines Throws OutOfMemoryException on a 10 GB File in C#

The job runs fine against the 200 MB sample. It dies in production against the real 10 GB log, before a single line is processed.

Here is the version almost everyone writes first.

The junior version

string[] lines = File.ReadAllLines(@"C:\data\events.log");

foreach (string line in lines)
{
    if (line.Contains("ERROR"))
        Console.WriteLine(line);
}

// Unhandled exception. System.OutOfMemoryException: Insufficient memory
// to continue the execution of the program.
//    at System.Collections.Generic.List`1.AddWithResize(String)
//    at System.IO.File.ReadAllLines(String path)

Nothing was printed. The process never reached the loop.

Why it breaks

File.ReadAllLines is not a reading primitive, it is a materialising one. Internally it opens a StreamReader, appends every ReadLine() result to a List<string>, and returns list.ToArray(). The method only returns once the last line of the file is in memory.

Three things multiply the cost:

• UTF-16. .NET strings hold two bytes per character. Your 10 GB of ASCII on disk becomes ~20 GB of character data on the heap.
• Per-object overhead. Each line is a separate object: 8 bytes of header, 8 for the method table pointer, 4 for the length, a null terminator, padded to an 8-byte boundary — call it ~24 bytes per line — plus 8 bytes for its slot in the array. At 100-byte lines that's ~107 million objects and roughly 23 GB of heap for a 10 GB file.
• The doubling. The backing array of the List<string> grows by repeated doubling, and every array above 85,000 bytes lands on the Large Object Heap, which is not compacted by default. You allocate, abandon and fragment your way up through 100 MB, 200 MB, 400 MB of reference arrays, then ToArray() allocates one more full-size copy alongside the last one.

The GC cannot help. Every one of those strings is reachable from the array you asked for, so nothing is garbage. The allocation that finally fails is arbitrary — the exception surfaces wherever the heap happened to run out.

A detour worth taking: foreach does not make it lazy

This is the most common wrong fix:

foreach (string line in File.ReadAllLines(path))   // still OOMs
{
    Process(line);
}

A foreach evaluates its collection expression once, in full, before the first iteration. File.ReadAllLines(path) runs to completion and hands back a finished string[]; only then does enumeration start. The loop shape tells you nothing. The return type does: string[] is a result, IEnumerable<string> is a plan.

The other reach is File.ReadAllText(path).Split('\n'), which is strictly worse. A string is hard-capped at 1,073,741,791 characters — roughly 1.07 billion, no matter how much RAM the box has — so the 10 GB read fails on the string itself, before Split runs. If it did run, Split allocates every substring on top of the original, and on a CRLF file it leaves a stray \r on the end of every line.

And "just give the box more RAM" is not a fix, it is a subscription. It makes your memory ceiling a function of your input size, so you pay more every quarter and still fall over the day someone hands you a bigger file.

The senior version

foreach (string line in File.ReadLines(@"C:\data\events.log"))
{
    if (line.Contains("ERROR"))
        Console.WriteLine(line);
}

File.ReadLines returns a lazy iterator over a StreamReader. Each MoveNext() decodes exactly one line and yields it; the foreach disposes the enumerator — and with it the reader and the file handle — when the loop ends. Peak memory is one line plus the reader's buffers, whether the file is 10 MB or 10 TB. The 10 GB case now runs in a few megabytes.

Because it is a real IEnumerable<string>, LINQ composes over it without buffering:

var firstTen = File.ReadLines(path)
                   .Where(l => l.Contains("ERROR"))
                   .Take(10);

Siblings worth knowing:

File.ReadLines(path, Encoding.UTF8);           // explicit encoding

await foreach (string line in File.ReadLinesAsync(path))   // .NET 7+
    Process(line);

// Full control: bigger buffer, sequential-scan hint to the OS.
using var reader = new StreamReader(
    new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read,
                   bufferSize: 1 << 16, FileOptions.SequentialScan));
while (reader.ReadLine() is { } line)
    Process(line);

When streaming isn't enough

File.ReadLines opens the file when you call it, not on the first MoveNext() — the handle is live before the loop starts. So don't return File.ReadLines(...) out of a method that owns the path and expect the handle to be closed: it stays open until someone finishes enumerating or disposes the enumerator, a sequence nobody ever enumerates holds it until the finalizer runs, and with the default FileShare.Read it blocks writers on Windows for the whole run.

Enumerating twice reads the file twice. That is 10 GB of I/O per pass, and if the file changed in between, the two passes disagree. And the moment you call .OrderBy(...), .ToList() or .GroupBy(...) on the sequence, you have re-materialised the whole thing and bought the original bug back. .Count() is the one that looks like those and isn't: it walks the sequence and keeps nothing, so it costs you a full 10 GB pass in time while memory stays flat.

If you genuinely need random access or several passes, streaming alone won't carry you: build an index of line offsets on one pass, sort externally, or load only the projection you need. Streaming buys you constant memory over a single forward pass — that is the whole contract.

The takeaway

ReadAllLines gives you an array; ReadLines gives you a promise. Only one of them fits in RAM.

---

I post one of these every day — the same problem solved the way that works and the way that lasts.
One senior tip a week, by email: https://seniorvsjunior.higgsfield.app