Streaming Large HTTP Responses Without Killing Your Memory

Hey memory-conscious developers! 👋

Ever downloaded a 500MB file with HttpClient and watched your app's memory balloon? Today we'll fix that with streaming!

The Problem: Buffering Everything

By default, HttpClient buffers the entire response in memory:

// 😱 This loads EVERYTHING into memory!
var content = await client.GetStringAsync("https://example.com/huge-file.json");

For a 500MB response, that's 500MB of RAM. For 10 concurrent downloads? 5GB!

🧠 Fun Fact: The default HttpClient.MaxResponseContentBufferSize is 2GB on 64-bit systems. You can hit OutOfMemoryException before you even realize what's happening!

The Solution: HttpCompletionOption.ResponseHeadersRead

This magic option tells HttpClient to return as soon as headers arrive — before the body downloads:

using var response = await client.GetAsync(
    "https://example.com/huge-file.json",
    HttpCompletionOption.ResponseHeadersRead);

response.EnsureSuccessStatusCode();

// Now stream the content
await using var stream = await response.Content.ReadAsStreamAsync();

Streaming to a File

public async Task DownloadFileAsync(string url, string outputPath, CancellationToken ct)
{
    using var response = await _client.GetAsync(
        url, 
        HttpCompletionOption.ResponseHeadersRead, 
        ct);

    response.EnsureSuccessStatusCode();

    await using var contentStream = await response.Content.ReadAsStreamAsync(ct);
    await using var fileStream = new FileStream(
        outputPath, 
        FileMode.Create, 
        FileAccess.Write, 
        FileShare.None, 
        bufferSize: 81920, // 80KB buffer
        useAsync: true);

    await contentStream.CopyToAsync(fileStream, ct);
}

Streaming JSON with System.Text.Json

For large JSON arrays, you can deserialize while streaming:

public async IAsyncEnumerable<User> StreamUsersAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    using var response = await _client.GetAsync(
        "/api/users/export",
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(ct);

    await foreach (var user in JsonSerializer.DeserializeAsyncEnumerable<User>(
        stream, 
        new JsonSerializerOptions { PropertyNameCaseInsensitive = true },
        ct))
    {
        if (user != null)
            yield return user;
    }
}

Consuming Streamed Data

await foreach (var user in userClient.StreamUsersAsync(cancellationToken))
{
    Console.WriteLine($"Processing: {user.Name}");
    // Process one at a time - memory stays flat!
}

Progress Reporting

Want a download progress bar? Here's how:

public async Task DownloadWithProgressAsync(
    string url, 
    string outputPath,
    IProgress<double> progress,
    CancellationToken ct)
{
    using var response = await _client.GetAsync(
        url,
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    var totalBytes = response.Content.Headers.ContentLength ?? -1L;
    var canReportProgress = totalBytes > 0;

    await using var contentStream = await response.Content.ReadAsStreamAsync(ct);
    await using var fileStream = File.Create(outputPath);

    var buffer = new byte[81920];
    long totalBytesRead = 0;
    int bytesRead;

    while ((bytesRead = await contentStream.ReadAsync(buffer, ct)) > 0)
    {
        await fileStream.WriteAsync(buffer.AsMemory(0, bytesRead), ct);
        totalBytesRead += bytesRead;

        if (canReportProgress)
        {
            progress.Report((double)totalBytesRead / totalBytes);
        }
    }
}

💡 Pro Tip: Chunk Processing for APIs

Some APIs return newline-delimited JSON (NDJSON). Process line by line:

public async IAsyncEnumerable<LogEntry> StreamLogsAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    using var response = await _client.GetAsync(
        "/api/logs/stream",
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(ct);
    using var reader = new StreamReader(stream);

    while (!reader.EndOfStream)
    {
        ct.ThrowIfCancellationRequested();

        var line = await reader.ReadLineAsync(ct);
        if (string.IsNullOrWhiteSpace(line)) continue;

        var entry = JsonSerializer.Deserialize<LogEntry>(line);
        if (entry != null)
            yield return entry;
    }
}

Server-Sent Events (SSE)

SSE is HTTP streaming in action. Here's a client:

public async IAsyncEnumerable<ServerEvent> SubscribeToEventsAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    using var request = new HttpRequestMessage(HttpMethod.Get, "/api/events/stream");
    request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("text/event-stream"));

    using var response = await _client.SendAsync(
        request,
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(ct);
    using var reader = new StreamReader(stream);

    string? eventType = null;
    var dataBuilder = new StringBuilder();

    while (!reader.EndOfStream)
    {
        var line = await reader.ReadLineAsync(ct);

        if (string.IsNullOrEmpty(line))
        {
            // Empty line = event complete
            if (dataBuilder.Length > 0)
            {
                yield return new ServerEvent(eventType ?? "message", dataBuilder.ToString());
                eventType = null;
                dataBuilder.Clear();
            }
            continue;
        }

        if (line.StartsWith("event:"))
            eventType = line[6..].Trim();
        else if (line.StartsWith("data:"))
            dataBuilder.AppendLine(line[5..].Trim());
    }
}

public record ServerEvent(string Type, string Data);

Memory Comparison

Approach	100MB File Memory
GetStringAsync	~100MB + overhead
GetByteArrayAsync	~100MB
ResponseHeadersRead + Stream	~80KB (buffer only)

That's a 1000x+ reduction!

When to Stream

✅ Use streaming for:

• File downloads
• Large JSON arrays (1000+ items)
• Server-Sent Events
• WebSocket-like patterns
• Export endpoints
• Log tailing

❌ Don't bother for:

• Small API responses (< 1MB)
• Single objects
• When you need the whole payload anyway

Wrapping Up

Streaming is your secret weapon for handling large HTTP responses. The key insight: HttpCompletionOption.ResponseHeadersRead gives you control over when and how content downloads.

Your app's memory will thank you, and your users won't see mysterious OutOfMemoryException crashes!

Happy streaming! 🚀