DEV Community: M Nouman Berlas

Parallel Processing Gone Wrong

M Nouman Berlas — Thu, 09 Jul 2026 09:34:00 +0000

⚡ "Let's Make It Faster!" (Famous Last Words)

The Scene: Sprint planning. The feature? Bulk data processing.
Me (naive): "Let's just run everything in parallel! It'll be 10x faster!"
Production (3 days later): catches fire 🔥
What I Thought Would Happen:
• 100 items to process
• Run them all in parallel
• Finish in 1/10th the time
• Get promoted
• Become CTO
What Actually Happened:
• 100 concurrent database connections
• Database connection pool exhausted
• ALL requests start failing (not just ours)
• 2 AM phone call from the CTO (not a promotion call)
• Emergency rollback at 3 AM
• Very awkward Slack conversation
The Broken Code:

// "I'll just parallelize everything!"
foreach (var item in items) {
    Task.Run(() => ProcessItem(item)); // YOLO
}

Why This Implodes:
❌ No limit on concurrent operations
❌ No shared state protection
❌ No error handling per item
❌ Overwhelms downstream services
❌ Database can't handle the connections
❌ Network ports get exhausted
❌ And my ego gets crushed
The Fix That Saved My Job:

int passCount = 0;
int failCount = 0;
var errors = new ConcurrentDictionary<string, string>();

// Key: MaxDegreeOfParallelism controls concurrency
Parallel.For(0, items.Count, 
    new ParallelOptions { 
        MaxDegreeOfParallelism = Environment.ProcessorCount // Usually 4-8
    }, 
    (i) => {
        try {
            ProcessItem(items[i]);

            // Thread-safe counter increment
            Interlocked.Increment(ref passCount);
        } 
        catch (Exception ex) {
            Interlocked.Increment(ref failCount);

            // Thread-safe error collection
            errors.TryAdd(i.ToString(), ex.Message);
        }
    });

// Now you can safely check results
Console.WriteLine($"Success: {passCount}, Failed: {failCount}");

The Golden Rules I Learned:
1️⃣ Parallelism ≠ Performance - Too much parallelism kills performance
2️⃣ Know Your Bottleneck - If it's I/O bound, 4-8 threads is usually optimal
3️⃣ Protect Shared State - Use Interlocked or ConcurrentCollections
4️⃣ Configure, Don't Hardcode - Make MaxDegreeOfParallelism environment-specific
5️⃣ Test Under Load - It works on my machine ≠ it works in production
The Results After Proper Implementation:
📊 Controlled concurrency = predictable performance
✅ Database connections stay healthy
⚡ 6x speedup (not 100x, but sustainable)
💰 Zero production incidents
😊 Sleep peacefully without 3 AM alerts
The Hard Truth:
More threads ≠ More speed
It's like adding more cooks to a kitchen. At some point, they just bump into each other.
What's your parallel processing horror story? I can't be the only one who learned this the hard way 😅

dotnet #parallelprocessing #performanceoptimization #productionissues #softwareengineering #threading #scalability #lessonslearned #debugging #csharp #techleadership

P.S. If you're using Task.Run() in a loop or unbounded Parallel.ForEach(), you're one production deployment away from a very bad day.

The Middleware Bug That Took Down Our API (Twice)

M Nouman Berlas — Tue, 07 Jul 2026 15:52:33 +0000

Picture this: Your API is humming along perfectly. 99.9% success rate. Life is good.
Then suddenly - BAM! 🔥
Random 500 errors. Garbled responses. Users complaining about "weird characters" in their data.
And it only happens under load.
The Culprit? Our "innocent" logging middleware.
Here's what we did WRONG:

//
public async Task InvokeAsync(HttpContext context, RequestDelegate next) {
    await next(context); // Call next middleware

    // Try to log the response
    var body = await context.Response.Body.ReadAsStringAsync();
    Logger.Log(body); // Oops...
}
//

Why This Fails Spectacularly:
1.  The response stream is already consumed and sent to the client
2.  Multiple requests try to read/modify streams concurrently
3.  Streams aren't thread-safe by default
4.  Gzip-compressed responses come back as garbage characters
5.  Everything breaks in weird, unpredictable ways
The Fix That Actually Works:

`
public async Task InvokeAsync(HttpContext context, RequestDelegate next) {
    // Step 1: Enable request buffering
    context.Request.EnableBuffering();

    // Step 2: Swap response stream with a buffered one
    var originalBodyStream = context.Response.Body;
    using var memoryStream = new MemoryStream();
    context.Response.Body = memoryStream;

    try {
        // Step 3: Let the pipeline execute
        await next(context);

        // Step 4: Safely read the buffered response
        memoryStream.Position = 0;

        // Step 5: Handle compression properly
        string responseBody;
        if (context.Response.Headers.ContentEncoding.Contains("gzip")) {
            using var gzipStream = new GZipStream(memoryStream, CompressionMode.Decompress);
            using var reader = new StreamReader(gzipStream, Encoding.UTF8);
            responseBody = await reader.ReadToEndAsync();
        } else {
            using var reader = new StreamReader(memoryStream, Encoding.UTF8);
            responseBody = await reader.ReadToEndAsync();
        }

        // Step 6: Log it
        Logger.Log(responseBody);

        // Step 7: Copy back to original stream
        memoryStream.Position = 0;
        await memoryStream.CopyToAsync(originalBodyStream);
    } finally {
        // Step 8: Restore original stream
        context.Response.Body = originalBodyStream;
    }
}

The Hidden Gotcha Nobody Tells You:
Modern APIs use gzip/deflate compression by default. If you don't decompress before reading, you get binary garbage that looks like: �H��W�K�M�U��Z

Our Results:
✅ Zero stream-related exceptions
✅ Zero "garbage character" bug reports
✅ Proper logging at 10,000+ req/min
✅ My on-call rotation got way quieter 😴
The Brutal Truth:
Middleware is easy to write. Thread-safe, production-ready middleware? That's an art form.

Who else has spent hours debugging middleware issues? Let's commiserate in the comments 👇

dotnet #aspnetcore #middleware #threadsafety #debugging #productionissues #softwareengineering #webdevelopment #csharp #lessonslearned #cloudnative

P.S. Always test middleware under load. Your local machine running 1 request at a time will NEVER expose these bugs.

HttpClient Socket Exhaustion

M Nouman Berlas — Sat, 27 Jun 2026 13:00:21 +0000

🔥 Your .NET API is Slowly Dying (And You Don't Even Know It)

Last month, our production system started timing out randomly. No errors. No warnings. Just... slow death.
After 3 sleepless nights and diving through 60+ microservices, I found the culprit:

var client = new HttpClient(); // This single line
Here's what was happening:
Every HTTP call was creating a NEW TCP socket. Under load, we exhausted the system's connection pool. Requests started queuing. Response times went from 200ms to 30+ seconds.

The Silent Killer:
• ✅ Your code compiles fine
• ✅ Your tests pass
• ✅ It works in dev/staging
• ❌ It crashes in production under real load

The Fix That Saved Our SLA:

// In Startup.cs
services.AddHttpClient("MyService", client => { client.Timeout = TimeSpan.FromSeconds(120); })
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler { PooledConnectionLifetime = TimeSpan.FromMinutes(2), PooledConnectionIdleTimeout = TimeSpan.FromMinutes(5), MaxConnectionsPerServer = 20, AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate });

// In your service public class MyService { private readonly IHttpClientFactory _clientFactory; public MyService(IHttpClientFactory clientFactory) { _clientFactory = clientFactory; } public async Task<string> GetDataAsync() { var client = _clientFactory.CreateClient("MyService"); // Use the client } }

The Results:
📊 95% reduction in connection failures
⚡ 40% improvement in throughput
💰 Prevented potential revenue loss from timeouts

The Lesson:
IHttpClientFactory isn't just a "best practice" - it's a production survival tool.
Microsoft documented this in 2018, but I still see this anti-pattern EVERYWHERE.
Have you been bitten by socket exhaustion?

Share your battle story in comments 👇

#performance #dotnet #csharp #softwareengineering #productionissues #microservices #debugging #azure #performanceoptimization #lessonslearned #techleadership

Battle Stories Series

M Nouman Berlas — Sat, 27 Jun 2026 12:40:11 +0000

I will be writing about the real world instances that I have witnessed during my career from production issues. For almost all of the I have to spend the night figuring out the real cause and end up having to go through the team post mortem.
Love to hear about your production experiences as well. The first story to be followed soon :)