DEV Community: Nick

The N+1 Query Problem: Why Your LINQ Query Makes 1001 Database Calls

Nick — Fri, 25 Sep 2026 05:42:00 +0000

The N+1 Query Problem: Why Your LINQ Query Makes 1001 Database Calls

Your query returns 1000 orders. Your database log shows 1001 queries. One to fetch the orders, then one for each order's customer. This is the N+1 problem — and it's probably happening in your codebase right now.

The Innocent-Looking Code

var orders = dbContext.Orders.ToList();

foreach (var order in orders)
{
    Console.WriteLine($"Order {order.Id} by {order.Customer.Name}");
}

What you expect: one query fetching orders with customer names.

What happens:

SELECT * FROM Orders — fetches 1000 orders
SELECT * FROM Customers WHERE Id = 1 — for order 1
SELECT * FROM Customers WHERE Id = 2 — for order 2
... 998 more queries ...

If each query takes 5ms, you've turned a 10ms operation into a 5-second disaster.

Why It Happens

Entity Framework uses lazy loading by default (or used to — EF Core made it opt-in). When you access order.Customer, EF sees the navigation property isn't loaded and fires a query.

The insidious part: it works perfectly in development with 10 records. In production with 10,000? Your API times out.

The Solution: Eager Loading with Include

var orders = dbContext.Orders
    .Include(o => o.Customer)  // Load customers with orders
    .ToList();

foreach (var order in orders)
{
    Console.WriteLine($"Order {order.Id} by {order.Customer.Name}");
}

Now EF generates one query with a JOIN. Customer data comes with the order in a single round-trip.

Nested Includes

For deeper relationships:

var orders = dbContext.Orders
    .Include(o => o.Customer)
    .Include(o => o.OrderItems)
        .ThenInclude(oi => oi.Product)  // Nested: OrderItem's Product
    .ToList();

ThenInclude continues from the last included collection. You can chain them for complex graphs.

The Projection Alternative

Include loads entire entities. Often, you don't need all columns:

// Instead of Include...
var orders = dbContext.Orders
    .Include(o => o.Customer)
    .Include(o => o.OrderItems)
    .ToList();

// ...project to what you actually need
var orders = dbContext.Orders
    .Select(o => new 
    {
        o.Id,
        CustomerName = o.Customer.Name,
        ItemCount = o.OrderItems.Count,
        Total = o.OrderItems.Sum(oi => oi.Price)
    })
    .ToList();

The projection approach generates one query that fetches exactly the columns needed. No N+1, minimal data transfer.

Fun fact: The term "N+1" comes from the mathematical pattern: 1 query for the parent collection, plus N queries for each child. It was popularized by the Ruby on Rails community around 2006, but the problem existed anywhere ORMs did automatic lazy loading. The ActiveRecord pattern (from Martin Fowler's 2002 book) made it endemic.

AsSplitQuery: When Include Gets Too Wide

Wide includes (many navigation properties or deep nesting) produce massive JOINs. EF Core 5+ offers split queries:

var orders = dbContext.Orders
    .Include(o => o.OrderItems)
    .Include(o => o.ShippingAddress)
    .Include(o => o.BillingAddress)
    .AsSplitQuery()  // Separate queries instead of mega-JOIN
    .ToList();

AsSplitQuery fires multiple simple queries instead of one giant JOIN. For complex includes with many columns, split queries can actually be faster.

Trade-off: multiple round-trips vs one huge result set. Profile to decide.

Detecting N+1 in the Wild

Option 1: EF Core Logging

optionsBuilder.LogTo(Console.WriteLine, LogLevel.Information);

Watch the output. Multiple SELECT statements for navigation properties = N+1.

Option 2: Simple Query Count

int queryCount = 0;
optionsBuilder.LogTo(msg => 
{
    if (msg.Contains("Executing DbCommand")) queryCount++;
}, LogLevel.Information);

If queryCount >> 1 for what should be one query, investigate.

Option 3: Third-Party Tools

MiniProfiler, Glimpse, or similar tools visualize queries per request. N+1 shows as a fan of identical query patterns.

Common N+1 Patterns

The Loop Print

// Bad
foreach (var order in dbContext.Orders)
    Console.WriteLine(order.Customer.Name);  // N queries

// Good
foreach (var order in dbContext.Orders.Include(o => o.Customer))
    Console.WriteLine(order.Customer.Name);  // 1 query

The Innocent Serialize

// Bad — serializer triggers lazy loading
return Json(dbContext.Orders.ToList());  // Triggers N+1 during serialization

// Good
return Json(dbContext.Orders
    .Select(o => new { o.Id, CustomerName = o.Customer.Name })
    .ToList());

The Collection Count

// Bad
var orders = dbContext.Orders.ToList();
var withItems = orders.Where(o => o.OrderItems.Any());  // N queries

// Good
var orders = dbContext.Orders.Include(o => o.OrderItems).ToList();
var withItems = orders.Where(o => o.OrderItems.Any());  // 0 additional queries

The Rule

Accessing navigation in loops? Include before iterating
Don't need full entities? Project to DTOs (avoids N+1 and fetches less data)
Wide includes? Consider AsSplitQuery
Serialize entities? Project to anonymous/DTO first
Always profile — EF logging reveals the real query count

Next time, we'll look at async LINQ — when to use ToListAsync, why you can't make LINQ operators truly async, and the Task patterns that work with database queries. Hope to see you!

LINQ Joins: The Operator You Probably Don't Need

Nick — Thu, 24 Sep 2026 06:42:00 +0000

LINQ Joins: The Operator You Probably Don't Need

In SQL, you write JOINs constantly. In LINQ with Entity Framework, explicit joins are often a code smell.

Let me explain why — and when you actually need them.

Navigation Properties: The EF Way

If you have proper relationships configured:

public class Order
{
    public int Id { get; set; }
    public int CustomerId { get; set; }
    public Customer Customer { get; set; }  // Navigation property
}

public class Customer
{
    public int Id { get; set; }
    public string Name { get; set; }
    public List<Order> Orders { get; set; }  // Collection navigation
}

You don't need Join. Just navigate:

// EF automatically generates the JOIN
var orderDetails = dbContext.Orders
    .Where(o => o.Customer.Name.StartsWith("A"))  // Navigate through relationship
    .Select(o => new 
    {
        o.Id,
        CustomerName = o.Customer.Name  // EF handles the JOIN
    })
    .ToList();

SQL generated:

SELECT o.Id, c.Name as CustomerName
FROM Orders o
INNER JOIN Customers c ON o.CustomerId = c.Id
WHERE c.Name LIKE 'A%'

Clean. No explicit join syntax.

When You Actually Need Join

Explicit Join makes sense when:

No navigation property — raw tables without relationships
In-memory collections — joining two lists
Non-key relationships — joining on arbitrary columns

Scenario 1: No Relationship Configured

var orderWithCustomer = dbContext.Orders
    .Join(
        dbContext.Customers,
        order => order.CustomerId,      // Outer key
        customer => customer.Id,        // Inner key
        (order, customer) => new        // Result selector
        {
            OrderId = order.Id,
            CustomerName = customer.Name
        }
    )
    .ToList();

The syntax is verbose — four parameters. This is why navigation properties are preferred.

Scenario 2: In-Memory Collections

var products = GetProducts();
var prices = GetPriceUpdates();

var updated = products
    .Join(
        prices,
        p => p.Sku,
        u => u.Sku,
        (product, update) => new 
        {
            product.Name,
            OldPrice = product.Price,
            NewPrice = update.Price
        }
    )
    .ToList();

For in-memory joins, there's no EF to generate SQL. The Join operator uses hash-based matching — efficient even on large collections.

Fun fact: LINQ's Join uses a hash join algorithm internally. It builds a lookup table from the inner sequence, then probes it for each outer element. Time complexity: O(n + m) instead of O(n × m) for nested loops. The same optimization that makes database joins fast.

Left Join: The GroupJoin Pattern

LINQ's Join is an inner join — no match, no result. For left joins, you need GroupJoin:

var customersWithOrders = dbContext.Customers
    .GroupJoin(
        dbContext.Orders,
        customer => customer.Id,
        order => order.CustomerId,
        (customer, orders) => new 
        {
            customer.Name,
            OrderCount = orders.Count(),
            Orders = orders.ToList()
        }
    )
    .ToList();

GroupJoin gives you each customer with a collection of their orders (possibly empty).

For the classic LEFT JOIN behavior (one row per result, null if no match):

var leftJoin = dbContext.Customers
    .GroupJoin(
        dbContext.Orders,
        c => c.Id,
        o => o.CustomerId,
        (customer, orders) => new { customer, orders }
    )
    .SelectMany(
        x => x.orders.DefaultIfEmpty(),  // Flatten, keeping nulls
        (x, order) => new 
        {
            CustomerName = x.customer.Name,
            OrderId = order != null ? order.Id : (int?)null
        }
    )
    .ToList();

Ugly, right? This is why navigation properties with .Include() are the EF way.

The Query Syntax Alternative

Join looks cleaner in query syntax:

var result = 
    from order in dbContext.Orders
    join customer in dbContext.Customers 
        on order.CustomerId equals customer.Id
    select new { order.Id, customer.Name };

// Left join in query syntax
var leftJoin = 
    from customer in dbContext.Customers
    join order in dbContext.Orders 
        on customer.Id equals order.CustomerId into orders
    from order in orders.DefaultIfEmpty()
    select new { customer.Name, OrderId = order.Id };

The into keyword creates a grouped join. The second from with DefaultIfEmpty() flattens it into a left join.

Multiple Joins

Chaining joins gets messy fast:

var result = dbContext.Orders
    .Join(dbContext.Customers, o => o.CustomerId, c => c.Id, (o, c) => new { o, c })
    .Join(dbContext.Products, oc => oc.o.ProductId, p => p.Id, (oc, p) => new 
    {
        OrderId = oc.o.Id,
        CustomerName = oc.c.Name,
        ProductName = p.Name
    })
    .ToList();

Versus navigation properties:

var result = dbContext.Orders
    .Select(o => new 
    {
        o.Id,
        CustomerName = o.Customer.Name,
        ProductName = o.Product.Name
    })
    .ToList();

Night and day.

The Rule

Have navigation properties? Just navigate — EF handles joins
In-memory collections? Join is the right tool
Need left join in EF? Use GroupJoin + SelectMany, or include nullable navigation
Complex multi-table queries? Query syntax is more readable than chained Join calls

Next time, we'll explore the dangerous world of N+1 queries — the silent performance killer that makes your 1-second query take 30 seconds. Hope to see you!

LINQ GroupBy: The Operator Everyone Uses Wrong

Nick — Wed, 23 Sep 2026 07:41:00 +0000

LINQ GroupBy: The Operator Everyone Uses Wrong

GroupBy in LINQ looks like SQL GROUP BY. It isn't. At least, not in the way you'd expect.

When you grasp the difference, you stop fighting the operator and start wielding it.

The SQL Mental Model (That Misleads You)

In SQL, GROUP BY collapses rows into aggregates:

SELECT CategoryId, COUNT(*) as Count, AVG(Price) as AvgPrice
FROM Products
GROUP BY CategoryId

Result: one row per category with aggregate values. Clean.

So you write this in LINQ:

var groups = dbContext.Products
    .GroupBy(p => p.CategoryId)
    .ToList();

And you expect... what? Aggregates? No. You get a list of IGrouping<int, Product> objects. Each grouping has a Key (the CategoryId) and contains all the products in that category.

foreach (var group in groups)
{
    Console.WriteLine($"Category {group.Key}:");
    foreach (var product in group)  // Iterate individual products
    {
        Console.WriteLine($"  - {product.Name}");
    }
}

LINQ's GroupBy is more powerful than SQL's — it doesn't force you to aggregate. But that power confuses SQL-trained developers.

The Aggregation Pattern

To get SQL-like grouped aggregates, you combine GroupBy with Select:

var categoryStats = dbContext.Products
    .GroupBy(p => p.CategoryId)
    .Select(g => new 
    {
        CategoryId = g.Key,
        Count = g.Count(),
        AvgPrice = g.Average(p => p.Price),
        MaxPrice = g.Max(p => p.Price)
    })
    .ToList();

Now you get one row per category with computed values. This is what translates cleanly to SQL.

The Database vs In-Memory Split

Here's where it gets tricky. Not all GroupBy operations translate to SQL:

// This works in SQL
var simple = dbContext.Products
    .GroupBy(p => p.CategoryId)
    .Select(g => new { g.Key, Count = g.Count() })
    .ToList();

// This might NOT work in SQL (depending on EF version)
var complex = dbContext.Products
    .GroupBy(p => p.CategoryId)
    .Select(g => new 
    { 
        g.Key, 
        Products = g.ToList()  // Trying to materialize full objects in group
    })
    .ToList();

EF Core has historically struggled with "group then expand" patterns. The workaround is to group in memory:

var products = await dbContext.Products.ToListAsync();
var grouped = products
    .GroupBy(p => p.CategoryId)
    .Select(g => new { CategoryId = g.Key, Products = g.ToList() })
    .ToList();

Fun fact: SQL's GROUP BY was standardized in SQL-86 (yes, 1986). But the concept of grouping as collections — where each group is a container you can enumerate — comes from functional programming. LINQ merged both worlds, which is why it feels different from either pure SQL or pure functional code.

Grouping by Multiple Keys

Group by a composite key using anonymous types:

var grouped = dbContext.Products
    .GroupBy(p => new { p.CategoryId, p.SupplierId })
    .Select(g => new 
    {
        g.Key.CategoryId,
        g.Key.SupplierId,
        Count = g.Count()
    })
    .ToList();

The anonymous type becomes your grouping key. All properties must match for items to land in the same group.

GroupBy with Element Selector

You can transform elements during grouping:

// Group products, but only keep their names
var productNamesByCategory = products
    .GroupBy(
        p => p.CategoryId,           // Key selector
        p => p.Name                   // Element selector
    );

foreach (var group in productNamesByCategory)
{
    Console.WriteLine($"Category {group.Key}: {string.Join(", ", group)}");
}

The Lookup Alternative

When you'll access groups multiple times by key, ToLookup is more efficient:

var lookup = products.ToLookup(p => p.CategoryId);

// Direct key access — O(1) instead of iterating
var electronics = lookup[5];
var clothing = lookup[7];

ToLookup is like GroupBy but immediately materialized into a dictionary-like structure. Unlike Dictionary, it returns an empty collection for missing keys instead of throwing.

The "I Want This But Grouped" Pattern

Common scenario: you have a flat list, need it grouped for display:

var ordersByMonth = orders
    .GroupBy(o => new { o.OrderDate.Year, o.OrderDate.Month })
    .OrderByDescending(g => g.Key.Year)
    .ThenByDescending(g => g.Key.Month)
    .Select(g => new 
    {
        Period = $"{g.Key.Year}-{g.Key.Month:D2}",
        Orders = g.OrderBy(o => o.OrderDate).ToList(),
        Total = g.Sum(o => o.Amount)
    })
    .ToList();

The Rule

For aggregates only? GroupBy + Select with aggregation functions — translates to SQL
Need full objects per group? Materialize first, then GroupBy in memory
Multiple key access? Use ToLookup instead of GroupBy
Composite keys? Anonymous types with all grouping columns

Next time, we'll dive into Join operations — the LINQ equivalent of SQL JOINs, and why they're often unnecessary when you have proper navigation properties. Hope to see you!

Select Like a Surgeon: LINQ Projection That Doesn't Bleed Data

Nick — Tue, 22 Sep 2026 06:21:00 +0000

Select Like a Surgeon: LINQ Projection That Doesn't Bleed Data

Your entity has 47 columns. Your UI needs 3. You're loading all 47 every single time.

This is the silent performance killer in most .NET applications — and Select is the scalpel that fixes it.

The Problem With Loading Everything

var products = dbContext.Products
    .Where(p => p.CategoryId == 5)
    .ToList();

// In the view, you only use:
foreach (var p in products)
{
    Console.WriteLine($"{p.Name}: {p.Price}");
}

Entity Framework dutifully loads every column: Id, Name, Description, Price, Cost, Stock, SupplierId, CreatedAt, ModifiedAt, ImageBlob, Specifications... the full 47-column monstrosity.

You used two.

The Select Fix

var products = dbContext.Products
    .Where(p => p.CategoryId == 5)
    .Select(p => new { p.Name, p.Price })  // Only these two columns
    .ToList();

Generated SQL:

SELECT Name, Price FROM Products WHERE CategoryId = 5

Less data transferred. Faster query. Lower memory. Same result.

Anonymous Types vs DTOs

For quick projections, anonymous types work great:

var results = query.Select(p => new { p.Name, p.Price, p.Stock });

But you can't return anonymous types from methods. For that, create a DTO:

public record ProductSummary(string Name, decimal Price);

public List<ProductSummary> GetSummaries()
{
    return dbContext.Products
        .Select(p => new ProductSummary(p.Name, p.Price))
        .ToList();
}

Fun fact: The record keyword in C# 9+ generates Equals, GetHashCode, and ToString automatically — perfect for DTOs. Before records, we wrote 30 lines of boilerplate for what's now a single line.

Projection With Computed Values

Select isn't just for cherry-picking columns — you can compute:

var analysis = dbContext.Products
    .Select(p => new 
    {
        p.Name,
        Profit = p.Price - p.Cost,
        InStock = p.Stock > 0,
        DisplayPrice = "$" + p.Price.ToString("F2")  // Careful!
    })
    .ToList();

Wait — that DisplayPrice line. Will it run in SQL or C#?

The answer: EF Core will try to translate it. Simple string operations often work. Complex formatting might fail. When in doubt, project the raw value and format after materialization.

Nested Projections

Here's where it gets powerful. Related entities without loading full objects:

var orders = dbContext.Orders
    .Select(o => new 
    {
        o.OrderNumber,
        CustomerName = o.Customer.Name,  // Traverses relationship
        Items = o.OrderItems.Select(i => new 
        {
            i.Product.Name,
            i.Quantity,
            Total = i.Quantity * i.UnitPrice
        }).ToList()
    })
    .ToList();

This generates a single efficient query with JOINs — no N+1 problem, no loading full Customer or Product entities.

The Conditional Projection

Sometimes you need different shapes based on conditions:

var products = dbContext.Products
    .Select(p => new 
    {
        p.Name,
        p.Price,
        Status = p.Stock > 10 ? "Available" 
               : p.Stock > 0 ? "Low Stock" 
               : "Out of Stock"
    })
    .ToList();

This ternary logic translates to SQL CASE WHEN statements.

SelectMany: Flattening Nested Collections

When each source element has a collection, and you want a flat result:

// Each order has multiple items. Get all items across all orders:
var allItems = dbContext.Orders
    .SelectMany(o => o.OrderItems)  // Flatten
    .Select(i => new { i.ProductId, i.Quantity })
    .ToList();

SelectMany is the LINQ equivalent of SQL's implicit join that expands rows.

The Rule

Always project when you don't need all columns
Project early in the query chain — before ToList()
Use DTOs when returning from methods or crossing boundaries
Nest projections to avoid N+1 and reduce transferred data
Test complex expressions — not everything translates to SQL

Next time, we'll tackle the GroupBy operator — the one LINQ feature that's simultaneously powerful and confusing. We'll see how it differs between in-memory and database execution, and why your grouping query might not behave as expected. Hope to see you!

IQueryable vs IEnumerable: The Line Between C# and SQL

Nick — Mon, 21 Sep 2026 06:19:00 +0000

IQueryable vs IEnumerable: The Line Between C# and SQL

You're filtering a million records. With one interface, the database does the work. With the other, your application drowns in memory. Same LINQ syntax. Wildly different execution.

Let's demystify where C# ends and SQL begins.

The Two Worlds

IEnumerable<Product> inMemory = products.Where(p => p.Price > 100);
IQueryable<Product> inDatabase = dbContext.Products.Where(p => p.Price > 100);

Same Where. Same lambda. Completely different animals.

IEnumerable — evaluates in your app's memory using delegates
IQueryable — builds an expression tree, translated to SQL by the database provider

The Million-Row Mistake

Watch this common anti-pattern:

public IEnumerable<Product> GetExpensive()
{
    return dbContext.Products;  // Oops — cast to IEnumerable
}

// Somewhere else
var results = GetExpensive()
    .Where(p => p.Price > 100)  // This runs IN MEMORY
    .Take(10)
    .ToList();

The Where looks innocent. But because we returned IEnumerable, the database returns all products first, then C# filters them. On a million rows, your server weeps.

The fix:

public IQueryable<Product> GetExpensive()
{
    return dbContext.Products;  // Keep IQueryable
}

// Now filtering happens at the database
var results = GetExpensive()
    .Where(p => p.Price > 100)  // SQL: WHERE Price > 100
    .Take(10)                    // SQL: TOP 10
    .ToList();

Expression Trees: The Magic Behind IQueryable

When you write .Where(p => p.Price > 100) on an IQueryable, the lambda isn't compiled to IL code. Instead, it becomes a data structure — an expression tree:

GreaterThan
├── MemberAccess: p.Price  
└── Constant: 100

Entity Framework walks this tree and generates:

SELECT * FROM Products WHERE Price > 100

Fun fact: the LINQ expression tree system was so powerful that other languages adopted it. F# got quotations, and JavaScript transpilers like TypeScript use similar AST concepts. Your LINQ query is code describing code.

The Boundary: Where IQueryable Fails

Not everything translates to SQL:

// This breaks at runtime
var results = dbContext.Products
    .Where(p => MyCustomMethod(p.Name))  // Can't translate to SQL!
    .ToList();

IQueryable doesn't know what MyCustomMethod does — it can only translate standard LINQ operators and expressions the provider understands.

Two solutions:

// Option 1: Filter in the database first, then apply custom logic
var results = dbContext.Products
    .Where(p => p.Price > 100)        // Runs in SQL
    .AsEnumerable()                    // Switch to in-memory
    .Where(p => MyCustomMethod(p.Name)) // Runs in C#
    .ToList();

// Option 2: Project to anonymous type first
var results = dbContext.Products
    .Where(p => p.Price > 100)
    .Select(p => new { p.Name, p.Price })  // Only fetch needed columns
    .AsEnumerable()
    .Where(x => MyCustomMethod(x.Name))
    .ToList();

The Rule of Thumb

Keep IQueryable as long as possible — push filtering to the database
Use AsEnumerable() or ToList() only when you need C#-specific logic
Return IQueryable from repository methods when callers need to add conditions
Return IEnumerable (or materialized List) when results are final

Quick Test: Where Does This Run?

dbContext.Products
    .Where(p => p.Price > 100)        // ?
    .OrderBy(p => p.Name)             // ?
    .ToList()                          // ?
    .Where(p => p.Stock > 0);         // ?

First three: SQL. Last one: C# (after ToList() materializes).

Next time we'll explore the dark art of projection — turning database rows into exactly the shape you need while minimizing data transfer. Hope to see you!

The LINQ Query That Never Runs (Until You Force It)

Nick — Fri, 18 Sep 2026 09:19:22 +0000

The LINQ Query That Never Runs (Until You Force It)

You write a LINQ query. It compiles. It looks correct. But something is off — it executes at the weirdest times, or worse, executes multiple times when you expected once.

Welcome to deferred execution — LINQ's most misunderstood feature.

The Illusion of Immediate

Most developers assume this runs immediately:

var filtered = products.Where(p => p.Price > 100);

It doesn't. Not a single item is touched. The filtered variable holds a recipe, not a meal. The actual filtering happens only when you iterate — foreach, ToList(), Count(), First().

Here's where it gets interesting: LINQ remembers the source, not the snapshot.

var numbers = new List<int> { 1, 2, 3 };
var doubled = numbers.Select(n => n * 2);

numbers.Add(4);  // Modify the source

foreach (var n in doubled)
    Console.WriteLine(n);  // Prints 2, 4, 6, 8 — includes the new element!

The query re-evaluates against the current state of numbers. Every. Single. Time.

The Multiple Execution Trap

This pattern destroys performance in real codebases:

var expensive = GetProducts()
    .Where(p => p.Category == "Electronics")
    .Select(p => new { p.Name, p.Price });

// First enumeration — database hit
var count = expensive.Count();

// Second enumeration — ANOTHER database hit
var first = expensive.First();

// Third enumeration — you get the idea
foreach (var item in expensive) { }

Three database calls. Same query. Because expensive is a recipe, not results.

Force Immediate Execution

The fix is simple — materialize when you need stable data:

var products = GetProducts()
    .Where(p => p.Category == "Electronics")
    .ToList();  // Execute NOW, store results

var count = products.Count;     // Just counts the list
var first = products.First();   // Just reads from memory

ToList(), ToArray(), ToDictionary() — these force execution and store results.

Fun Fact

The term "deferred execution" comes from functional programming's "lazy evaluation" — a concept dating back to 1976 with SASL language. LINQ brought this academic concept into mainstream C# development in 2007, and developers have been confused by it ever since.

When Deferred Execution Shines

It's not a bug — it's a feature. Consider:

IEnumerable<Product> BuildQuery(bool includeInactive)
{
    var query = products.Where(p => p.Price > 0);

    if (!includeInactive)
        query = query.Where(p => p.IsActive);

    return query.OrderBy(p => p.Name);
}

You can compose queries conditionally before execution. The database sees one optimized query, not three separate ones.

The Rule

Building a query? Let it stay deferred — compose freely.
Passing results around? Materialize with ToList().
Multiple enumerations? Materialize first.
Single enumeration? Deferred is fine.

Next time, we'll look at how IQueryable takes deferred execution even further — translating C# into SQL you never wrote. Hope to see you!

DelegatingHandler Magic: Build a Pipeline for Your HTTP Calls

Nick — Thu, 10 Sep 2026 10:14:00 +0000

What's up, .NET devs! 👋

Today I want to show you one of the most underrated features in HttpClient — the DelegatingHandler pipeline. It's like middleware for your HTTP calls!

What's a DelegatingHandler?

Think of DelegatingHandler as a chain of interceptors. Every HTTP request passes through each handler, and every response comes back through the same chain (in reverse).

🔗 Fun Fact: The handler pipeline is modeled after the "Chain of Responsibility" design pattern from the Gang of Four book!

Request:  Client → Handler1 → Handler2 → Handler3 → Network
Response: Client ← Handler1 ← Handler2 ← Handler3 ← Network

Your First Handler: Logging

public class LoggingHandler : DelegatingHandler
{
    private readonly ILogger<LoggingHandler> _logger;

    public LoggingHandler(ILogger<LoggingHandler> logger)
    {
        _logger = logger;
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var stopwatch = Stopwatch.StartNew();

        _logger.LogInformation("→ {Method} {Uri}", request.Method, request.RequestUri);

        var response = await base.SendAsync(request, cancellationToken);

        stopwatch.Stop();

        _logger.LogInformation("← {StatusCode} in {ElapsedMs}ms",
            (int)response.StatusCode, stopwatch.ElapsedMilliseconds);

        return response;
    }
}

Register It

services.AddTransient<LoggingHandler>();

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddHttpMessageHandler<LoggingHandler>();

Correlation ID Propagation

In microservices, trace requests across services:

public class CorrelationIdHandler : DelegatingHandler
{
    private readonly IHttpContextAccessor _httpContextAccessor;

    public CorrelationIdHandler(IHttpContextAccessor httpContextAccessor)
    {
        _httpContextAccessor = httpContextAccessor;
    }

    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var correlationId = _httpContextAccessor.HttpContext?
            .Request.Headers["X-Correlation-ID"].FirstOrDefault()
            ?? Guid.NewGuid().ToString();

        request.Headers.Add("X-Correlation-ID", correlationId);
        return base.SendAsync(request, cancellationToken);
    }
}

Auth Token Injection

Never manually add auth headers again:

public class AuthTokenHandler : DelegatingHandler
{
    private readonly ITokenService _tokenService;

    public AuthTokenHandler(ITokenService tokenService)
    {
        _tokenService = tokenService;
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var token = await _tokenService.GetAccessTokenAsync(cancellationToken);
        request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
        return await base.SendAsync(request, cancellationToken);
    }
}

Metrics Collection

public class MetricsHandler : DelegatingHandler
{
    private readonly IMeterFactory _meterFactory;
    private readonly Histogram<double> _requestDuration;

    public MetricsHandler(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("HttpClient.Metrics");
        _requestDuration = meter.CreateHistogram<double>("http_request_duration_ms");
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var stopwatch = Stopwatch.StartNew();
        var tags = new TagList
        {
            { "method", request.Method.Method },
            { "host", request.RequestUri?.Host ?? "unknown" }
        };

        try
        {
            var response = await base.SendAsync(request, cancellationToken);
            tags.Add("status_code", ((int)response.StatusCode).ToString());
            return response;
        }
        finally
        {
            stopwatch.Stop();
            _requestDuration.Record(stopwatch.Elapsed.TotalMilliseconds, tags);
        }
    }
}

Chaining Multiple Handlers

services.AddTransient<LoggingHandler>();
services.AddTransient<CorrelationIdHandler>();
services.AddTransient<AuthTokenHandler>();
services.AddTransient<MetricsHandler>();

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddHttpMessageHandler<CorrelationIdHandler>()  // First in, last out
    .AddHttpMessageHandler<AuthTokenHandler>()
    .AddHttpMessageHandler<LoggingHandler>()
    .AddHttpMessageHandler<MetricsHandler>();       // Last in, first out

🎯 Pro Tip: Order Matters!

Handlers execute in the order you add them for requests, and reverse order for responses.

Logging should be first (sees raw request) or last (sees final request)
Auth should be after correlation ID
Retry policies (Polly) should wrap most other handlers

Wrapping Up

DelegatingHandler is your Swiss Army knife for HTTP concerns. Instead of polluting your business logic with logging, auth, and metrics — put it in handlers!

Happy pipelining! 🚀

HttpClient Observability: Logs, Metrics, and Traces Done Right

Nick — Wed, 09 Sep 2026 08:13:00 +0000

Hey observability enthusiasts! 👋

Your HttpClient is making calls, but can you see what's happening? Today we'll add proper logging, metrics, and distributed tracing to your HTTP calls!

The Built-in Logging

.NET's IHttpClientFactory includes logging out of the box. Just configure your log levels:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "System.Net.Http.HttpClient": "Information"
    }
  }
}

This gives you:

info: System.Net.Http.HttpClient.MyClient.LogicalHandler[100]
      Start processing HTTP request GET https://api.example.com/users
info: System.Net.Http.HttpClient.MyClient.ClientHandler[100]
      Sending HTTP request GET https://api.example.com/users
info: System.Net.Http.HttpClient.MyClient.ClientHandler[101]
      Received HTTP response headers after 145.2ms - 200

🔍 Fun Fact: HttpClient emits events at two levels: LogicalHandler (before/after your handlers) and ClientHandler (actual network call). This helps you see what your handlers add!

Custom Logging Handler

Want more control? Build a logging handler:

public class HttpLoggingHandler : DelegatingHandler
{
    private readonly ILogger<HttpLoggingHandler> _logger;

    public HttpLoggingHandler(ILogger<HttpLoggingHandler> logger)
    {
        _logger = logger;
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var requestId = Guid.NewGuid().ToString("N")[..8];
        var sw = Stopwatch.StartNew();

        _logger.LogInformation(
            "[{RequestId}] → {Method} {Uri}",
            requestId,
            request.Method,
            request.RequestUri);

        if (request.Content != null && _logger.IsEnabled(LogLevel.Debug))
        {
            var body = await request.Content.ReadAsStringAsync(cancellationToken);
            _logger.LogDebug("[{RequestId}] Request body: {Body}", requestId, body);
        }

        try
        {
            var response = await base.SendAsync(request, cancellationToken);
            sw.Stop();

            _logger.LogInformation(
                "[{RequestId}] ← {StatusCode} in {ElapsedMs}ms",
                requestId,
                (int)response.StatusCode,
                sw.ElapsedMilliseconds);

            return response;
        }
        catch (Exception ex)
        {
            sw.Stop();
            _logger.LogError(
                ex,
                "[{RequestId}] ✗ Failed after {ElapsedMs}ms: {Message}",
                requestId,
                sw.ElapsedMilliseconds,
                ex.Message);
            throw;
        }
    }
}

Metrics with .NET 8

.NET 8 introduced native metrics for HttpClient:

public class HttpMetricsHandler : DelegatingHandler
{
    private readonly IMeterFactory _meterFactory;
    private readonly Histogram<double> _duration;
    private readonly Counter<long> _requests;

    public HttpMetricsHandler(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("MyApp.HttpClient");

        _duration = meter.CreateHistogram<double>(
            "http.client.request.duration",
            unit: "ms",
            description: "HTTP request duration");

        _requests = meter.CreateCounter<long>(
            "http.client.request.count",
            description: "Total HTTP requests");
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var sw = Stopwatch.StartNew();
        var tags = new TagList
        {
            { "http.method", request.Method.Method },
            { "server.address", request.RequestUri?.Host ?? "unknown" }
        };

        try
        {
            var response = await base.SendAsync(request, cancellationToken);

            tags.Add("http.status_code", ((int)response.StatusCode).ToString());
            tags.Add("http.response.status_code", ((int)response.StatusCode).ToString());

            return response;
        }
        catch (TaskCanceledException)
        {
            tags.Add("error.type", "timeout");
            throw;
        }
        catch (HttpRequestException ex)
        {
            tags.Add("error.type", ex.GetType().Name);
            throw;
        }
        finally
        {
            sw.Stop();
            _duration.Record(sw.Elapsed.TotalMilliseconds, tags);
            _requests.Add(1, tags);
        }
    }
}

Export to Prometheus/OpenTelemetry

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation() // Built-in HttpClient metrics!
            .AddMeter("MyApp.HttpClient")   // Our custom metrics
            .AddPrometheusExporter();
    });

Distributed Tracing

Trace requests across services:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation(options =>
            {
                options.RecordException = true;
                options.EnrichWithHttpRequestMessage = (activity, request) =>
                {
                    activity.SetTag("http.request.body_size", 
                        request.Content?.Headers.ContentLength ?? 0);
                };
                options.EnrichWithHttpResponseMessage = (activity, response) =>
                {
                    activity.SetTag("http.response.body_size",
                        response.Content.Headers.ContentLength ?? 0);
                };
            })
            .AddJaegerExporter();
    });

💡 Pro Tip: EventSource for Deep Debugging

.NET's HttpClient emits detailed events via EventSource:

public class HttpEventListener : EventListener
{
    protected override void OnEventSourceCreated(EventSource eventSource)
    {
        if (eventSource.Name == "System.Net.Http" ||
            eventSource.Name == "System.Net.Sockets" ||
            eventSource.Name == "System.Net.Security")
        {
            EnableEvents(eventSource, EventLevel.Verbose);
        }
    }

    protected override void OnEventWritten(EventWrittenEventArgs eventData)
    {
        Console.WriteLine($"[{eventData.EventSource.Name}] {eventData.EventName}");

        if (eventData.Payload != null)
        {
            for (int i = 0; i < eventData.Payload.Count; i++)
            {
                Console.WriteLine($"  {eventData.PayloadNames?[i]}: {eventData.Payload[i]}");
            }
        }
    }
}

// Enable in your app
using var listener = new HttpEventListener();

This shows DNS resolution, socket connections, TLS handshakes — everything!

Health Checks

Monitor your downstream dependencies:

services.AddHealthChecks()
    .AddUrlGroup(
        new Uri("https://api.example.com/health"),
        name: "example-api",
        tags: new[] { "ready" })
    .AddUrlGroup(
        new Uri("https://api.payment.com/ping"),
        name: "payment-api",
        failureStatus: HealthStatus.Degraded,
        tags: new[] { "ready" });

Custom Health Check

public class ApiHealthCheck : IHealthCheck
{
    private readonly HttpClient _client;

    public ApiHealthCheck(IHttpClientFactory factory)
    {
        _client = factory.CreateClient("HealthCheck");
    }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            var sw = Stopwatch.StartNew();
            var response = await _client.GetAsync("/health", cancellationToken);
            sw.Stop();

            var data = new Dictionary<string, object>
            {
                { "responseTime", sw.ElapsedMilliseconds },
                { "statusCode", (int)response.StatusCode }
            };

            if (response.IsSuccessStatusCode)
            {
                return sw.ElapsedMilliseconds > 1000
                    ? HealthCheckResult.Degraded("Slow response", data: data)
                    : HealthCheckResult.Healthy("OK", data);
            }

            return HealthCheckResult.Unhealthy(
                $"Status: {response.StatusCode}", 
                data: data);
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy(ex.Message, ex);
        }
    }
}

Complete Setup

// Program.cs
builder.Services.AddTransient<HttpLoggingHandler>();
builder.Services.AddTransient<HttpMetricsHandler>();

builder.Services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddHttpMessageHandler<HttpLoggingHandler>()
    .AddHttpMessageHandler<HttpMetricsHandler>();

builder.Services.AddOpenTelemetry()
    .WithMetrics(m => m
        .AddHttpClientInstrumentation()
        .AddMeter("MyApp.HttpClient")
        .AddPrometheusExporter())
    .WithTracing(t => t
        .AddHttpClientInstrumentation()
        .AddJaegerExporter());

builder.Services.AddHealthChecks()
    .AddCheck<ApiHealthCheck>("downstream-api");

What to Monitor

Metric	Why
Request duration	Spot slowdowns early
Error rate	Know when things break
Status code distribution	Understand failure patterns
Request count	Capacity planning
Connection pool usage	Detect exhaustion

Wrapping Up

Observability isn't optional — it's how you know your app is healthy. With .NET's built-in instrumentation plus custom handlers, you can:

Debug production issues faster
Track SLOs and SLAs
Correlate requests across services
Catch problems before users do

Start with the built-in AddHttpClientInstrumentation() and add custom metrics as needed.

Happy observing! 👁️🚀

HTTP/2 and HTTP/3 in C#: The Speed You Didn't Know You Were Missing

Nick — Tue, 08 Sep 2026 07:12:00 +0000

Hey performance-minded developers! 👋

Are you still using HTTP/1.1 without realizing it? Let's unlock the modern protocols that can dramatically improve your app's network performance!

Quick Protocol Comparison

Feature	HTTP/1.1	HTTP/2	HTTP/3
Multiplexing	❌	✅	✅
Header Compression	❌	✅ (HPACK)	✅ (QPACK)
Connection	TCP	TCP	QUIC/UDP
Head-of-line Blocking	✅	Partial	❌
0-RTT	❌	❌	✅

🚀 Fun Fact: HTTP/3 uses QUIC, a protocol developed by Google that runs over UDP. It can establish connections up to 3x faster than TCP+TLS because it combines the transport and security handshakes into one round-trip!

Enabling HTTP/2

HTTP/2 is supported since .NET Core 3.0. Enable it:

var handler = new SocketsHttpHandler
{
    // Enable HTTP/2 explicitly
    EnableMultipleHttp2Connections = true
};

var client = new HttpClient(handler);

With HttpClientFactory

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        EnableMultipleHttp2Connections = true
    });

Force HTTP/2

By default, HttpClient negotiates the protocol via ALPN. To force HTTP/2:

var request = new HttpRequestMessage(HttpMethod.Get, "https://api.example.com/data")
{
    Version = HttpVersion.Version20,
    VersionPolicy = HttpVersionPolicy.RequestVersionExact
};

var response = await client.SendAsync(request);
Console.WriteLine($"Protocol: {response.Version}"); // 2.0

Client-Level Default

var client = new HttpClient()
{
    DefaultRequestVersion = HttpVersion.Version20,
    DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrLower
};

Enabling HTTP/3

HTTP/3 support came in .NET 6 (preview) and .NET 7 (stable).

// .NET 7+
var client = new HttpClient()
{
    DefaultRequestVersion = HttpVersion.Version30,
    DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrLower
};

Per-Request HTTP/3

var request = new HttpRequestMessage(HttpMethod.Get, "https://cloudflare.com")
{
    Version = HttpVersion.Version30,
    VersionPolicy = HttpVersionPolicy.RequestVersionOrHigher
};

var response = await client.SendAsync(request);
Console.WriteLine($"Protocol: HTTP/{response.Version}"); // Might be 3.0!

Check Server Support

public static async Task<string> GetBestProtocolAsync(HttpClient client, string url)
{
    // Try HTTP/3 first
    try
    {
        var request = new HttpRequestMessage(HttpMethod.Head, url)
        {
            Version = HttpVersion.Version30,
            VersionPolicy = HttpVersionPolicy.RequestVersionExact
        };
        var response = await client.SendAsync(request);
        return $"HTTP/{response.Version}";
    }
    catch (HttpRequestException)
    {
        // Fall back to HTTP/2
    }

    try
    {
        var request = new HttpRequestMessage(HttpMethod.Head, url)
        {
            Version = HttpVersion.Version20,
            VersionPolicy = HttpVersionPolicy.RequestVersionExact
        };
        var response = await client.SendAsync(request);
        return $"HTTP/{response.Version}";
    }
    catch
    {
        return "HTTP/1.1";
    }
}

Why HTTP/2 is Faster

Multiplexing

HTTP/1.1 can only process one request per connection at a time. To get parallelism, browsers open 6+ connections per domain.

HTTP/2 multiplexes unlimited requests over a single connection:

// These all share ONE TCP connection with HTTP/2
var tasks = new[]
{
    client.GetAsync("/api/users"),
    client.GetAsync("/api/products"),
    client.GetAsync("/api/orders"),
    client.GetAsync("/api/settings"),
    client.GetAsync("/api/notifications")
};

await Task.WhenAll(tasks);
// Much faster than sequential HTTP/1.1!

Header Compression

HTTP headers are verbose and repetitive. HTTP/2 uses HPACK compression:

HTTP/1.1 headers per request: ~800 bytes
HTTP/2 after compression: ~20-50 bytes (after first request)

That's 95% smaller headers!

💡 Pro Tip: Multiple HTTP/2 Connections

Sometimes you want multiple HTTP/2 connections (different auth contexts, load balancing):

var handler = new SocketsHttpHandler
{
    EnableMultipleHttp2Connections = true,
    MaxConnectionsPerServer = 10 // Allow up to 10 HTTP/2 connections
};

Performance Tuning

Connection Pooling

var handler = new SocketsHttpHandler
{
    // HTTP/2 settings
    EnableMultipleHttp2Connections = true,
    InitialHttp2StreamWindowSize = 65536 * 16, // 1MB window

    // Connection lifetime
    PooledConnectionLifetime = TimeSpan.FromMinutes(15),
    PooledConnectionIdleTimeout = TimeSpan.FromMinutes(2),

    // Keep connections warm
    KeepAlivePingPolicy = HttpKeepAlivePingPolicy.WithActiveRequests,
    KeepAlivePingDelay = TimeSpan.FromSeconds(60),
    KeepAlivePingTimeout = TimeSpan.FromSeconds(30)
};

GRPC (Built on HTTP/2)

If you're using gRPC, you're already on HTTP/2:

var channel = GrpcChannel.ForAddress("https://api.example.com", new GrpcChannelOptions
{
    HttpHandler = new SocketsHttpHandler
    {
        EnableMultipleHttp2Connections = true,
        KeepAlivePingDelay = TimeSpan.FromSeconds(60),
        KeepAlivePingTimeout = TimeSpan.FromSeconds(30)
    }
});

Debugging Protocol Version

public class ProtocolLoggingHandler : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        Console.WriteLine($"→ Request version: HTTP/{request.Version}");

        var response = await base.SendAsync(request, cancellationToken);

        Console.WriteLine($"← Response version: HTTP/{response.Version}");

        return response;
    }
}

Real-World Impact

I ran a benchmark fetching 100 resources from an API:

Protocol	Time	Connections
HTTP/1.1	4.2s	6
HTTP/2	1.1s	1
HTTP/3	0.9s	1

HTTP/2 was 4x faster with 6x fewer connections!

When to Use What

Protocol	Best For
HTTP/1.1	Legacy systems, simple requests
HTTP/2	Most modern APIs, microservices, gRPC
HTTP/3	Mobile apps, high-latency networks, CDNs

Wrapping Up

Upgrading to HTTP/2 is usually a one-liner change that can significantly improve performance — especially for apps making many parallel requests to the same host.

HTTP/3 is the future, with even better performance on unreliable networks. Start testing it now!

Check your protocol version with response.Version and make sure you're not leaving performance on the table.

Happy speeding! 🚀

Testing HttpClient the Right Way: Mock Handlers, Not Interfaces

Nick — Mon, 07 Sep 2026 01:11:00 +0000

Hey test-loving developers! 👋

I see this pattern ALL the time:

// ❌ The wrong way
public interface IHttpClientWrapper
{
    Task<T> GetAsync<T>(string url);
    Task<T> PostAsync<T>(string url, object data);
}

Stop wrapping HttpClient! There's a much better way to test HTTP calls. Let me show you!

The Problem with Wrapper Interfaces

You're testing your wrapper, not real behavior — Mocking IHttpClientWrapper.GetAsync<User>() doesn't test serialization, headers, or error handling.
You lose HttpClient features — Timeouts, handlers, resilience policies... all gone behind your abstraction.
It's unnecessary — HttpClient is already designed for testability!

The Right Way: Mock the Handler

HttpClient takes an HttpMessageHandler in its constructor. That's your test seam!

public class MockHttpMessageHandler : HttpMessageHandler
{
    private readonly Func<HttpRequestMessage, CancellationToken, Task<HttpResponseMessage>> _handler;

    public MockHttpMessageHandler(
        Func<HttpRequestMessage, CancellationToken, Task<HttpResponseMessage>> handler)
    {
        _handler = handler;
    }

    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, 
        CancellationToken cancellationToken)
    {
        return _handler(request, cancellationToken);
    }
}

🎯 Fun Fact: The handler pattern is why HttpClient implements IDisposable but you're told not to dispose it frequently. The handler does the real work and is what's expensive to create/destroy!

Your First Mock Test

[Fact]
public async Task GetUser_ReturnsUser_WhenApiSucceeds()
{
    // Arrange
    var expectedUser = new User { Id = 1, Name = "John" };

    var handler = new MockHttpMessageHandler((request, ct) =>
    {
        Assert.Equal(HttpMethod.Get, request.Method);
        Assert.Equal("/api/users/1", request.RequestUri?.PathAndQuery);

        var response = new HttpResponseMessage(HttpStatusCode.OK)
        {
            Content = JsonContent.Create(expectedUser)
        };
        return Task.FromResult(response);
    });

    var client = new HttpClient(handler)
    {
        BaseAddress = new Uri("https://api.example.com")
    };

    var userClient = new UserApiClient(client);

    // Act
    var user = await userClient.GetByIdAsync(1);

    // Assert
    Assert.NotNull(user);
    Assert.Equal(expectedUser.Id, user.Id);
    Assert.Equal(expectedUser.Name, user.Name);
}

Using Moq (More Flexible)

[Fact]
public async Task GetUser_Throws_WhenApiReturns500()
{
    // Arrange
    var handlerMock = new Mock<HttpMessageHandler>();

    handlerMock
        .Protected()
        .Setup<Task<HttpResponseMessage>>(
            "SendAsync",
            ItExpr.IsAny<HttpRequestMessage>(),
            ItExpr.IsAny<CancellationToken>())
        .ReturnsAsync(new HttpResponseMessage(HttpStatusCode.InternalServerError)
        {
            Content = new StringContent("Server error")
        });

    var client = new HttpClient(handlerMock.Object)
    {
        BaseAddress = new Uri("https://api.example.com")
    };

    var userClient = new UserApiClient(client);

    // Act & Assert
    await Assert.ThrowsAsync<HttpRequestException>(
        () => userClient.GetByIdAsync(1));
}

A Reusable Test Helper

public static class HttpClientTestHelper
{
    public static HttpClient CreateMockClient<TResponse>(
        TResponse response,
        HttpStatusCode statusCode = HttpStatusCode.OK,
        Action<HttpRequestMessage>? requestValidator = null)
    {
        var handler = new MockHttpMessageHandler((request, ct) =>
        {
            requestValidator?.Invoke(request);

            return Task.FromResult(new HttpResponseMessage(statusCode)
            {
                Content = JsonContent.Create(response)
            });
        });

        return new HttpClient(handler)
        {
            BaseAddress = new Uri("https://test.example.com")
        };
    }

    public static HttpClient CreateErrorClient(
        HttpStatusCode statusCode,
        string? errorMessage = null)
    {
        var handler = new MockHttpMessageHandler((request, ct) =>
        {
            return Task.FromResult(new HttpResponseMessage(statusCode)
            {
                Content = errorMessage != null 
                    ? new StringContent(errorMessage) 
                    : null
            });
        });

        return new HttpClient(handler)
        {
            BaseAddress = new Uri("https://test.example.com")
        };
    }
}

Clean Tests

[Fact]
public async Task GetUser_ReturnsUser()
{
    // Arrange
    var expected = new User { Id = 1, Name = "John" };
    var client = HttpClientTestHelper.CreateMockClient(expected);
    var userClient = new UserApiClient(client);

    // Act
    var user = await userClient.GetByIdAsync(1);

    // Assert
    Assert.Equal(expected.Id, user?.Id);
}

[Fact]
public async Task GetUser_ReturnsNull_WhenNotFound()
{
    // Arrange
    var client = HttpClientTestHelper.CreateErrorClient(HttpStatusCode.NotFound);
    var userClient = new UserApiClient(client);

    // Act
    var user = await userClient.GetByIdAsync(999);

    // Assert
    Assert.Null(user);
}

💡 Testing with WebApplicationFactory (Integration)

For integration tests, use the real HTTP stack:

public class UserApiIntegrationTests : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly HttpClient _client;

    public UserApiIntegrationTests(WebApplicationFactory<Program> factory)
    {
        _client = factory.CreateClient();
    }

    [Fact]
    public async Task GetUsers_ReturnsUsers()
    {
        // Act
        var users = await _client.GetFromJsonAsync<List<User>>("/api/users");

        // Assert
        Assert.NotNull(users);
        Assert.NotEmpty(users);
    }
}

Testing Resilience Policies

Want to verify your retry logic works?

[Fact]
public async Task Client_RetriesOnTransientFailure()
{
    // Arrange
    var callCount = 0;

    var handler = new MockHttpMessageHandler((request, ct) =>
    {
        callCount++;

        // Fail first 2 times, succeed on 3rd
        if (callCount < 3)
        {
            return Task.FromResult(
                new HttpResponseMessage(HttpStatusCode.ServiceUnavailable));
        }

        return Task.FromResult(new HttpResponseMessage(HttpStatusCode.OK)
        {
            Content = JsonContent.Create(new User { Id = 1 })
        });
    });

    var client = new HttpClient(handler)
    {
        BaseAddress = new Uri("https://api.example.com")
    };

    // Add resilience (in real code, use IHttpClientFactory)
    var userClient = new UserApiClient(client);

    // Act
    var user = await userClient.GetByIdAsync(1);

    // Assert
    Assert.Equal(3, callCount); // Verified it retried
    Assert.NotNull(user);
}

Testing Request Content

[Fact]
public async Task CreateUser_SendsCorrectPayload()
{
    // Arrange
    HttpRequestMessage? capturedRequest = null;

    var handler = new MockHttpMessageHandler(async (request, ct) =>
    {
        capturedRequest = request;
        return new HttpResponseMessage(HttpStatusCode.Created)
        {
            Content = JsonContent.Create(new User { Id = 1, Name = "John" })
        };
    });

    var client = new HttpClient(handler)
    {
        BaseAddress = new Uri("https://api.example.com")
    };

    var userClient = new UserApiClient(client);

    // Act
    var createRequest = new CreateUserRequest { Name = "John", Email = "john@test.com" };
    var user = await userClient.CreateAsync(createRequest);

    // Assert
    Assert.NotNull(capturedRequest);
    Assert.Equal(HttpMethod.Post, capturedRequest.Method);

    var body = await capturedRequest.Content!.ReadFromJsonAsync<CreateUserRequest>();
    Assert.Equal("John", body?.Name);
    Assert.Equal("john@test.com", body?.Email);
}

Cheat Sheet: What to Test

Test	Approach
Happy path	Mock handler returns expected data
Error handling	Mock handler returns error codes
Request format	Capture and inspect request
Retry logic	Count handler invocations
Timeout behavior	Mock handler with `Task.Delay`
Headers	Inspect `request.Headers` in mock
Integration	`WebApplicationFactory`

Wrapping Up

Stop creating wrapper interfaces around HttpClient. The built-in handler pattern is:

More realistic (tests actual serialization)
More flexible (test any HTTP behavior)
More maintainable (no fake abstraction layer)

Mock the handler, test the real client! 🎯

Happy testing! 🚀

Streaming Large HTTP Responses Without Killing Your Memory

Nick — Fri, 04 Sep 2026 16:10:00 +0000

Hey memory-conscious developers! 👋

Ever downloaded a 500MB file with HttpClient and watched your app's memory balloon? Today we'll fix that with streaming!

The Problem: Buffering Everything

By default, HttpClient buffers the entire response in memory:

// 😱 This loads EVERYTHING into memory!
var content = await client.GetStringAsync("https://example.com/huge-file.json");

For a 500MB response, that's 500MB of RAM. For 10 concurrent downloads? 5GB!

🧠 Fun Fact: The default HttpClient.MaxResponseContentBufferSize is 2GB on 64-bit systems. You can hit OutOfMemoryException before you even realize what's happening!

The Solution: HttpCompletionOption.ResponseHeadersRead

This magic option tells HttpClient to return as soon as headers arrive — before the body downloads:

using var response = await client.GetAsync(
    "https://example.com/huge-file.json",
    HttpCompletionOption.ResponseHeadersRead);

response.EnsureSuccessStatusCode();

// Now stream the content
await using var stream = await response.Content.ReadAsStreamAsync();

Streaming to a File

public async Task DownloadFileAsync(string url, string outputPath, CancellationToken ct)
{
    using var response = await _client.GetAsync(
        url, 
        HttpCompletionOption.ResponseHeadersRead, 
        ct);

    response.EnsureSuccessStatusCode();

    await using var contentStream = await response.Content.ReadAsStreamAsync(ct);
    await using var fileStream = new FileStream(
        outputPath, 
        FileMode.Create, 
        FileAccess.Write, 
        FileShare.None, 
        bufferSize: 81920, // 80KB buffer
        useAsync: true);

    await contentStream.CopyToAsync(fileStream, ct);
}

Streaming JSON with System.Text.Json

For large JSON arrays, you can deserialize while streaming:

public async IAsyncEnumerable<User> StreamUsersAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    using var response = await _client.GetAsync(
        "/api/users/export",
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(ct);

    await foreach (var user in JsonSerializer.DeserializeAsyncEnumerable<User>(
        stream, 
        new JsonSerializerOptions { PropertyNameCaseInsensitive = true },
        ct))
    {
        if (user != null)
            yield return user;
    }
}

Consuming Streamed Data

await foreach (var user in userClient.StreamUsersAsync(cancellationToken))
{
    Console.WriteLine($"Processing: {user.Name}");
    // Process one at a time - memory stays flat!
}

Progress Reporting

Want a download progress bar? Here's how:

public async Task DownloadWithProgressAsync(
    string url, 
    string outputPath,
    IProgress<double> progress,
    CancellationToken ct)
{
    using var response = await _client.GetAsync(
        url,
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    var totalBytes = response.Content.Headers.ContentLength ?? -1L;
    var canReportProgress = totalBytes > 0;

    await using var contentStream = await response.Content.ReadAsStreamAsync(ct);
    await using var fileStream = File.Create(outputPath);

    var buffer = new byte[81920];
    long totalBytesRead = 0;
    int bytesRead;

    while ((bytesRead = await contentStream.ReadAsync(buffer, ct)) > 0)
    {
        await fileStream.WriteAsync(buffer.AsMemory(0, bytesRead), ct);
        totalBytesRead += bytesRead;

        if (canReportProgress)
        {
            progress.Report((double)totalBytesRead / totalBytes);
        }
    }
}

💡 Pro Tip: Chunk Processing for APIs

Some APIs return newline-delimited JSON (NDJSON). Process line by line:

public async IAsyncEnumerable<LogEntry> StreamLogsAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    using var response = await _client.GetAsync(
        "/api/logs/stream",
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(ct);
    using var reader = new StreamReader(stream);

    while (!reader.EndOfStream)
    {
        ct.ThrowIfCancellationRequested();

        var line = await reader.ReadLineAsync(ct);
        if (string.IsNullOrWhiteSpace(line)) continue;

        var entry = JsonSerializer.Deserialize<LogEntry>(line);
        if (entry != null)
            yield return entry;
    }
}

Server-Sent Events (SSE)

SSE is HTTP streaming in action. Here's a client:

public async IAsyncEnumerable<ServerEvent> SubscribeToEventsAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    using var request = new HttpRequestMessage(HttpMethod.Get, "/api/events/stream");
    request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("text/event-stream"));

    using var response = await _client.SendAsync(
        request,
        HttpCompletionOption.ResponseHeadersRead,
        ct);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(ct);
    using var reader = new StreamReader(stream);

    string? eventType = null;
    var dataBuilder = new StringBuilder();

    while (!reader.EndOfStream)
    {
        var line = await reader.ReadLineAsync(ct);

        if (string.IsNullOrEmpty(line))
        {
            // Empty line = event complete
            if (dataBuilder.Length > 0)
            {
                yield return new ServerEvent(eventType ?? "message", dataBuilder.ToString());
                eventType = null;
                dataBuilder.Clear();
            }
            continue;
        }

        if (line.StartsWith("event:"))
            eventType = line[6..].Trim();
        else if (line.StartsWith("data:"))
            dataBuilder.AppendLine(line[5..].Trim());
    }
}

public record ServerEvent(string Type, string Data);

Memory Comparison

Approach	100MB File Memory
`GetStringAsync`	~100MB + overhead
`GetByteArrayAsync`	~100MB
`ResponseHeadersRead` + Stream	~80KB (buffer only)

That's a 1000x+ reduction!

When to Stream

✅ Use streaming for:

File downloads
Large JSON arrays (1000+ items)
Server-Sent Events
WebSocket-like patterns
Export endpoints
Log tailing

❌ Don't bother for:

Small API responses (< 1MB)
Single objects
When you need the whole payload anyway

Wrapping Up

Streaming is your secret weapon for handling large HTTP responses. The key insight: HttpCompletionOption.ResponseHeadersRead gives you control over when and how content downloads.

Your app's memory will thank you, and your users won't see mysterious OutOfMemoryException crashes!

Happy streaming! 🚀

Building Bulletproof HTTP Calls with Polly in C#

Nick — Thu, 03 Sep 2026 13:09:00 +0000

Hey friends! 👋

Ever had your app crash because a third-party API had a hiccup? Or watched users get errors because of a temporary network blip? Today we're going to make your HTTP calls bulletproof with resilience patterns!

The Problem: Networks Are Unreliable

The internet is held together by hope and duct tape. Services go down, networks get congested, and that API you depend on? It's going to fail eventually.

🎲 Fun Fact: Amazon found that even 100ms of latency costs them 1% in sales. Netflix reports that 0.1% of their requests fail due to transient errors. If the big players deal with this, so will you!

Enter Polly (and the New .NET 8+ Standard)

For years, Polly has been the go-to library for resilience in .NET. But with .NET 8, Microsoft introduced Microsoft.Extensions.Http.Resilience — a first-party solution built on Polly v8!

Option 1: Microsoft.Extensions.Http.Resilience (.NET 8+)

The new standard way — minimal config, sensible defaults:

// Install: dotnet add package Microsoft.Extensions.Http.Resilience

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddStandardResilienceHandler();

That's it! You get:

⏱️ Timeout: 30 seconds total
🔄 Retry: 3 attempts with exponential backoff
⚡ Circuit Breaker: Opens after 10% failure rate
🎚️ Rate Limiter: Prevents overwhelming the target

Customize It

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.MaxRetryAttempts = 5;
        options.Retry.Delay = TimeSpan.FromMilliseconds(500);
        options.CircuitBreaker.FailureRatio = 0.1; // 10% failures trips it
        options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(10);
    });

Option 2: Raw Polly (Full Control)

Need more control? Use Polly directly:

// Install: dotnet add package Microsoft.Extensions.Http.Polly

var retryPolicy = Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .OrResult(r => !r.IsSuccessStatusCode)
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)),
        onRetry: (outcome, delay, attempt, ctx) =>
        {
            Console.WriteLine($"Retry {attempt} after {delay.TotalSeconds}s");
        });

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddPolicyHandler(retryPolicy);

The Circuit Breaker Pattern

This is my favorite pattern. Think of it like an electrical circuit breaker:

Closed (normal): Requests flow through
Open (tripped): Requests fail immediately — don't waste time on a dead service
Half-Open (testing): Let one request through to check if service recovered

var circuitBreaker = Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .OrResult(r => r.StatusCode == HttpStatusCode.ServiceUnavailable)
    .CircuitBreakerAsync(
        handledEventsAllowedBeforeBreaking: 5,
        durationOfBreak: TimeSpan.FromSeconds(30),
        onBreak: (result, duration) =>
        {
            Console.WriteLine($"Circuit OPEN for {duration.TotalSeconds}s!");
        },
        onReset: () => Console.WriteLine("Circuit CLOSED - back to normal"),
        onHalfOpen: () => Console.WriteLine("Circuit HALF-OPEN - testing..."));

💡 Pro Tips

1. Add Jitter to Retries

Without jitter, all your retrying clients hit the recovering server at the same moment. Add randomness:

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.UseJitter = true; // Adds randomness to retry delays
    });

2. Be Idempotency-Aware

Only retry operations that are safe to repeat! A GET is fine, but retrying a POST that creates an order? You might create duplicates.

// Good: Retry GETs
var response = await _client.GetAsync("/api/orders/123");

// Careful: Don't blindly retry POSTs
// Use idempotency keys or check if operation already completed

3. Timeout Correctly

Set both per-attempt AND total timeouts:

services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddStandardResilienceHandler(options =>
    {
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(5);  // Per attempt
        options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(30); // Overall
    });

Real-World Example

services.AddHttpClient<IPaymentGateway, PaymentGateway>(client =>
{
    client.BaseAddress = new Uri("https://api.stripe.com/");
})
.AddStandardResilienceHandler(options =>
{
    // Payment APIs need quick failures
    options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(5);
    options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(15);

    // Retry on transient errors only
    options.Retry.MaxRetryAttempts = 2;
    options.Retry.ShouldHandle = args => ValueTask.FromResult(
        args.Outcome.Exception is HttpRequestException ||
        args.Outcome.Result?.StatusCode == HttpStatusCode.TooManyRequests);
});

Wrapping Up

Stop hoping your HTTP calls will work. Plan for failure, and your app will gracefully handle the chaos of distributed systems.

Start with AddStandardResilienceHandler() in .NET 8+ — it's an excellent default. Then customize as you learn your services' failure patterns.

Your users will never know how many retries and circuit breaker trips happen behind the scenes. They'll just see a reliable app! 🎉

Happy resilient coding! 🚀