My /robots.txt Was Paying Full Pipeline Price

I was reading request logs on a small service last week and caught something dumb. Every hit to /favicon.ico and /robots.txt, the junk requests browsers and crawlers fire without being asked, was walking the exact same middleware chain as a real API call. Auth, logging, a tenant lookup, rate limiting, the whole line. All of it just to hand back a 404 or three lines of text. The pipeline had no idea those routes were throwaway. Why would it?

I wanted to see what that actually costs, so I built a tiny version I could count.

Counting the pipeline

The trick is a shared counter that only moves when a request walks the chain. I wired up five small middlewares, nothing heavy, just the kind of per-request bookkeeping every real app carries, and had each one bump the number before calling the next. Then I used ASP.NET Core's short-circuit routing to let a couple of routes skip the whole thing.

using System.Diagnostics;

long middlewareRuns = 0;

var builder = WebApplication.CreateBuilder(args);
builder.Logging.ClearProviders();
builder.WebHost.UseUrls("http://127.0.0.1:5199");

var app = builder.Build();

// Routing has to be explicit: short-circuiting happens inside the routing
// middleware, so anything registered AFTER UseRouting is what gets skipped.
app.UseRouting();

string[] stages = ["header-check", "request-log", "tenant-lookup", "rate-limit", "correlation-id"];
foreach (var stage in stages)
{
    var name = stage;
    app.Use(async (ctx, next) =>
    {
        Interlocked.Increment(ref middlewareRuns);
        _ = $"{name}:{ctx.Request.Path}:{ctx.Request.Method}".Length; // a little real work
        await next(ctx);
    });
}

// Normal endpoint: every request walks the whole chain above.
app.MapGet("/data", () => Results.Ok(new { ok = true }));

// Noise endpoints answered right inside routing. The rest of the chain never runs.
app.MapShortCircuit(StatusCodes.Status404NotFound, "/favicon.ico", "/robots.txt");

// You can also short-circuit a route you own.
app.MapGet("/health", () => Results.Ok("healthy")).ShortCircuit();

await app.StartAsync();

using var http = new HttpClient { BaseAddress = new Uri("http://127.0.0.1:5199") };

async Task<(long runs, double ms)> Hammer(string path, int count)
{
    await http.GetAsync(path); // warm up
    var before = Interlocked.Read(ref middlewareRuns);
    var sw = Stopwatch.StartNew();
    for (int i = 0; i < count; i++)
        await http.GetAsync(path);
    sw.Stop();
    return (Interlocked.Read(ref middlewareRuns) - before, sw.Elapsed.TotalMilliseconds);
}

const int N = 20_000;
var data = await Hammer("/data", N);
var favicon = await Hammer("/favicon.ico", N);
var health = await Hammer("/health", N);

Console.WriteLine($"/data         (full pipeline)   runs: {data.runs,8:N0}  {data.ms,7:N0} ms");
Console.WriteLine($"/favicon.ico  (MapShortCircuit) runs: {favicon.runs,8:N0}  {favicon.ms,7:N0} ms");
Console.WriteLine($"/health       (.ShortCircuit()) runs: {health.runs,8:N0}  {health.ms,7:N0} ms");

await app.StopAsync();

The ordering is the part that's easy to get wrong. Short-circuiting happens inside the routing middleware, so only the middlewares registered after UseRouting get skipped. MapShortCircuit takes a status code and a list of paths and answers them right there, before the chain. .ShortCircuit() does the same thing to a route you actually own, like a health check.

What the counter said

Three routes, 20,000 requests each, from inside the same process. Here's the run, on .NET 10, Release build, in a small Linux container. Not a lab, just my harness and a loopback HttpClient.

/data         (full pipeline)   runs:  100,000    2,082 ms
/favicon.ico  (MapShortCircuit) runs:        0    1,227 ms
/health       (.ShortCircuit()) runs:        0      689 ms

Twenty thousand requests to /data, a hundred thousand middleware runs. Five per request, exactly as you'd expect. The two short-circuited routes? Zero. Not "fewer," zero. Two hundred thousand middleware executions across the two of them that simply never happened.

The wall-clock side backs it up without me having to squint. The full-pipeline batch sat around two seconds; the short-circuited routes came in at roughly half that, run after run. I won't pretend that's a precise benchmark. It's loopback traffic in a container and I care about the ratio, not the millisecond. But the ratio is real and it tracks the counter: skip the work, skip the time.

When I wouldn't reach for this

Here's the honest catch. Short-circuiting skips everything after routing, and "everything" includes the stuff you might actually want. If one of those middlewares writes the access logs you care about, or stamps a correlation id, a short-circuited /health won't get it. That's the whole point of the feature, but it bites if you forget. I'd short-circuit static noise like favicon and robots without a second thought. I'd think harder before doing it to an endpoint I might want to see in logs later.

One gotcha worth knowing: MapShortCircuit defaults to just completing the response if you don't pass a status code, so a bare call returns a 200 with an empty body. Pass the code you mean. I pass 404 for favicon because that's what a missing favicon should be anyway.

Honestly, I think the value here is less about the microseconds and more about the intent. Marking a route as "this does not deserve the pipeline" is decent documentation on its own, sitting right next to the route. The speed is a bonus you get for free.

Go look at what your favicon and robots requests are doing right now. I'd bet they're paying full fare. What's the most pointless thing you've caught walking your middleware chain?

— still counting middleware nobody asked to run