Amitesh0512

Posted on Nov 12 Originally published at amiteshsurwar.com

Optimizing Batch Scheduling with Async Pipelines in .NET

#performancetuning #net #aiarchitecture #scalability

Quick Answer

Learn how to boost throughput and cut latency by Optimizing Batch Scheduling with Async Pipelines in .NET—complete architecture, code, and production tips for modern AI workloads.

Naïve Request Loop Wastes GPU Capacity

When a .NET service fronts a high‑cost inference engine—GPU, Azure OpenAI, or a custom accelerator—every millisecond a request sits idle is a dollar lost. In production we often see 20–30 % of the GPU capacity wasted because batches are formed too sLowly or too large, pushing the 99th‑percentile latency beyond SLA limits. The root of the problem is a naïve request‑per‑task loop that ties a thread to every incoming HTTP call and lets the GPU sit half‑filled.

Why async pipelines matter

Async pipelines decouple the request ingestion thread‑pool from the compute thread that drives the accelerator. By using lock‑free, bounded channels we can back‑pressure producers, keep the pool shallow, and expose fine‑grained knobs for size‑vs‑time batching.

Real‑World Example

Consider a fintech platform that handles 12 k requests per second (RPS) of short chat‑style LLM calls. The service runs on a single A100 GPU (40 GB) behind an ASP.NET Core API. The initial implementation used a synchronous loop:

foreach (var req in incoming) {
    var res = await CallModelAsync(req); // blocks a thread per request
    await RespondAsync(res);
}

At peak traffic the thread‑pool saturated, queue latency rose to 350 ms, and the GPU was only 55 % utilized. Switching to an async pipeline with a bounded channel and a 64‑item batch size increased GPU utilization to 88 % and cut the 99th‑percentile latency to 118 ms.

Trade‑Offs

Batch Size vs Latency: Larger batches squeeze more tokens per GPU launch but increase tail latency. A 64‑item batch on a 40 GB A100 balances throughput and 99th‑percentile under 120 ms in the example.
Size‑Based vs Time‑Based Triggers: Size‑based is deterministic but can stall under low traffic. Time‑based (e.g., 20 ms max wait) guarantees a latency ceiling but can leave the GPU partially idle.
Channel Capacity vs Memory Footprint: A 10 k bounded channel keeps the pool shallow but can still consume ~2 MB for small request objects. Oversizing leads to memory pressure and potential OOM.
Concurrent Batches vs GPU Memory: Running two batches in parallel on an A100 uses ~80 % of memory. Adding a third batch would trigger evictions or crashes.
Priority Handling vs Complexity: Splitting high‑ and low‑priority queues adds code but can reduce SLA violations for premium customers.

Metric‑Based Pipeline Decision Matrix

Use the following matrix when designing your pipeline.

Metric	Decision Point	Action
GPU Utilization < 80 %	Increase batch size or add concurrent batches (if memory allows)	Raise `BatchSize` by 16 or bump `MaxConcurrentBatches` by 1
99th‑Percentile > 120 ms	Reduce batch size or tighten max wait	Decrease `BatchSize` by 8 or set `MaxWait` to 15 ms
Channel Back‑Pressure < 1 k items	Capacity is too low; risk of request loss	Increase capacity to 20 k and enable `BoundedChannelFullMode.Wait`
Memory < 1 GB used by channel	Capacity is high; consider trimming	Reduce capacity to 10 k or use a distributed queue
CPU < 30 %	Pipeline is efficient; focus on GPU	Monitor for spikes; adjust `MaxConcurrentBatches`

When This Fails in Production

Thread‑Starvation: If any stage blocks synchronously (e.g., .Result on a Task), the channel’s reader can deadlock. The pipeline stalls, queue grows, and latency explodes.
Memory Blow‑Up: A bounded channel with a capacity of 1 M items can consume >10 GB before back‑pressure kicks in, leading to OOM and process termination.
GPU Oversubscription: Running too many concurrent batches can exhaust memory and cause the GPU driver to evict tensors, throwing CudaOutOfMemoryException or causing silent throttling.
Back‑Pressure Not Propagating: If the producer uses a non‑bounded queue (e.g., ConcurrentQueue) the channel never signals back‑pressure, and upstream services keep flooding the pipeline.

Common Mistakes Engineers Make

Using Task.WhenAll on all queued requests instead of batching, leading to OOM.
Ignoring priority fields and treating every request the same, causing SLA violations for premium tiers.
Hard‑coding batch sizes without observing GPU utilization or latency trends.
Re‑throwing exceptions from inside the batch loop without isolating the bad item.
Not instrumenting batch boundaries, making it impossible to correlate spikes with specific pipeline stages.

Better Approach Based on Experience

In a production deployment I adopted a hybrid, adaptive batching strategy:

Adaptive Size: Start with a base size (e.g., 64). If GPU utilization > 90 % and 99th‑percentile < 100 ms, increment size by 8. If latency rises, decrement.
Dynamic Max Wait: Base max wait at 15 ms. If traffic drops below 1 k RPS, reduce to 10 ms to keep latency low.
Priority Queues: Two bounded channels (high/low). The batch builder selects from the high channel first, but if it’s empty it falls back to low. This keeps premium traffic within SLA without a separate pipeline.
Semaphore‑Controlled Concurrency: Use SemaphoreSlim to cap MaxConcurrentBatches dynamically based on GPU memory metrics exposed via NVML.
Observability Hooks: Emit a trace event every time a batch is formed, processed, and results dispatched. Correlate with Azure Monitor or Prometheus metrics.
Retry Isolation: Wrap the entire batch call in a Polly policy, but re‑queue only the failed items with a retry counter. This keeps retries from blocking other batches.

Code Skeleton (Production‑Ready)

public class AdaptiveBatchScheduler<TRequest, TResponse>
{
    private readonly Channel<TRequest> _high;
    private readonly Channel<TRequest> _low;
    private readonly SemaphoreSlim _concurrency;
    private int _batchSize = 64;
    private TimeSpan _maxWait = TimeSpan.FromMilliseconds(15);
    private readonly Func<IReadOnlyList<TRequest>, Task<IReadOnlyList<TResponse>>> _process;

    public AdaptiveBatchScheduler(Func<IReadOnlyList<TRequest>, Task<IReadOnlyList<TResponse>>> process,
        int maxConcurrentBatches)
    {
        _high = Channel.CreateBounded<TRequest>(new BoundedChannelOptions(10_000){FullMode = BoundedChannelFullMode.Wait});
        _low = Channel.CreateBounded<TRequest>(new BoundedChannelOptions(20_000){FullMode = BoundedChannelFullMode.Wait});
        _concurrency = new SemaphoreSlim(maxConcurrentBatches);
        _process = process;
    }

    public async ValueTask EnqueueAsync(TRequest req, bool highPriority, CancellationToken ct = default)
    {
        var channel = highPriority ? _high : _low;
        await channel.Writer.WriteAsync(req, ct);
    }

    public async Task RunAsync(CancellationToken ct = default)
    {
        var buffer = new List<TRequest>();
        var timer = Stopwatch.StartNew();
        while (!ct.IsCancellationRequested)
        {
            // Prefer high‑priority channel
            var ch = await _high.Reader.WaitToReadAsync(ct) ? _high.Reader : _low.Reader;
            while (ch.TryRead(out var req))
            {
                buffer.Add(req);
                if (buffer.Count >= _batchSize) break;
            }

            if (buffer.Count > 0 && (buffer.Count >= _batchSize || timer.Elapsed >= _maxWait))
            {
                await _concurrency.WaitAsync(ct);
                _ = ProcessBatchAsync(buffer.ToArray(), ct).ContinueWith(t => _concurrency.Release());
                buffer.Clear();
                timer.Restart();
            }
        }
    }

    private async Task ProcessBatchAsync(TRequest[] batch, CancellationToken ct)
    {
        try
        {
            var results = await _process(batch);
            // Dispatch results via TaskCompletionSource stored per request
        }
        catch (Exception ex)
        {
            // Log, retry individual items, or drop based on policy
        }
    }
}

Performance Considerations

GPU Utilization: Keep it > 80 % to justify the cost. Use NVML or Azure GPU metrics.
CPU Usage: Target 20–30 % on a 16‑core host. Exceeding 40 % indicates the channel or batch builder is CPU bound.
Memory Footprint: Bounded channel + batch buffer + result cache should stay below 2 GB for typical request sizes.
Latency Percentiles: 99th‑percentile < 120 ms is a hard SLA for interactive LLM services.

Scaling Notes

Horizontal Scaling: Deploy multiple instances behind a load balancer. Use a distributed queue (Azure Service Bus, Kafka) to ingest requests so each instance pulls its own batches.
GPU Scaling: Auto‑scale GPU nodes based on GPUUtilization and BatchThroughput metrics. Avoid overscaling—each GPU adds cost.
Observability‑Driven Autoscale: Tie MaxConcurrentBatches to a live metric (e.g., GPUUtilization > 90 %) so the system adapts to traffic spikes.
Fault Isolation: Run each pipeline instance in its own AppDomain or container to prevent a single crash from killing all batches.

Checklist for Shipping Safely

All I/O is fully async; no blocking .Result or .Wait().
Bounded channels with BoundedChannelFullMode.Wait and realistic capacity.
Separate priority queues if SLA differs per tier.
Observability: traces for batch formation, processing start/finish, and result dispatch.
Metrics exported to Prometheus/Azure Monitor with alerts on BatchSizeUtilization, 99thPercentileLatency, and ChannelBackPressure.
Retry logic outside the batch loop, with per‑item retry counters.
Graceful shutdown: drain the channel before terminating the service.

Conclusion

Async pipelines in .NET are not a novelty; they are the proven, low‑overhead mechanism for turning a CPU‑bound request loop into a data‑flow engine that keeps GPUs humming. The real art lies in tuning the batch size, wait time, and concurrency to your specific traffic profile, and in instrumenting each stage so you can react before latency or cost spirals. Apply the decision guide, avoid the common pitfalls, and iterate on the adaptive strategy—your inference economics will thank you.

DEV Community