Why Local LLMs Don't Need C++ or Python: Building a 15MB Native AOT Inference Engine in .NET 10

Bare-Metal LLM Inference in Pure C#: Bypassing CUDA Toolkits and Native C++ DLLs

The conventional consensus across AI engineering is simple: high-performance local LLM execution belongs exclusively to C++ runtimes, multi-gigabyte CUDA toolkits, and bindings over llama.cpp or vLLM.

When orchestrating local models from managed languages like C#, typical implementations rely on interop wrappers over unmanaged native binaries (cudart64.dll, cublas64.dll, or libllama). This introduces DLL distribution overhead, host-to-device PCIe bandwidth bottlenecks during token sampling, and severe desktop instability when running compute on display-bound integrated GPUs.

By stripping out runtime layers and interacting directly with driver interfaces, managed runtimes can match and outpace conventional native daemons. Glacier.Inference runs direct memory-mapped GGUF models in pure C# .NET 10 across NVIDIA, AMD, and Intel silicon without external C++ binaries.

┌────────────────────────────────────────────────────────────────────────┐
│                        Glacier.Inference Core                          │
│  ├─ MemoryMappedFile Zero-Copy GGUF Reader (Sub-30ms cold mapping)     │
│  ├─ Pure C# Bare-Metal SASS Engine (Direct P/Invoke nvcuda.dll)        │
│  ├─ Bare-Metal Direct3D 12 Compute (HLSL Wave32 via Vortice.D3D12)     │
│  ├─ SpeculativeEngine (N-gram Prompt Lookup & Batched Verification)    │
│  ├─ Fused In-VRAM GPU Argmax Reduction (Warp-shuffle, 4-byte transfer) │
│  └─ Adaptive Unmanaged KV-Cache (FP16 / FP8 Dynamic Ring Buffer)       │
└────────────────────────────────────────────────────────────────────────┘

---

1. Bypassing CUDA Runtime Bloat via Direct Driver SASS

Typical CUDA execution routes instructions through cudart64.dll and cublas64.dll. Glacier completely circumvents the CUDA runtime layer. It communicates directly with the base Windows GPU driver (nvcuda.dll) via low-level P/Invoke, dispatching precompiled fatbinaries straight to the Streaming Multiprocessors (SMs).

// Direct driver context and module dispatch without cudart64.dll
[DllImport("nvcuda.dll", EntryPoint = "cuLaunchKernel")]
public static unsafe extern CUresult LaunchKernel(
    CUfunction f,
    uint gridDimX, uint gridDimY, uint gridDimZ,
    uint blockDimX, uint blockDimY, uint blockDimZ,
    uint sharedMemBytes,
    CUstream hStream,
    void** kernelParams,
    void** extra);

By compiling modular .cuh routines into an embedded universal cubin containing dedicated slices (sm_75, sm_80, sm_86, sm_89, sm_90), the engine verifies zero DRAM stack spills (STACK: 0 via cuobjdump). Registers house all dequantization multipliers and accumulators without hitting DRAM stack frames.

The result is a self-contained ~15 MB single-file Native AOT executable replacing a 4.5 GB toolchain.

---

2. Eliminating PCIe Starvation: In-VRAM Warp-Shuffle Argmax

In standard architectures, evaluating greedy token selection involves transferring logit tensors back over PCIe to the host CPU:

Logit Transfer Per Step = 152K vocab × 4 bytes ≈ 608 KB

At 45 tokens per second, transferring 608 KB back and forth over the host interface introduces micro-stalls and PCIe latency. Glacier fuses the final linear projection and reduction directly on the device with a 512-thread warp-shuffle kernel:

// GPU-side warp reduction eliminating CPU transfer overhead
[numthreads(512, 1, 1)]
void ArgmaxReduction(uint3 tid : SV_DispatchThreadID, uint3 lid : SV_GroupThreadID) {
    // 512-thread tree reduction within registers across 152k logits
    // Emits exactly 1 int32 winning token index into device memory
}

Instead of copying 608 KB across the bus every token, Glacier transfers exactly 4 bytes (one int32 token ID), dropping reduction latency to ~3.2 µs.

---

3. Register-Tiled Direct3D 12 Compute for Integrated GPUs

Deploying local inference on consumer hardware often fails on laptops where AMD RDNA or Intel Arc graphics double as the primary display adapter. Long compute dispatches trigger Windows Timeout Detection and Recovery (TDR), resetting the display driver.

Glacier implements a pure Direct3D 12 compute engine using Vortice.D3D12 and HLSL Wave32 compute shaders:

• Fine-Grained Command Dispatches: Eliminates TDR resets by cooperating cleanly with Desktop Window Manager (DWM).
• Unified Memory (UMA) Saturation: On architectures like the AMD Radeon 890M (16 CUs, RDNA 3.5), weights map straight into unified LPDDR5X memory without secondary PCIe staging copies.
• 32-Token GEMM Tiling: Prefill GEMV loops unroll into 4-tile register chunks, keeping dequantized vectors inside fast SIMD32 wave registers.

---

4. Beating Hardware Bandwidth Ceilings with Speculative Decoding

Autoregressive transformer generation is inherently memory-bandwidth bound. Every generated token requires streaming the entire model weight footprint through the compute core.

On a 128-bit GDDR6 memory bus running at 256 GB/s, reading a 4.68 GB model sets a strict theoretical wall clock limit:

Max Theoretical Serial Throughput = 256 GB/s ÷ 4.68 GB ≈ 54.7 tokens/sec

Glacier integrates batched speculative verification (VerifyBatch) using zero-cost prompt suffix lookup (PromptLookupDraftProvider):

using var target = new InferenceSession("models/Qwen2.5-7B-Instruct-Q4_K_M.gguf");
using var engine = new SpeculativeEngine(target);

var options = new SpeculativeOptions
{
    MaxDraftTokens = 4, // Propose 4 candidate tokens in <1 µs
    MaxTokens = 256
};

var result = await engine.GenerateAsync("Explain quicksort in C#", options);

Instead of reading 4.68 GB from VRAM $K$ times for $K$ tokens, the candidate sequence is verified against the transformer in a single batch pass. The weights are pulled through the memory bus only once, accelerating generation rates to 72–104+ tokens/sec on an RTX 4060 laptop GPU.

---

5. Empirical Head-to-Head Benchmarks

The following runs compare Glacier against an Ollama local daemon on identical hardware.

NVIDIA GeForce RTX 4060 Laptop (8 GB GDDR6, 256 GB/s)

Model: DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf (4.68 GB)

Metric	Glacier.Inference (.NET 10)	Local Daemon (Go + C++ CUDA)	Margin
Runtime Architecture	Pure C# (Native AOT)	C++ / cuBLAS / libllama	Zero external DLLs
Binary Footprint	~15 MB Single File	~4.5 GB Toolkit + Engine	300x smaller
Cold Start to First Token	1.50 s	3.50+ s	2.3x faster
Serial Generation	41.92 tok/s (208 GB/s)	43.20 tok/s (216 GB/s)	Within 3% of cuBLAS
Speculative Generation	72.5 – 104.8 tok/s	N/A (Serial decode)	Up to 2.4x faster
Sampling Overhead	~3.2 µs (In-VRAM)	~800 µs (DtoH transfer)	250x reduction

AMD Radeon 890M Integrated GPU (16 CUs, 15.5 GB Unified LPDDR5X)

Model: Qwen3-30B-A3B-Instruct-Q3_K_L.gguf (13.58 GB MoE, 3B Active)

Execution Pipeline	Memory Model	Generation Rate	Turnaround Time
Direct3D 12 Compute (HLSL Wave32)	Unified LPDDR5X (Direct)	21.68 tok/s	1.54 s
Host CPU SIMD (24T AVX-512)	System Memory	0.89 tok/s	18.20 s

Running a 30B parameter Mixture-of-Experts architecture in pure C# directly on an integrated APU delivers 21.68 tok/s, beating multi-threaded AVX-512 CPU execution by 24.4x.

---

Summary Takeaway

Managed languages don't have to surrender low-level compute workloads to external runtime stacks. By combining MemoryMappedFile zero-allocation weight access, Direct3D 12 compute pipelines, direct driver P/Invokes, and in-VRAM warp reductions, pure .NET 10 delivers bare-metal throughput while keeping deployment to a single, portable binary.

The complete code, benchmarks, and standalone CLI binaries are available on GitHub: Glacier.Inference.

Comments

[Raknaos]

The number I keep turning over is 608 KB per token — 152K vocab times 4 bytes, every step, both directions. It's nothing next to a weight load, which is exactly why a fused in-VRAM argmax is the right call: the cost is latency and micro-stalls, not bandwidth.

What I'd want to know is what you had to re-implement once you dropped the CUDA runtime. Bypassing cudart also means giving up its stream ordering, pinned buffers and device selection on multi-GPU boxes. Is the KV ring buffer carrying all of that yourself, or does part of it still ride on nvcuda defaults? Curious how much of the 15 MB is your code versus the runtime you kept.

[Ian Cowley]

With Google Gemini doing most of the heavy lifting in pair-programming and kernel design, we were able to strip out the layers very cleanly!

To clarify the architecture:

1. Driver API (nvcuda.dll) vs Runtime (cudart64.dll)

We dropped cudart64.dll (the C++ runtime helper library), but we speak directly to nvcuda.dll (the CUDA Driver API already in Windows System32). Many people assume cudart provides streams and events, but it’s just a heavyweight C++ convenience wrapper with hidden global state over the underlying driver.

Without cudart, Glacier implements everything in pure C#:

• Stream Ordering & Hardware Barriers: We allocate explicit non-blocking streams (cuStreamCreate). $Q$ runs on the main stream, while $K$ and $V$ projections launch concurrently on dedicated CUDA streams. Synchronization uses cuEventRecord and cuStreamWaitEvent, meaning stream barriers happen entirely on the GPU hardware work distributor with zero CPU round-trips.
• Device Selection: Replaced cudaSetDevice() with a custom DeviceManager using DXGI and cuDeviceGet / cuCtxCreate_v2, automatically choosing between discrete SASS compute and display-safe Direct3D 12.
• Fatbinary Dispatch: The precompiled multi-arch .cubin is loaded directly via cuModuleLoadData, with entry points called via cuLaunchKernel decorated with [SuppressGCTransition].

2. The KV Ring Buffer is 100% Custom

Nothing rides on nvcuda defaults. Each layer’s KV cache is allocated as a flat CUdeviceptr block in VRAM. Sequence indexing, head strides, and an adaptive precision ring (lossless FP16 under 4k tokens; dynamic FP8 compression for 16k–32k context) are calculated via raw pointer offsets in C# and passed directly into our custom GQA attention kernel.

3. Where does the ~15 MB come from?

0 KB of the CUDA runtime is bundled or kept. The ~15 MB Native AOT binary breaks down as:

• ~10 MB: Stripped .NET 10 AOT runtime (pruned BCL, minimal GC, thread pool).
• ~3 MB: Glacier C# engine (zero-copy MemoryMappedFile GGUF reader, BPE tokenizer, speculative decoding, HTTP server).
• ~1.5 MB: Embedded universal fatbinary (kernels.cubin) with dedicated SASS slices for sm_75 to sm_90 (STACK: 0 spills).
• ~200 KB: Embedded Direct3D 12 HLSL compute bytecodes.

All memory management, stream DAGs, and ring buffers are pure C#; nvcuda.dll is just the raw pipe to the SMs!