This article was originally published on our engineering blog, WithNext.NET. It's reposted here with the canonical link pointing back to the original.
On the local-LLM community r/LocalLLaMA, a .NET inference engine called TensorSharp drew attention by publishing benchmarks against llama.cpp. Most inference engines are built on C/C++ or Python, so a project that implements the core of LLM inference in pure C# / .NET is worth a close look — especially if .NET is your home turf and you'd like to pull AI inference into that stack.
This post summarizes what TensorSharp actually does and how it compares to llama.cpp, based on the author's Reddit post and GitHub repository. The numbers below are the author's own measurements, and I add a .NET-developer perspective on top.
What is TensorSharp?
TensorSharp is a native .NET inference engine for models in the GGUF format. It's developed by zhongkaifu and released under the BSD-3-Clause license. It ships with a console app, a web-based chatbot UI, and Ollama- / OpenAI-compatible HTTP APIs so you can call it from your own programs. It runs on Windows, macOS, and Linux, and can use the GPU.
The important part: this is not just a C# wrapper around llama.cpp. The author states the engine was implemented bottom-up, and when running on the CPU backend it is 100% pure C#. On top of that it implements CUDA, MLX, and GGML backends — the GGML backend is referenced as an external project, with several fused operations layered on top. Supported backends span Metal (macOS), CUDA (NVIDIA), Vulkan (general GPU), and CPU, all with CPU fallback.
Model coverage is broad, too: Gemma 3 / 4, Qwen 3 / 3.5 / 3.6, GPT-OSS, Nemotron-H, Mistral 3, and even multimodal models such as DiffusionGemma and Qwen-Image-Edit. It handles image, video, and audio input, PDF documents, tool calling, thinking mode, and structured output. Notably, quantizations like Q4_K_M, Q8_0, MXFP4, and IQ2_XXS are computed directly, without a dequantization step.
Benchmarks against llama.cpp
The author's benchmarks compare TensorSharp and llama.cpp on the same backend, reporting the geometric mean of the speed ratio per scenario. Above 1.00× means TensorSharp is faster; decode and prefill are throughput, while TTFT is the latency to the first token. Measurements cover both CUDA and Vulkan across several models.
TensorSharp vs. llama.cpp benchmarks (source: Reddit r/LocalLLaMA / the project author)
The results vary by model and backend. Prefill and TTFT favor TensorSharp in many cases — for example, Gemma 4 E4B on CUDA shows prefill at 1.28× and TTFT at 1.27×. Decode, on the other hand, hovers around 1.0, and on Qwen 3.6 35B-A3B (Vulkan) it drops to 0.87×, below llama.cpp. On Qwen 3.6 27B dense, prefill / TTFT fall under 1.0 instead. In other words, it is not uniformly faster. The author describes the overall result as "on par with llama.cpp," and that cautious framing matches what the numbers show.
What's genuinely notable is that a pure .NET implementation reaches this level at all. Inference engines are a dense mix of matrix math and memory management — historically the exclusive domain of C/C++. Seeing C# land in the same range says something about the maturity of .NET's runtime performance and ecosystem.
Optimizations it brings in
TensorSharp actively borrows recent inference optimizations. The author says they adapted paged KV cache and continuous batching from vLLM, an SSD-based cache for MoE models from oMLX, and GGUF quantization from llama.cpp. The KV cache supports prefix sharing via block hashing and an iteration-level scheduler, and continuous batching is on by default.
For Qwen 3.6 and Gemma 4 it supports speculative decoding via MTP / NextN, and tensor parallelism follows Megatron-LM's column/row parallel patterns, extending toward multiple GPUs and multi-node TCP clustering. Multi-GPU support came up in the Reddit comments; the author replied that PRs for both single-node multi-GPU and multi-node are in progress and under active development. It's fair to read this area as still evolving.
What it means for .NET teams
When teams bring local LLMs into production, the common shape is a Python-side model server that the application calls over HTTP. That works, but it doubles the runtime and adds deployment and operational surface. A credible option where the inference engine itself runs on .NET invites you to revisit that architecture. Because TensorSharp exposes Ollama- / OpenAI-compatible APIs, you can often swap it in without rewriting much of your existing client code.
That said, production adoption calls for a level head. The benchmarks scatter across models and backends, and multi-GPU support is still in development. The sound approach is to measure with your own target models and hardware, compare against llama.cpp or your current setup, and decide from there.
Wrap-up
TensorSharp implements GGUF inference in pure .NET and, in the author's benchmarks, reaches the same range as llama.cpp. Prefill and TTFT lead in many cases, while decode ties or trails — it isn't a blanket win. Even so, the fact that completing LLM inference in C# alone is now within practical reach is significant, and it's worth watching for any team that wants to fold AI into a .NET stack.
If you'd like the fuller Japanese write-up with additional context, it lives on the original post: TensorSharp — a pure C# GGUF inference engine that takes on llama.cpp | WithNext.NET.
Sources
We write about .NET / C# modernization, performance, and applied AI at WithNext.NET.
Top comments (0)