TensorSharp is an native .NET open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Thanks recent contribtions from open source community, TensorSharp is able to run inference over multiple GPUs and nodes. So I updated it to support deepseek v4 flash model, and have better performance than llama.cpp. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8
Model: DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
| TensorSharp (cuda backend) | TensorSharp (ggml_cuda backend) | llama.cpp | |
|---|---|---|---|
| prefill u/16K | 836 tok/s | 963 | 558 |
| decode short | 31.5 | 37.0 | 35.3 |
| decode u/16K | 28.5 | 33.6 | 32.2 |
DSpark is also supported for speculative decoding and here is the benchmark with/without DSpark:
| Turn | Baseline | + DSpark | Acceptance |
|---|---|---|---|
| short (53 tok) | 25.6 | 44.5 (1.74x) | 87% |
| long generation (512) | 26.4 | 40.3 (1.53x) | 66% |
| follow-up (470) | 26.4 | 46.8 (1.77x) | 76% |
| 10K-token document (214) | 25.3 | 51.3 (2.03x) | 85% |
| second question on it (156) | 25.4 | 49.4 (1.94x) | 82% |
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
Top comments (0)