My new book is now available on Amazon! 😀
This book is written for AI application developers who are not satisfied with simply calling an LLM endpoint and want to understand model architectures and the internal workings of inference engines. It uses the open-source TensorSharp project and Google’s Gemma 4 E4B GGUF model as practical examples.
TensorSharp has achieved performance parity with llama.cpp across the main benchmarks, while outperforming it in several scenarios. The book explains some of the key performance optimizations and their implementations, including paged and prefix KV caching, continuous batching, GPU kernel fusion, and more.
I chose Gemma 4 E4B, a dense model, because it is a compact multimodal model that supports images, audio, and video, making it suitable for a wide range of devices. TensorSharp also supports and is optimized for MoE and diffusion architectures, as well as model families such as Qwen and GPT-OSS. However, due to limitations in time and book length, these topics are not covered in this edition. Those interested can explore the project directly on GitHub or contact me for further discussion.
I selected GGUF because it is an inference- and edge-device-friendly model format. This is particularly relevant to the .NET ecosystem, where local applications, mobile applications, and game development are important use cases. TensorSharp also supports the Safetensors format, which it currently uses for VAE and LoRA models.
For clarity and ease of understanding, the book primarily presents the CPU code path. In practice, however, TensorSharp supports and is extensively optimized for multiple GPU backends, including NVIDIA CUDA, Apple Metal/MLX, and Vulkan for AMD, Intel, and other devices. More implementation details are available in the GitHub repository.
TensorSharp and this book focus exclusively on model inference. For model training, readers can refer to Seq2SeqSharp, one of my earlier open-source projects. Adding a trainer to TensorSharp itself would not be particularly difficult, but building a modern, efficient architecture that unifies training and inference is a much larger undertaking. I do not currently have sufficient resources to explore that topic properly, so it remains outside the project’s scope for now.
One more point: TensorSharp is a project built with the assistance of a native coding agent. Throughout the book, I have also incorporated my own views on how code-agent-driven projects should be developed and managed—perhaps a little personal perspective woven into the technical content.
The philosophy can be summarized as:
Contracts as the source of truth.
Test-driven development.
Evaluation before optimization.
Here is the book link on Amazon:From Tensors to Tokens: Building a Multimodal LLM Inference Engine from Scratch with TensorSharp and Gemma 4 E4B
Here is TensorSharp Github Repo: https://github.com/zhongkaifu/TensorSharp
Top comments (2)
I found it interesting how you leveraged the TensorSharp project and Gemma 4 E4B model to demonstrate the internal workings of inference engines, particularly the performance optimizations such as paged and prefix KV caching, and GPU kernel fusion. The fact that TensorSharp has achieved performance parity with llama.cpp across main benchmarks is impressive, and I'm curious to dive deeper into the book to understand the implementation details. The support for multiple GPU backends, including NVIDIA CUDA and Apple Metal/MLX, is also a notable aspect, and I'd love to explore how this can be applied to real-world applications. How do you envision the integration of model training capabilities into TensorSharp in the future, and what potential challenges do you foresee in unifying training and inference architectures?
Thanks for these feedbacks in details. For training, update TensorSharp to support pre-train and supervised fine-tuning (SFT) is relative easier. The most challenge part is inter-communication and sync over multiple nodes (and GPUs).
For reinforcement learning fine-tune (RLFT), the whole system becomes more complicated, since some new components, such as rollout, environments and grading, will be involoved into a unified training/inference system.
There are some existing open RLFT source projects we could learn, but to actually working on it, plenty of resources will be required, such as computing resources, data resources. This is also a challenges part for RLFT.