DEV Community

Durgesh
Durgesh

Posted on

10 Free resources for LLM engineering

10 free resources to learn LLM inference engineering

Covering topics such as:

  • KV cache and batching
  • Serving engines
  • GPU kernels and benchmarks

Start here and go in order:

1) LLM Inference Handbook (Modular)
↳ Learn the words first: prefill, decode, KV cache, latency, throughput.
↳ A reference you keep open while you read the rest.
https://handbook.modular.com/

2) Continuous batching from first principles (Hugging Face)
↳ Starts at attention and KV caching, ends at continuous batching.
↳ Shows why mixing prefill and decode in one batch helps throughput.

Continuous batching from first principles

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

3) Stanford CS336, Lecture 10: Inference (Spring 2025)
↳ Percy Liang explains why inference is memory limited.
↳ Then the fixes: quantization, speculative decoding and paged attention.

4) All About Transformer Inference (How To Scale Your Model, Google DeepMind)
↳ The math behind latency, throughput, memory and KV caches.
↳ Harder than the rest. Worth the extra effort.

All About Transformer Inference | How To Scale Your Model

Performing inference on a Transformer can be very different from training. Partly this is because inference adds a new factor to consider: latency. In this section, we will go all the way from sampling a single new token from a model to efficiently scaling a large Transformer across many slices of accelerators as part of an inference engine.

favicon jax-ml.github.io

5) MIT 6.5940 EfficientML (Fall 2024, Song Han)
↳ Watch the quantization lectures and Lecture 13 on LLM deployment.
↳ Slides and videos sit on the course page.

6) Inside vLLM (Aleksa Gordić)
↳ Scheduling, paged attention, prefix caching, speculative decoding, multi-GPU serving.
↳ Based on the vLLM V1 engine at one August 2025 commit, so some code has moved since.

7) Mini-SGLang (SGLang project)
↳ A compact serving engine with radix cache, chunked prefill and overlap scheduling.
↳ Reading the code is free. Running it needs Linux and an NVIDIA GPU with CUDA.

GitHub logo sgl-project / mini-sglang

A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.

Mini-SGLang

A lightweight yet high-performance inference framework for Large Language Models.


Mini-SGLang is a compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems. With a compact codebase of ~5,000 lines of Python, it serves as both a capable inference engine and a transparent reference for researchers and developers.

✨ Key Features

  • High Performance: Achieves state-of-the-art throughput and latency with advanced optimizations.
  • Lightweight & Readable: A clean, modular, and fully type-annotated codebase that is easy to understand and modify.
  • Advanced Optimizations
    • Radix Cache: Reuses KV cache for shared prefixes across requests.
    • Chunked Prefill: Reduces peak memory usage for long-context serving.
    • Overlap Scheduling: Hides CPU scheduling overhead with GPU computation.
    • Tensor Parallelism: Scales inference across multiple GPUs.
    • Optimized Kernels: Integrates FlashAttention and FlashInfer for maximum efficiency.
    • ...

🚀 Quick Start

⚠️ Platform Support: Mini-SGLang currently supports…





8) GPU MODE Lectures
↳ Start with Lectures 1, 3 and 4: profiling, getting started with CUDA, compute and memory architecture.
↳ The repo has the code and slides. The recordings are on their YouTube channel.

GitHub logo gpu-mode / lectures

Material for gpu-mode lectures

Supplementary Material for Lectures

YouTube Channel

The PMPP Book: Programming Massively Parallel Processors: A Hands-on Approach (Amazon link)

Lecture 1: Profiling and Integrating CUDA kernels in PyTorch

Lecture 2: Recap Ch. 1-3 from the PMPP book

Lecture 3: Getting Started With CUDA

Lecture 4: Intro to Compute and Memory Architecture

Lecture 5: Going Further with CUDA for Python Programmers

Lecture 6: Optimizing PyTorch Optimizers

Lecture 7: Advanced Quantization

Lecture 8: CUDA Performance Checklist

9) Triton Tutorials (official docs)
↳ Go from vector addition to fused softmax, matrix multiplication and fused attention.
↳ Each tutorial is runnable code you can download.

10) AIPerf (NVIDIA)
↳ Measure time to first token, inter token latency, throughput and goodput.
↳ Its guides cover trace replay, so you can test with real traffic patterns.

GitHub logo ai-dynamo / aiperf

AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution.

AIPerf

PyPI version License Codecov Discord Ask DeepWiki

AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution. It provides detailed metrics using a command line display as well as extensive benchmark performance reports.

AIPerf UI Dashboard

Quick Start

This quick start guide leverages Ollama via Docker Desktop.

Setting up a Local Server

In order to set up an Ollama server, run granite4:350m using the following commands:

docker run -d \
  --name ollama \
  -p 11434:11434 \
  -v ollama-data:/root/.ollama \
  ollama/ollama:latest
docker exec -it ollama ollama pull granite4:350m
Enter fullscreen mode Exit fullscreen mode

Basic Usage

Create a virtual environment and install AIPerf:

python3 -m venv venv
source venv/bin/activate
pip install aiperf
Enter fullscreen mode Exit fullscreen mode

Note

The optional crick dependency has no prebuilt wheel for some platforms including Linux aarch64 (arm64), and falls back to an sdist that needs a C compiler at install time. crick backs the non-default tdigest list-metric backend, so a plain pip…

Basic Python, PyTorch and Transformer knowledge will help.

All 10 are free to read or watch. Running the code examples may need a paid GPU.

Top comments (0)