Today's engineering digest is packed with AI and GPU advancements, including vLLM v0.26.0's CUDA graph improvements and Inkling support, plus llama.cpp adding GLM-5.2 speculative decoding. Alongside these, AMD released GPU Operator v1.5.0 with Kubernetes infrastructure control and automated node recovery, while Stockfish 16.1 and new NVIDIA TensorRT developer tools also shipped.
Local AI & Open Models
This week's highlights feature significant updates to local inference engines: llama.cpp adds speculative decoding for GLM-5.2, vLLM introduces Inkling model support and performance boosts with piecewise CUDA graphs. Additionally, the renowned Stockfish 16.1 chess engine sees its latest official release, underscoring continued advancements in game AI.
llama.cpp b10174 adds GLM-5.2 speculative decoding support (llama.cpp)
Source: llama.cpp
llama.cpp, the highly popular C/C++ inference engine for large language models, has released version b10174, bringing a notable enhancement in inference acceleration. This update specifically introduces NextN/MTP speculative decoding support for the GLM_DSA (GLM-5.2) model. Speculative decoding is a cutting-edge technique that significantly speeds up token generation by having a smaller, faster draft model predict several tokens ahead, which are then verified in parallel by the larger target model. If the predictions are correct, multiple tokens are accepted simultaneously, leading to a dramatic increase in effective inference speed.
This release not only expands the range of models supported by llama.cpp—adding GLM-5.2 to its robust lineup—but also democratizes advanced inference techniques for users running models on consumer-grade hardware. The ability to utilize speculative decoding for a new model type means that developers and enthusiasts can experience faster, more efficient local AI inference without requiring top-tier data center GPUs. Users can leverage this feature by specifying --spec-type draft-mtp with GLM-5.2 models, making high-performance local AI more accessible and practical for everyday use.
Integrating speculative decoding for GLM-5.2 into llama.cpp is a big win for local inference. It means faster responses from this model on my GPU, directly improving the user experience for local LLM applications.
vLLM v0.26.0 adds Inkling model family support, improves CUDA graph for performance (vLLM)
Source: vLLM
vLLM, a high-throughput and low-latency LLM serving engine, has launched its v0.26.0 release, packed with substantial improvements for model compatibility and inference performance. A key highlight is the full support for the new Inkling model family, allowing users to efficiently deploy and serve these models with vLLM's optimized architecture. This expands the ecosystem of open-weight models that can benefit from vLLM's advanced features, such as PagedAttention for efficient KV cache management.
Furthermore, vLLM v0.26.0 introduces piecewise CUDA graph support. CUDA graphs are critical for reducing overhead in GPU operations by capturing and replaying sequences of GPU commands, significantly boosting inference speed, especially for smaller batches or dynamic workloads. This piecewise implementation likely offers more flexibility and broader applicability compared to monolithic CUDA graphs. Building upon previous releases, where Model Runner V2 became the default for dense models in v0.25.0 to enhance quantized model support and general efficiency, v0.26.0 continues this trajectory of optimizing GPU utilization and overall inference throughput. These advancements make vLLM an even more powerful tool for local LLM deployments demanding both performance and broad model support.
The addition of Inkling model support and piecewise CUDA graph optimization in vLLM v0.26.0 is excellent. It streamlines deployment for new models and provides tangible performance gains, which is crucial for maximizing throughput on a local GPU setup.
Stockfish 16.1 chess engine released, maintaining strength and accessibility (Stockfish (chess NNUE))
Source: Stockfish (chess NNUE)
The world-renowned open-source chess engine, Stockfish, has announced the official release of version 16.1. Stockfish consistently ranks as one of the strongest chess engines globally, making each new iteration a significant event for chess enthusiasts, researchers, and AI developers alike. This release continues Stockfish's tradition of incremental improvements in its neural network-based evaluation (NNUE) functions and search algorithms, which are pivotal to its extraordinary strength. NNUE allows the engine to evaluate positions with human-like intuition blended with brute-force calculation, leveraging neural networks trained on vast datasets of chess games.
Stockfish 16.1 remains freely available for download across various operating systems, including Windows, Linux, and macOS. This accessibility ensures that anyone with a consumer-grade CPU can run a world-class AI, demonstrating the power of optimized, open-source AI runnable on local hardware. The continuous refinement of Stockfish serves as a powerful benchmark for what's achievable in game AI, showcasing advances in quantized inference and the delicate balance between search depth and evaluation accuracy that defines cutting-edge chess engines.
Stockfish 16.1 is another testament to the power of open-source game AI. Its continued dominance and accessibility provide a fantastic benchmark for anyone interested in high-performance, locally executable AI.
Full Local AI & Open Models archive
GPU, CUDA & Autonomous Driving
Today's top tech news features the release of AMD GPU Operator v1.5.0, enhancing Kubernetes deployments with Dynamic Resource Allocation and automated node recovery. NVIDIA provides crucial updates for TensorRT, making long-running engine builds observable and cancelable, while AMD details optimizing Kimi-K2.5-MXFP4 on MI355X GPUs using ATOM.
AMD GPU Operator v1.5.0: DRA Support, Automated GPU Node Recovery, and Expanded Kubernetes Infrastructure Control (AMD ROCm Blog)
Source: AMD ROCm Blog
The AMD GPU Operator has released version 1.5.0, delivering significant infrastructure enhancements for orchestrating AMD GPU deployments within Kubernetes environments. A cornerstone of this update is the robust support for Kubernetes Dynamic Resource Allocation (DRA). This crucial feature empowers users with more flexible and efficient assignment of GPU resources to various workloads, dynamically adapting to demands and preventing resource contention, thereby optimizing overall cluster utilization.
Further bolstering reliability, version 1.5.0 introduces automated GPU node remediation workflows. This automation is vital for maintaining high performance and uptime in large-scale AI training and HPC inference deployments, as it ensures that nodes experiencing GPU-related issues are automatically detected and recovered. Additionally, the update broadens general Kubernetes infrastructure control, providing administrators with more granular options for configuring and managing their AMD GPU hardware alongside other cluster resources. These improvements collectively simplify operations for complex AI/ML pipelines and scientific computing, making it easier to deploy and manage AMD Instinct accelerators efficiently.
This release dramatically improves the manageability and stability of AMD GPU clusters in Kubernetes, especially with the automated node recovery and dynamic resource allocation. It’s a huge step towards more robust and efficient MLOps.
Make Long-Running NVIDIA TensorRT Engine Builds Observable and Cancelable in Python or C++ (NVIDIA Developer Blog)
Source: NVIDIA Developer Blog
NVIDIA has published a new guide demonstrating how to make TensorRT engine builds more manageable by enabling observability and cancelation. As model checkpoints grow and optimization processes become more intricate, TensorRT engine builds can often span minutes, if not longer, due to factors like large strongly-typed models, deep tactic searches, or cold timing caches on new GPU SKUs. This blog post introduces the use of callbacks in both Python and C++ to provide real-time updates on build progress and offers the ability to cancel an ongoing build.
The new approach allows developers to integrate progress reporting into their applications, offering visibility into the various stages of the TensorRT build process, such as parsing, optimization, and timing. This functionality is crucial for improving the developer experience, particularly for iterative development and debugging cycles where quick feedback and the ability to halt unproductive builds are invaluable. By implementing these patterns, developers can avoid wasted compute cycles and streamline their workflow when optimizing complex neural networks for deployment on NVIDIA GPUs.
Being able to observe and cancel TensorRT builds is a game-changer for iterating on complex models. This will save countless hours of development time and frustration, especially when fine-tuning inference pipelines.
Serve Kimi-K2.5-MXFP4 on MI355X with ATOM (AMD ROCm Blog)
Source: AMD ROCm Blog
AMD's ROCm blog details an in-depth optimization strategy for deploying the Kimi-K2.5-MXFP4 model on AMD MI355X GPUs using the ATOM framework. This post builds upon previous work on MoE (Mixture of Experts) optimization, specifically focusing on kernel and quantization techniques to achieve high performance for demanding agentic AI inference workloads. The core of the optimization involves applying W4A8 and W8A8 quantization with AMD Quark, demonstrating significant efficiency gains in serving large language models.
The article provides a comprehensive guide on leveraging ATOM for mixed-precision quantization, detailing how to implement specialized kernels to accelerate the dominant MoE parts of the Kimi-K2.5-MXFP4 model. This ensures that the model can be served with optimal throughput and reduced memory footprint on the MI355X architecture. Such deep-level kernel and quantization work is essential for pushing the boundaries of what's possible with LLM inference, especially for high-volume, low-latency agentic AI applications that are increasingly relying on hardware-specific optimizations for peak performance.
This guide provides excellent, actionable insights for maximizing performance of large language models on AMD's MI355X GPUs. The focus on kernel and quantization work with ATOM is exactly what's needed for pushing LLM inference efficiency.
Full GPU, CUDA & Autonomous Driving archive
Compiled daily from official release feeds, vendor changelogs and engineering blogs. Archive: https://media.patentllm.org
Top comments (0)