Today's digest highlights Ollama v0.32.4 with new Apple MLX GPU support, alongside llama.cpp gaining preliminary vision capabilities. Additionally, we see releases for AMD ROCm 7.14.0 and TensorRT-LLM, plus details on NVIDIA's Rubin GPU architecture.
Local AI & Open Models
Ollama v0.32.4 delivers robust Apple GPU MLX support and advanced quantized speculative decoding for efficient local inference. Concurrently, llama.cpp expands into multimodal AI with preliminary Minimax-M3 vision integration, while the YaneuraOu Shogi engine achieves a significant 30% NPS increase supported by bundled GPU runtimes.
Ollama v0.32.4 Brings Apple GPU MLX Support for Laguna, Enhances Quantized Speculative Decoding (Ollama)
Source: Ollama
The v0.32.4 release significantly enhances Ollama's capabilities for local AI inference, particularly for users on Apple Silicon hardware. This update introduces support for the Laguna model family via Apple's MLX engine, leveraging the integrated GPUs for accelerated inference. This integration promises improved performance and efficiency for running large language models directly on Mac devices, making powerful AI more accessible for local development and experimentation.
Further advancing local inference techniques, Ollama v0.32.4 refines speculative decoding by quantizing draft-model output heads to the requested type. This optimization helps maintain efficiency and accuracy when generating text using this acceleration method, which relies on a smaller, faster model to predict tokens for a larger, more accurate one. The release also addresses critical bug fixes, including rectifying Qwen3 MoE decoding issues that arose with differently-quantized experts, ensuring consistent performance across varied quantization schemes for this popular open-weight model.
The MLX integration is a game-changer for Mac users, and the quantized speculative decoding improvements mean faster, more efficient local inference for everyone, especially for complex MoE models like Qwen3.
llama.cpp b10142 Adds Preliminary Vision Support for Minimax-M3 (llama.cpp)
Source: llama.cpp
The latest llama.cpp release, b10142, marks a significant expansion of its capabilities with the preliminary addition of vision support for the Minimax-M3 model. While currently a text-only port that re-uses existing components, this update lays the groundwork for multimodal AI inference directly on local hardware. The integration leverages Minimax-M2 style Grouped-Query Attention (GQA) with per-head QK-norm and partial rotary embeddings, along with DeepSeek-V3 style lead token processing.
This development is crucial for advancing local AI, as it enables the foundational llama.cpp framework to begin processing and understanding visual information in addition to text. For developers and enthusiasts, this means the potential to run multimodal models on consumer-grade CPUs and GPUs, democratizing access to more sophisticated AI applications. The initial text-only port focuses on architectural compatibility, setting the stage for full visual processing in future iterations and opening new avenues for local multimodal AI exploration.
Adding vision support to llama.cpp is a huge step for local multimodal AI, proving that advanced models can run efficiently even on consumer hardware. This is a clear indicator of where local inference is heading.
ふかうら王 V9.40 Shogi Engine Achieves 30% NPS Boost, Bundles CUDA/TensorRT/cuDNN Runtimes (やねうら王 / YaneuraOu (将棋))
Source: やねうら王 / YaneuraOu (将棋)
The release of ふかうら王 V9.40 marks a substantial advancement for the popular YaneuraOu Shogi engine, delivering a significant performance uplift of approximately 30% in Nodes Per Second (NPS) for smaller models. This enhancement in search speed directly contributes to stronger play and more efficient analysis, a critical factor for competitive Shogi AI. The V9.40 update also includes comprehensive runtime bundling, incorporating CUDA, TensorRT, and cuDNN, which simplifies setup and ensures optimized GPU inference for users with NVIDIA hardware.
Beyond raw speed, ふかうら王 V9.40 introduces support for 入玉特徴量 (nyugyoku features), indicating potential refinements to its NNUE evaluation function, which is crucial for handling complex end-game scenarios where a king enters the opponent's camp. This technical improvement in evaluation, combined with the packaged runtimes, makes the engine more accessible and performant for a wider range of users. A Windows x64 CPU-only version is also provided, ensuring broad compatibility regardless of GPU availability, although stressing that the highest NPS gains are typically observed with GPU acceleration.
A 30% NPS boost with pre-bundled CUDA/TensorRT/cuDNN means instant, powerful upgrades for Shogi players and researchers. This release exemplifies how game AI pushes the boundaries of efficient local inference.
Full Local AI & Open Models archive
GPU, CUDA & Autonomous Driving
Today's top tech news features NVIDIA's unveiling of its Rubin GPU architecture for Agentic AI, alongside significant official software releases from both NVIDIA and AMD. ROCm 7.14.0 introduces a new build system for AMD's open compute stack, while TensorRT-LLM v1.3.0rc22 enhances LLM inference optimization on NVIDIA GPUs.
AMD ROCm 7.14.0 Release (AMD ROCm)
Source: AMD ROCm
AMD has officially released ROCm Core SDK 7.14.0, marking a significant transition to "TheRock," a new build and release system. This update introduces a modular architecture aimed at enhancing flexibility, maintainability, and alignment with community standards for AMD's open compute platform. Developers and researchers leveraging AMD Instinct™ GPUs can expect improvements in the foundational stability and integration capabilities for their AI and HPC workloads.
The modular design of ROCm 7.14.0 is intended to streamline future development and integration, making it easier for contributors to work with the framework and for users to deploy custom configurations. This release underpins a more robust and adaptable ecosystem for high-performance computing and artificial intelligence on AMD hardware. The transition to "TheRock" signifies AMD's commitment to evolving ROCm as a leading open-source alternative for GPU programming, promising long-term benefits for the developer community and enterprise users.
Upgrading to ROCm 7.14.0 is a critical step for AMD users; the 'TheRock' build system promises better long-term stability and easier custom module integration, making ROCm development smoother.
TensorRT-LLM v1.3.0rc22 Released (TensorRT-LLM)
Source: TensorRT-LLM
NVIDIA has released TensorRT-LLM v1.3.0rc22, a release candidate for its powerful library designed to accelerate large language model inference on NVIDIA GPUs. This update addresses several known issues, particularly highlighting stability concerns with torch.compile crashes in the PyTorch compilation backend. Furthermore, it notes that certain multi-GPU accuracy paths may fail with KeyError during the remove_copy pass when torch_compile=True, affecting models like DeepSeek-V3-Lite and Llama-3.1-8 across various floating-point formats including bf16, FP8, and NVFP4.
This release is crucial for developers relying on TensorRT-LLM to optimize their LLM deployments. By identifying and detailing these issues, NVIDIA provides transparency and guidance for users working with the latest PyTorch integrations and advanced quantization schemes. While a release candidate, it offers an early look at ongoing improvements and known limitations, enabling users to prepare for the stable release and tailor their development workflows accordingly to maximize performance and accuracy on NVIDIA hardware.
This TensorRT-LLM rc22 is essential for anyone pushing state-of-the-art LLMs on NVIDIA hardware; being aware of the
torch.compileand multi-GPU issues is key to stable deployment with advanced formats like FP8.
Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI (NVIDIA Developer Blog)
Source: NVIDIA Developer Blog
NVIDIA unveiled details about its upcoming Rubin GPU architecture, designed to power the escalating demands of "Agentic AI." This architecture builds upon the foundation of its predecessors to support the shift from discrete AI model training and human-facing chat interfaces to always-on AI factories that generate intelligence at scale. The Rubin GPU is positioned as a critical component in NVIDIA's accelerated roadmap, aiming to meet the unprecedented compute needs driven by increasingly complex AI workloads.
The blog post emphasizes how Agentic AI requires not just raw compute power but also sophisticated memory and interconnect solutions. While specific technical specifications were not fully disclosed, the announcement signals NVIDIA's strategic direction in hardware innovation, focusing on architectures optimized for the next generation of AI applications. This insight into the Rubin architecture provides a glimpse into the future of GPU capabilities and NVIDIA's commitment to advancing the core hardware that underpins the AI revolution, laying the groundwork for future breakthroughs.
Rubin GPU architecture is a big roadmap item; understanding NVIDIA's next-gen silicon direction is crucial for planning future AI infrastructure and anticipating performance leaps for agentic AI workloads.
Full GPU, CUDA & Autonomous Driving archive
Compiled daily from official release feeds, vendor changelogs and engineering blogs. Archive: https://media.patentllm.org
Top comments (0)