Today sees the release of Stockfish 18, bringing a major update to the world's strongest chess engine. Alongside this, the LLM ecosystem received significant attention with new model integrations, audio input capabilities, and advancements in Mixture-of-Experts models.
Local AI & Open Models
Today features the launch of the Inkling multimodal model with immediate support in Hugging Face Transformers v5.14.0 and vLLM v0.26.0. Additionally, llama.cpp b10155 brings MiMo-V2.5 audio input and RVQ model support, while chess AI sees a major release with Stockfish 18.
Hugging Face Transformers v5.14.0 Integrates New Inkling Model, vLLM Adds Support (Hugging Face Transformers / vLLM)
Source: Hugging Face Transformers / vLLM
Hugging Face Transformers v5.14.0 officially introduces the Inkling model family, a new general-purpose multimodal model from Thinking Machines. Weighing in at 975 billion total parameters with 41 billion active parameters, Inkling marks a significant new entry in the open-weight model landscape. Its inclusion in the Transformers library means developers can immediately access and experiment with this substantial model, leveraging the comprehensive toolset provided by Hugging Face for model loading, processing, and inference. This release positions Inkling for broad adoption and further development within the AI community, especially for those working with large-scale multimodal tasks.
Complementing this, vLLM v0.26.0 has simultaneously rolled out full support for the new Inkling model family. This includes base modeling capabilities and, crucially, piecewise CUDA graph support. The integration of CUDA graphs is a key development for inference acceleration, enabling more efficient and lower-latency execution of large models like Inkling on NVIDIA GPUs. For practitioners focused on local inference, this means Inkling can be run with optimized performance, making it more practical for real-world applications on consumer-grade hardware. The rapid, dual-platform support from both Transformers and vLLM underscores Inkling's immediate readiness for deployment and research.
The immediate integration of Inkling into both Transformers and vLLM means developers can start experimenting with this large multimodal model on local hardware with accelerated inference right away, which is crucial for evaluating its real-world performance.
llama.cpp Adds MiMo-V2.5 Audio Input and RVQ-based Model Support (llama.cpp)
Source: llama.cpp
The latest llama.cpp release, b10155, significantly expands its capabilities by adding support for the MiMo-V2.5 audio input model. This update marks an important stride into multimodal AI for llama.cpp, moving beyond its traditional text-only focus to handle audio inputs directly. The ability to process audio with the MiMo-V2.5 model, which is described as an RVQ-based model, is particularly relevant for local inference. RVQ (Residual Vector Quantization) is a technique known for its efficiency in compressing models, making them more amenable to running on consumer-grade CPUs and GPUs with limited memory resources.
This enhancement means developers can now leverage llama.cpp for a broader range of applications, such as local speech-to-text processing or AI agents that respond to spoken commands, all while benefiting from the project's optimization for efficient, local execution. The continuous integration of new, diverse models and efficient quantization methods like RVQ reinforces llama.cpp's position as a vital tool for making advanced AI accessible on everyday hardware. This release ensures that multimodal AI research and deployment can increasingly occur without reliance on cloud-based services.
Adding audio input and support for RVQ-based multimodal models to
llama.cppis a big step, enabling more diverse local AI applications and pushing the boundaries of what's possible on consumer hardware for multimodal tasks.
Stockfish 18 Released: Major Update for the World's Strongest Chess Engine (Stockfish (chess NNUE))
Source: Stockfish (chess NNUE)
Stockfish 18, a new major release, cements its position as the world's strongest open-source chess engine. This update continues Stockfish's legacy of pushing the boundaries of competitive computer chess, offering significant improvements in its evaluation function and search algorithms. For the "Local AI & Open Models" category, Stockfish is a prime example of high-performance AI runnable on consumer hardware, particularly through its use of Neural Network Uci Engine (NNUE) evaluation functions. NNUE networks are compact neural networks specifically designed for efficient inference on CPUs and GPUs, allowing Stockfish to achieve incredible strength through a blend of deep search and accurate, learned positional evaluation.
The release of Stockfish 18 is not merely an incremental update; it represents the culmination of countless self-play games and architectural refinements by its global community of developers. Its open-source nature means the underlying technology, including its sophisticated NNUE architecture and optimization techniques, is fully transparent and available for study. For developers and AI enthusiasts, Stockfish 18 serves as an excellent case study in optimizing neural network inference for real-time decision-making in complex environments, making it a first-class story in the realm of game AI and local, high-performance computing.
Stockfish 18 continues to be the gold standard in chess AI, and its relentless improvements showcase the power of NNUE-based evaluation, providing a fantastic example of cutting-edge local inference on consumer hardware for game engines.
Full Local AI & Open Models archive
GPU, CUDA & Autonomous Driving
This week, NVIDIA released TensorRT-LLM v1.3.0rc21, enhancing LLM inference while their GB300 NVL72 system achieved a new world record in MoE pre-training. AMD also made strides with the introduction of Instella-MoE, an open Mixture-of-Experts language model trained on their Instinct MI300X GPUs.
[OFFICIAL RELEASE] TensorRT-LLM v1.3.0rc21 released (NVIDIA)
Source: NVIDIA
TensorRT-LLM v1.3.0rc21, an official release from NVIDIA, continues to advance the toolkit for optimizing large language model inference on NVIDIA GPUs. This version, identified by its release candidate tag, is crucial for developers seeking to maximize performance and efficiency for their LLM deployments. A notable change includes the deprecation of the AutoDeploy backend, signaling a shift in deployment strategies and focusing efforts on more robust, future-proof integration methods.
The release notes emphasize NVIDIA's commitment to improving model support and functionality, with ongoing work into "agentic approaches" to streamline the time-to-functionality for various LLM architectures. This indicates a future direction towards more automated and intelligent optimization pipelines within TensorRT-LLM, addressing critical pain points for users dealing with complex and evolving LLM ecosystems. Developers are encouraged to review the release notes for detailed changes and compatibility considerations to ensure smooth transitions and leverage the latest optimizations for their inference workloads.
This release, even as a release candidate, is critical for staying at the forefront of LLM inference optimization on NVIDIA hardware, especially with hints at agentic approaches simplifying future model support.
Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72 (NVIDIA Developer Blog)
Source: NVIDIA Developer Blog
NVIDIA has announced a new world record in Mixture-of-Experts (MoE) pre-training, achieved on their powerful NVIDIA GB300 NVL72 system. This milestone underscores the increasing shift towards MoE architectures for frontier model pre-training, which fundamentally alters the limitations of large-scale AI training by optimizing compute per token. The GB300 NVL72, representing NVIDIA's cutting-edge Blackwell-generation GPU and NVLink interconnect technology, demonstrates unparalleled capabilities in handling these demanding workloads.
The achievement highlights the performance advantages of NVIDIA's integrated hardware and software stack, designed to scale AI workloads efficiently. This level of performance is crucial for advancing the capabilities of next-generation AI models, enabling researchers and developers to train larger, more complex models faster than ever before. For practitioners, this translates into quicker iteration cycles and the potential for breakthroughs in various AI domains where MoE architectures are becoming increasingly prevalent.
Achieving a world record on the GB300 NVL72 for MoE pre-training validates NVIDIA's hardware and software stack for pushing the boundaries of large-scale AI, offering immense compute power for cutting-edge models.
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model (AMD ROCm Blog)
Source: AMD ROCm Blog
AMD has unveiled Instella-MoE, a significant new addition to the open-source AI ecosystem, presented as a state-of-the-art fully open Mixture-of-Experts (MoE) language model. This release marks a key effort by AMD to contribute advanced models optimized for their hardware, specifically highlighting its training from scratch on AMD Instinct™ MI300X GPUs. Instella-MoE features 16 billion total parameters with 2.8 billion active parameters, offering a powerful, yet efficient, solution for various AI applications, targeting the growing demand for specialized, efficient models.
The introduction of Instella-MoE demonstrates AMD's commitment to supporting the development and deployment of cutting-edge AI on its Instinct hardware through the ROCm software platform. By providing a fully open MoE model, AMD empowers developers and researchers to explore and utilize advanced model architectures without proprietary restrictions, fostering innovation and wider adoption of AMD's AI accelerator solutions. This initiative is particularly valuable for the community seeking high-performance, open alternatives for large-scale language model inference and training on AMD hardware.
Instella-MoE is a compelling open-source MoE LLM that directly showcases the capabilities of AMD Instinct MI300X, providing a clear path for developers to leverage AMD's hardware with a modern, efficient model architecture.
Full GPU, CUDA & Autonomous Driving archive
Compiled daily from official release feeds, vendor changelogs and engineering blogs. Archive: https://media.patentllm.org
Top comments (0)