Today brings significant advancements in AI, led by Ollama v0.32.9's integration of NVIDIA Nemotron 3.5 Lightning. We also cover vLLM v0.27.0, a new AMD Mixture-of-Experts model, further Ollama updates, and NVIDIA's latest agentic AI tools and JetPack release.
Local AI & Open Models
This week's top stories feature major updates to local inference runtimes: Ollama adds crucial support for NVIDIA's new Nemotron 3.5 Lightning and Meta's Muse Glimmer models, while vLLM delivers a substantial v0.27.0 release with Kimi K3 support and numerous improvements.
Ollama v0.32.9 Integrates NVIDIA Nemotron 3.5 Lightning (Ollama)
Source: Ollama
Ollama, the popular open-source tool for running large language models locally, has released version v0.32.9, prominently featuring support for NVIDIA's new Nemotron 3.5 Lightning. This 30B mixture-of-experts (MoE) model, with 3B active parameters, is specifically designed for agentic workloads and optimized for high-volume execution of tool calls, result validation, and subagent delegation. Its integration into Ollama significantly lowers the barrier for developers and researchers to experiment with this advanced open-weight model on consumer GPUs.
Nemotron 3.5 Lightning, as an open model, aligns perfectly with the local AI ecosystem, enabling users to leverage its specialized task execution capabilities without relying on cloud APIs. This release empowers practitioners to build and deploy sophisticated, always-on AI agents directly on their local machines, benefiting from the model's efficiency and NVIDIA's optimizations. The Ollama update underscores the growing trend of making powerful, frontier open models accessible for local inference, catering to the increasing demand for privacy-preserving and cost-effective AI development.
Nemotron 3.5 Lightning in Ollama is huge for anyone building local agents. Being able to run a specialized MoE model like this on my own hardware makes rapid iteration and experimentation so much easier.
vLLM v0.27.0 Release Brings Kimi K3 Support and Extensive Improvements (vLLM)
Source: vLLM
The vLLM project, known for its high-throughput inference engine, has rolled out its v0.27.0 release, featuring a massive update with 561 commits from 242 contributors. A highlight of this release is the full-stack landing of Kimi K3 support, encompassing core model files, kernels, and Python bindings. This integration allows users to efficiently run the Kimi K3 model, an important addition to the growing landscape of open-weight models for local and accelerated inference environments.
Beyond Kimi K3, vLLM v0.27.0 introduces a plethora of general improvements to its architecture, enhancing performance and stability for a wide range of LLMs. The substantial number of contributions reflects the active development and community engagement around vLLM, continuously pushing the boundaries of inference acceleration through techniques like PagedAttention, KV cache optimizations, and efficient batching. This release solidifies vLLM's position as a go-to framework for demanding local inference workloads, especially beneficial for users with consumer GPUs aiming for high throughput.
vLLM 0.27.0 is a significant upgrade, demonstrating continuous innovation in inference efficiency. Getting Kimi K3 support out of the box means I can immediately tap into the latest models with top-tier performance on my local GPU setup.
Ollama v0.32.7 Adds Initial Muse Glimmer Support, Optimized for Apple Silicon (Ollama)
Source: Ollama
Ollama's v0.32.7 release introduces initial support for Meta's Muse Glimmer model, a new open-source, agentic, and multimodal model. Notably, this initial implementation leverages Ollama's MLX engine, providing optimized performance specifically for Apple Silicon. This makes Muse Glimmer readily accessible for a significant segment of users equipped with consumer-grade Apple hardware, allowing them to run advanced multimodal AI agents directly on their Macs.
Muse Glimmer represents a push towards more capable local agents, capable of answering users and calling tools within a single generation. The focus on Apple Silicon for initial support highlights a commitment to broadening the accessibility of cutting-edge models for local inference, aligning with the blog's focus on consumer GPU compatibility. While future updates are expected to expand support and optimizations to NVIDIA, AMD, and other platforms, this release provides a crucial entry point for developers interested in exploring Muse Glimmer's multimodal agentic capabilities on a popular local platform.
Muse Glimmer's arrival in Ollama, especially with MLX optimization for Apple Silicon, is great for local multimodal experiments. It's fantastic to see these advanced models becoming available on consumer hardware right away.
Full Local AI & Open Models archive
GPU, CUDA & Autonomous Driving
This week, NVIDIA released Nemotron 3.5 Lightning and NeMo Switchyard for advanced AI agents, alongside JetPack 7.2.1 for Jetson devices. AMD introduced Instella-MoE, an open Mixture-of-Experts LLM optimized for Instinct MI300X, marking significant advancements in GPU-accelerated AI and model management.
NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI (NVIDIA Blog)
Source: NVIDIA Blog
NVIDIA announced significant updates for AI agents with the release of Nemotron 3.5 Lightning and NeMo Switchyard. Nemotron 3.5 Lightning is an expansion of NVIDIA's Nemotron 3 model family, designed for rapid and accurate specialized task execution in long-running AI agents. This model aims to improve efficiency in high-volume operations like tool calls, result validation, and subagent delegation, making AI agents more capable and robust for complex tasks. Developers can access these models for deployment on NVIDIA platforms, including RTX GPUs and DGX systems, catering to needs from local AI to enterprise-scale deployments.
Complementing Nemotron 3.5 Lightning is NeMo Switchyard, a framework engineered to route AI agent workloads intelligently across multiple models. This allows developers to leverage the unique strengths and cost profiles of different models dynamically, optimizing performance and resource utilization. With NeMo Switchyard, builders can construct more sophisticated and adaptable agents by orchestrating various specialized models, ensuring the right model is used for the right task at the right time. Both releases underscore NVIDIA's focus on advancing the ecosystem for autonomous AI agents, providing practical tools for developers to build, customize, and deploy AI locally and at scale.
These releases are crucial for anyone developing intelligent agents on NVIDIA hardware, offering both powerful foundational models and the tooling to manage their execution efficiently across diverse tasks.
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model (AMD ROCm Blog)
Source: AMD ROCm Blog
AMD has unveiled Instella-MoE, a new state-of-the-art, fully open Mixture-of-Experts (MoE) language model, marking a significant contribution to the open-source AI community. This model boasts a substantial architecture with 16 billion total parameters and 2.8 billion active parameters, offering a powerful option for developers seeking advanced LLMs. Notably, Instella-MoE was trained from scratch on AMD Instinct™ MI300X accelerators, demonstrating AMD's commitment to building and optimizing AI models for its hardware ecosystem. The release emphasizes the model's open nature, providing transparency and accessibility for researchers and developers.
Instella-MoE is designed to be highly efficient, leveraging the Mixture-of-Experts architecture to activate only a subset of parameters for any given input, leading to faster inference and reduced computational costs compared to dense models of similar total parameter count. This efficiency makes it particularly well-suited for demanding AI workloads on AMD's Instinct platform. The availability of such a large, open model from AMD directly supports the expansion of the ROCm software stack and its capabilities for high-performance AI, allowing users to harness the full potential of AMD's AI hardware for complex language understanding and generation tasks.
An open-source MoE LLM specifically trained on Instinct MI300X is a huge win for AMD's ROCm ecosystem, giving developers a direct path to high-performance AI on their hardware.
NVIDIA JetPack 7.2.1 Adds Agentic Video Skills and T3000 Emulation (NVIDIA Developer Blog)
Source: NVIDIA Developer Blog
NVIDIA has announced the release of JetPack 7.2.1, a significant update for its Jetson platform, which powers a wide range of edge AI applications including robotics, intelligent video analytics, industrial automation, and healthcare. This new version introduces "Agentic Video Skills," enhancing the capabilities of Jetson devices to process and understand video streams with more autonomy and intelligence. These skills are crucial for building advanced applications that can make decisions and take actions based on real-time video analysis, moving beyond traditional video processing to more sophisticated agent-based AI workflows directly at the edge.
A key technical highlight of JetPack 7.2.1 is the addition of T3000 emulation. This feature allows developers to simulate the behavior and performance characteristics of the NVIDIA T3000 GPU on Jetson platforms. Such emulation capabilities are invaluable for development and testing, enabling engineers to design and optimize applications for target hardware that might not be immediately available or for future deployments, streamlining the development cycle for AI-intensive embedded systems. The enhancements in JetPack 7.2.1 reinforce Jetson's role as a leading platform for deploying powerful, efficient, and intelligent AI solutions at the edge, leveraging NVIDIA's unified software stack.
JetPack 7.2.1's agentic video skills will accelerate autonomous decision-making in embedded systems, and T3000 emulation is a practical win for hardware-agnostic development on Jetson.
Full GPU, CUDA & Autonomous Driving archive
Compiled daily from official release feeds, vendor changelogs and engineering blogs. Archive: https://media.patentllm.org
Top comments (0)