DEV Community: KAMAL KISHOR

Pydantic for LLM Structured Output

KAMAL KISHOR — Thu, 12 Nov 2026 11:42:51 +0000

TL;DR: This article explains how to use Pydantic to get strictly structured JSON output from Large Language Models. Readers will learn to define Pydantic models, prompt LLMs for structured data, and validate the results for robust AI applications.

Integrating large language models (LLMs) into software systems often hits a snag: LLMs are fantastic at generating human-like text, but when your application needs a specific data format, like JSON, they can be unreliable. You might ask for JSON, only to get a slightly malformed string, a missing field, or an unexpected data type, leading to frustrating parsing errors and fragile integrations.

This is where Pydantic comes in. It's a Python library for data validation and settings management that allows you to define data schemas using standard Python type hints. When you combine Pydantic with LLMs, you can enforce strict output formats, ensuring that the data you receive is always predictable and programmatically usable. Frankly, I think most AI agent projects break in production because of weak validation and error handling, not because of the underlying model. This technique directly addresses that.

Setting Up Your Environment

First, you'll need Pydantic. If you don't have it, a quick pip command handles it. Pydantic v2.14.3 is the latest stable release at the time of writing, offering performance improvements and new features over v1.

pip install pydantic

For interacting with LLMs, you'll typically use an SDK like OpenAI's or a framework like LangChain. While the direct LLM interaction part isn't strictly Pydantic, the principle is the same: you'll instruct the LLM to return JSON that matches your defined schema.

Defining Your Output Schema

The core of this technique involves defining a Pydantic model that represents the exact structure and types you expect from the LLM. Think of this as a contract for your data.

Let's say you're building a system that extracts key details from product reviews. You might want the LLM to return the product name, the sentiment of the review, and any detected features.

from pydantic import BaseModel, Field
from typing import List, Literal, Optional

class ReviewAnalysis(BaseModel):
    product_name: str = Field(..., description="The name of the product being reviewed.")
    sentiment: Literal["positive", "negative", "neutral"] = Field(..., description="The overall sentiment of the review.")
    features_mentioned: List[str] = Field(default_factory=list, description="A list of product features explicitly mentioned in the review.")
    reviewer_rating: Optional[int] = Field(None, ge=1, le=5, description="The numerical rating given by the reviewer, if any, on a scale of 1 to 5.")

# You can even nest models for more complex structures
class ProductOverview(BaseModel):
    product_id: str
    analyses: List[ReviewAnalysis]

In this example, ReviewAnalysis specifies that product_name must be a string, sentiment can only be "positive", "negative", or "neutral", features_mentioned is a list of strings (defaulting to an empty list), and reviewer_rating is an optional integer between 1 and 5. Pydantic handles type checking, default values, and even value constraints (like ge=1, le=5 for "greater than or equal to 1" and "less than or equal to 5").

Guiding the LLM and Parsing Output

Once you have your Pydantic model, the next step is to tell the LLM exactly what you want. Many modern LLM APIs and frameworks (like LangChain) have built-in support for Pydantic or JSON Schema. They can automatically generate a schema from your Pydantic model and pass it to the LLM.

If your LLM integration doesn't have direct Pydantic support, you can still get a JSON schema from your model using ReviewAnalysis.model_json_schema() and include it directly in your prompt. This tells the LLM the exact JSON structure it should return.

Here's an illustrative sketch of how you might prompt an LLM and then parse its response (not using a specific LLM SDK, but showing the conceptual flow):

import json

# ... (ReviewAnalysis model definition from above) ...

# This is the prompt you'd send to the LLM
prompt = f"""
Analyze the following product review and extract the specified information as a JSON object.
Adhere strictly to the following JSON schema:
{ReviewAnalysis.model_json_schema_json(indent=2)}

Product Review: "This vacuum cleaner is amazing! The suction power is incredible (5/5 stars), and it's surprisingly quiet. The only downside is the battery life, which is a bit short. Definitely a positive experience overall."
"""

# Imagine 'llm_response_text' is the raw string output from your LLM
# Example of a good response:
llm_response_text_good = """
{
  "product_name": "Vacuum Cleaner",
  "sentiment": "positive",
  "features_mentioned": ["suction power", "quiet operation", "battery life"],
  "reviewer_rating": 5
}
"""

# Example of a potentially bad response (e.g., LLM hallucinated 'color')
llm_response_text_bad = """
{
  "product_name": "Vacuum Cleaner",
  "sentiment": "positive",
  "features_mentioned": ["suction power"],
  "color": "blue",
  "reviewer_rating": 5
}
"""

try:
    # Attempt to parse the LLM's raw text response using Pydantic
    parsed_data = ReviewAnalysis.model_validate_json(llm_response_text_good)
    print("Successfully parsed data:")
    print(parsed_data.model_dump_json(indent=2))
    print(f"Product name: {parsed_data.product_name}")
    print(f"Sentiment: {parsed_data.sentiment}")
    # You can now safely access attributes with correct types
except Exception as e:
    print(f"Validation Error: {e}")

try:
    # This will raise a ValidationError because 'color' is not in the schema
    parsed_data_bad = ReviewAnalysis.model_validate_json(llm_response_text_bad)
except Exception as e:
    print(f"\nCaught expected validation error for bad response: {e}") # cite: 12
    # Specific error handling for structured output issues is crucial,
    # as I discussed in [Validating AI Agent Outputs with Explicit Pass Conditions](https://dev.to/koolkamalkishor/validating-ai-agent-outputs-with-explicit-pass-conditions-5hm)

When using model_validate_json(), Pydantic attempts to parse the JSON string and then validates it against your ReviewAnalysis model. If the data types don't match, or if required fields are missing, or if extra unexpected fields are present, Pydantic raises a ValidationError with clear details. This means you get explicit failure signals instead of silent errors further down your processing pipeline.

A Practical Limitation

While Pydantic is powerful for ensuring structured output, it's not a magic bullet. The LLM still needs to be capable enough to follow the instructions and generate some form of valid JSON. Smaller or less capable models might struggle more consistently, leading to more frequent ValidationError exceptions. In such cases, you might need to iterate on your prompt engineering or consider using a more robust LLM. It's a trade-off between model capability and the strictness you need for downstream systems.

Sources

Setting Up Your Environment

pip install pydantic

Defining Your Output Schema

The core of this technique involves defining a Pydantic model that represents the exact structure and types you expect from the LLM. Think of this as a contract for your data.

Let's say you're building a system that extracts key details from product reviews. You might want the LLM to return the product name, the sentiment of the review, and any detected features.

from pydantic import BaseModel, Field
from typing import List, Literal, Optional

class ReviewAnalysis(BaseModel):
    product_name: str = Field(..., description="The name of the product being reviewed.")
    sentiment: Literal["positive", "negative", "neutral"] = Field(..., description="The overall sentiment of the review.")
    features_mentioned: List[str] = Field(default_factory=list, description="A list of product features explicitly mentioned in the review.")
    reviewer_rating: Optional[int] = Field(None, ge=1, le=5, description="The numerical rating given by the reviewer, if any, on a scale of 1 to 5.")

# You can even nest models for more complex structures
class ProductOverview(BaseModel):
    product_id: str
    analyses: List[ReviewAnalysis]

Guiding the LLM and Parsing Output

Here's an illustrative sketch of how you might prompt an LLM and then parse its response (not using a specific LLM SDK, but showing the conceptual flow):

import json

# ... (ReviewAnalysis model definition from above) ...

# This is the prompt you'd send to the LLM
prompt = f"""
Analyze the following product review and extract the specified information as a JSON object.
Adhere strictly to the following JSON schema:
{ReviewAnalysis.model_json_schema_json(indent=2)}

Product Review: "This vacuum cleaner is amazing! The suction power is incredible (5/5 stars), and it's surprisingly quiet. The only downside is the battery life, which is a bit short. Definitely a positive experience overall."
"""

# Imagine 'llm_response_text' is the raw string output from your LLM
# Example of a good response:
llm_response_text_good = """
{
  "product_name": "Vacuum Cleaner",
  "sentiment": "positive",
  "features_mentioned": ["suction power", "quiet operation", "battery life"],
  "reviewer_rating": 5
}
"""

# Example of a potentially bad response (e.g., LLM hallucinated 'color')
llm_response_text_bad = """
{
  "product_name": "Vacuum Cleaner",
  "sentiment": "positive",
  "features_mentioned": ["suction power"],
  "color": "blue",
  "reviewer_rating": 5
}
"""

try:
    # Attempt to parse the LLM's raw text response using Pydantic
    parsed_data = ReviewAnalysis.model_validate_json(llm_response_text_good)
    print("Successfully parsed data:")
    print(parsed_data.model_dump_json(indent=2))
    print(f"Product name: {parsed_data.product_name}")
    print(f"Sentiment: {parsed_data.sentiment}")
    # You can now safely access attributes with correct types
except Exception as e:
    print(f"Validation Error: {e}")

try:
    # This will raise a ValidationError because 'color' is not in the schema
    parsed_data_bad = ReviewAnalysis.model_validate_json(llm_response_text_bad)
except Exception as e:
    print(f"\nCaught expected validation error for bad response: {e}")
    # Specific error handling for structured output issues is crucial,
    # as I discussed in [Validating AI Agent Outputs with Explicit Pass Conditions](https://dev.to/koolkamalkishor/validating-ai-agent-outputs-with-explicit-pass-conditions-5hm)

A Practical Limitation

Sources

Written by **Kamal Kishor* (@koolkamalkishor) — Sr. Frontend Developer building AI agents (React, TypeScript, LangChain). LinkedIn · GitHub · Portfolio*

Published: 12 Nov 2026. This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

CocoIndex for Fresh RAG Data: A Step-by-Step Guide

KAMAL KISHOR — Wed, 11 Nov 2026 11:22:41 +0000

TL;DR: This article explains CocoIndex, a tool that automates incremental data updates for AI pipelines, saving computation by only reprocessing changed inputs. You will learn how to set it up and use its declarative Python API to keep your RAG and AI agent data current.

If you're building RAG systems or AI agents, you know the pain of stale data. Your agent's performance hinges on fresh information, but fully reprocessing entire datasets or embeddings every time a small change occurs is slow, resource-intensive, and frankly, unnecessary. This is where tools that handle incremental updates become critical.

CocoIndex tackles this directly. It provides a declarative Python API, powered by a Rust engine, to automate these incremental updates. This means it tracks your function inputs and outputs, re-running only the parts of your pipeline that are affected by actual changes. It's particularly useful for keeping embeddings up-to-date for RAG and agent workflows, even supporting local embedding models like those from Ollama or sentence-transformers without needing powerful GPUs. I prefer free, local AI tools over paid subscriptions when the quality is good enough, so seeing this kind of local support is a big plus for me, especially since I've written about running local LLMs with Ollama before.

Setting Up CocoIndex

Getting started with CocoIndex is straightforward. You'll need Python installed. A dedicated virtual environment is always a good idea to manage your dependencies.

First, create and activate a virtual environment:

python -m venv venv
source venv/bin/activate # On Windows use `venv\Scripts\activate`

Next, install CocoIndex using pip. The official documentation provides the exact command.

pip install cocoindex

This command pulls in the CocoIndex library, ready for use in your Python projects.

Using CocoIndex for Incremental Updates

Let's look at a quick example from the official documentation to see CocoIndex in action. Imagine a function that processes some input data. We'll use a simple addition, but in a real RAG pipeline, this could be an embedding generation function or a data transformation step.

Create a Python file, say incremental_example.py:

import coco

@coco.index
def add_numbers(a: int, b: int) -> int:
    print(f"Adding {a} and {b} (this should only run on input change)")
    return a + b

# First run: Should print the message
result1 = add_numbers(5, 3)
print(f"Result 1: {result1}")

# Second run with same inputs: Should use cached result, no print
result2 = add_numbers(5, 3)
print(f"Result 2: {result2}")

# Third run with changed inputs: Should print the message again
result3 = add_numbers(7, 3)
print(f"Result 3: {result3}")

# To clear the cache for a fresh start or testing
coco.clear_cache()
print("Cache cleared.")

# Fourth run after cache clear: Should print the message again
result4 = add_numbers(5, 3)
print(f"Result 4: {result4}")

Now, run this script:

python incremental_example.py

You'll observe that the print statement inside add_numbers only executes when the inputs change or the cache is explicitly cleared. The second call to add_numbers(5, 3) reuses the previously computed result without re-running the function, demonstrating the core incremental update mechanism.

Common Considerations

While CocoIndex simplifies incremental processing, there are a few things to keep in mind. The tool shines when your data pipelines involve expensive computations (like generating embeddings) where re-running everything is a significant time or cost burden. For very small datasets that change constantly and completely, or for functions that are inherently fast to recompute, the overhead of tracking changes might not always be worth the benefit. You also need to ensure that your functions are deterministic; given the same inputs, they should always produce the same output for CocoIndex to function correctly.

If you encounter unexpected behavior, like a function always re-running even with seemingly identical inputs, consider using coco.clear_cache() to reset the state and verify your inputs are truly identical. This is often helpful during development or debugging.

CocoIndex automates a common problem in data-intensive AI applications, allowing developers to focus on building features rather than managing complex caching logic. For more on building RAG applications, you might find my article on Building a RAG (Retrieval-Augmented Generation) Application Using Deep Seek R1 from Scratch useful.

Sources

CocoIndex Documentation: Quickstart. https://cocoindex.io/docs/getting_started/quickstart

Written by **Kamal Kishor* (@koolkamalkishor) — Sr. Frontend Developer building AI agents (React, TypeScript, LangChain). LinkedIn · GitHub · Portfolio*

Published: 11 Nov 2026. This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

OpenJarvis: Local-First AI Agents for Developers

KAMAL KISHOR — Tue, 10 Nov 2026 11:34:13 +0000

TL;DR: OpenJarvis is an open-source framework designed to build AI agents that run primarily on your local device rather than relying heavily on cloud APIs. It focuses on privacy, cost efficiency, and low latency by enabling on-device inference and providing tools to measure energy and computational perfor

The current state of AI agents often means your "personal" AI relies heavily on external cloud APIs. This setup frequently leads to privacy concerns because your data leaves your device, drives up operational costs, and introduces latency as requests travel back and forth. OpenJarvis tackles these issues head-on by making local-first AI a practical reality for developers.

What OpenJarvis Is

OpenJarvis is an open-source framework for building personal AI agents designed to run directly on your hardware, with cloud APIs becoming an optional fallback. Developed by researchers at Stanford University and Lambda Labs, it's a foundational stack that prioritizes on-device inference, memory, and learning. It defines five core primitives: Intelligence (for model management), Engine (for inference runtime), Agents (for multi-step reasoning and tool use), Tools & Memory (for data integration and persistent storage), and Learning (for system optimization). This modular design helps developers avoid complex, hard-to-reproduce codebases often seen in local AI projects.

The project emerges from the "Intelligence Per Watt" research initiative, which found that local language models can already handle a significant portion (88.7%) of single-turn chat and reasoning queries at interactive latencies, with intelligence efficiency improving rapidly. OpenJarvis aims to be both a research platform and a production-ready foundation for local AI, much like PyTorch for deep learning.

How It Works

OpenJarvis operates by making local inference the default. It supports a variety of local inference engines like Ollama, vLLM, SGLang, and llama.cpp, and can also integrate with cloud APIs if necessary. The framework uses a "spec," which is a declarative configuration object that composes its five primitives. This spec defines everything from the choice of model and inference runtime to agent logic and tool-use policies.

A key innovation is its focus on efficiency-aware evaluation. OpenJarvis treats energy consumption, FLOPs, latency, and dollar cost as first-class constraints alongside traditional accuracy metrics. It includes a hardware-agnostic telemetry system that profiles energy usage across different GPUs (NVIDIA, AMD, Apple Silicon) and provides a jarvis bench command for standardized benchmarking of latency, throughput, and energy per query. This means you can tangibly see what each query costs in terms of energy and compare it to cloud alternatives, with reports showing up to 800x lower marginal API cost and 4x lower end-to-end latency for local configurations compared to cloud baselines like Claude Opus 4.6.

For instance, installing OpenJarvis typically involves a single command that sets up dependencies, Ollama (if not present), and a starter model, then launches a local UI. You can then interact with it via a CLI or a browser-based graphical interface. For a local coding agent, you might use a command like jarvis init --preset code-assistant, configuring it to write and run Python code on your device.

I've personally found that the ability to run powerful LLMs locally with tools like Ollama is a game-changer for developer productivity, as I highlighted in my article on running Llama 3.2 on Android: A Step-by-Step Guide Using Ollama. The promise of OpenJarvis, with its structured approach to on-device AI and built-in efficiency metrics, aligns perfectly with my preference for free, local AI tools over paid cloud subscriptions, especially after replacing a paid coding assistant with a local alternative, which I documented in I Replaced Cursor with a Free Local AI Coding Assistant (And Saved $240/Year).

Who Should Care

Developers building AI agents and applications where privacy is paramount should pay close attention to OpenJarvis. By keeping inference and data on-device, it offers a verifiable privacy-by-design approach, eliminating concerns about personal data routing through third-party servers. If you're building applications that handle sensitive user information, this is a significant advantage.

Secondly, anyone looking to reduce operational costs for AI workloads will find OpenJarvis compelling. The framework's ability to run models locally drastically cuts down on marginal API costs, potentially saving hundreds of dollars a month depending on usage. This is particularly relevant for projects with high query volumes.

Thirdly, developers aiming for low-latency AI interactions will benefit. Local execution removes network overhead, leading to significantly faster response times, which is crucial for interactive applications and real-time agentic workflows.

Finally, OpenJarvis is for those who appreciate a modular, inspectable, and production-ready foundation for personal AI. Its five-primitive architecture and emphasis on benchmarked performance (including energy efficiency) provide a robust platform for building reliable on-device agents. This framework is ideal for creating agents that handle tasks like morning briefs, code review, email triage, or deep research, all running 24/7 on local hardware.

One honest limitation to consider is the initial hardware requirement. While OpenJarvis strives for efficiency, running larger models locally still demands a capable device with sufficient RAM and GPU VRAM. Tiny local models are suitable for testing, but serious agentic workflows often require more robust local hardware, such as a Linux box with vLLM for real use cases. If your target users universally have very limited compute resources and simpler models aren't sufficient, a purely local-first approach might face adoption hurdles without some cloud fallback or client-side model optimization.

Sources

Written by **Kamal Kishor* (@koolkamalkishor) — Sr. Frontend Developer building AI agents (React, TypeScript, LangChain). LinkedIn · GitHub · Portfolio*

Published: 10 Nov 2026. This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

Locally Uncensored AI Desktop App: Run DeepSeek v4.1 Flash Locally

KAMAL KISHOR — Mon, 09 Nov 2026 11:59:29 +0000

TL;DR: This guide walks you through setting up the Locally Uncensored AI Desktop App, which brings the DeepSeek v4.1 Flash model to your desktop. You'll learn how to get a multi-modal AI agent running locally for free, without complex setups or cloud dependencies.

Dealing with cloud AI services often means worrying about data privacy, rate limits, or ongoing subscription costs. For developers, this can be a real friction point when you just need a capable AI agent for quick coding tasks, content generation, or local data processing without sending everything off your machine.

That's where the Locally Uncensored AI Desktop App comes in. It's a free, open-source application that brings the DeepSeek v4.1 Flash model directly to your Windows or Linux desktop. This isn't another CLI tool or a complex Docker setup; it's a straightforward desktop application designed for ease of use. It handles chat, image and video generation, and coding assistance entirely on your local hardware. As someone who prefers free, local AI tools when the quality is good enough, I've found that cutting down on cloud dependencies makes a lot of sense, especially for sensitive data or iterative development. I even wrote about how I replaced a paid cloud service with a free local coding assistant.

Getting the Application Running

First, you'll need the application itself. Head over to the official download page to get the installer for your operating system.

The app is available as a desktop installer for Windows (NSIS .exe or .msi) and as AppImage, .deb, and .rpm packages for Linux. You can find these on the project's GitHub Releases page. The core idea behind Locally Uncensored is that local AI should install like a normal program, without requiring command lines, Docker, or configuration files.

To get started:

Download the installer: Visit the official download page at https://locallyuncensored.app/download and select the appropriate package for your operating system.
Run the installer:
- Windows: Double-click the downloaded .exe or .msi file and follow the on-screen prompts like any other software installation.
- Linux: For an AppImage, you might need to make it executable (chmod +x YourApp.AppImage) and then run it. For .deb or .rpm packages, use your distribution's package manager (e.g., sudo dpkg -i your-package.deb for Debian/Ubuntu or sudo rpm -i your-package.rpm for Fedora/RHEL).
Initial Setup Wizard: On the first launch, a setup wizard will appear. This wizard automatically detects if an AI engine (like Ollama or LM Studio) is already running on your machine. If not, it can install one for you with a single click. It also handles the setup for ComfyUI, which the app uses for image and video generation.
Pick a model: Open the model manager within the app. It will show you which models are compatible with your hardware and allow you to download them in one click. If you already use an engine like Ollama or LM Studio, it can link to models already stored by those tools without duplicating them. The DeepSeek v4.1 Flash model will be available here.

Using the App for AI Tasks

Once the app and a model like DeepSeek v4.1 Flash are installed, you'll find different tabs in the main window: Chat, Create, Code, and Agent. The models you've picked carry across these different functionalities.

Quick Example: Coding Assistance
To use it for coding, navigate to the "Code" tab. You can paste code snippets, describe a problem, or ask for help with refactoring. For instance, you could paste a React component and ask: "Refactor this React component to use React hooks for state management and reduce boilerplate." The DeepSeek v4.1 Flash model will process your request locally and provide a revised code snippet. The app even offers a file explorer with preview and a workspace per project.

Common Issues and Limitations

While the Locally Uncensored AI Desktop App aims for simplicity, you might still encounter some common issues.

A primary concern for any local AI application is hardware compatibility. DeepSeek v4.1 Flash, while efficient, still requires sufficient CPU, RAM, and especially GPU VRAM to run effectively. The model manager in the app is designed to help by indicating which models fit your hardware. If a model says "Needs more VRAM," it might run slowly or struggle. I've found that ensuring your GPU drivers are up-to-date can often resolve performance issues with local LLMs, as I've covered in my guide on running Llama 3.2 on Android.

Another common problem can be related to download failures when fetching models. The app includes a download manager with features like pausing, canceling, and resuming, which uses HTTP Range headers for proper resumable downloads. This helps prevent restarting large downloads if your connection drops.

It's also worth noting an honest trade-off: while it offers chat, image/video generation, and coding assistance in one place, for highly specialized tasks, dedicated tools might offer more advanced features or finer control. For example, if you're deep into complex image generation workflows, a standalone ComfyUI installation might provide more granular control than the integrated version. However, for most day-to-day developer needs, the convenience of an all-in-one local studio like this often outweighs the slight loss of deep customization.

Sources

Locally Uncensored AI App - Download & Setup
Locally Uncensored download | SourceForge.net
GitHub - PurpleDoubleD/locally-uncensored: The all-in-one local AI studio for your desktop: chat, image and video generation and a coding agent in one free, open source app. Windows and Linux. No Docker, no terminal, no cloud required.
How to Run AI Locally in 2026: 5-Minute Beginner Guide
Locally Uncensored v1.5.5 — Zero-Config AI Models, VRAM Detection & Production-Ready Desktop App - DEV Community
Install, Chat, Agent, Code, Create, Cloud - Locally Uncensored Handbook
Plug and Play Local AI: One Installer, Zero Setup - Locally Uncensored

Written by **Kamal Kishor* (@koolkamalkishor) — Sr. Frontend Developer building AI agents (React, TypeScript, LangChain). LinkedIn · GitHub · Portfolio*

Published: 09 Nov 2026. This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

Prompt Chaining for LLMs: Build Reliable Workflows

KAMAL KISHOR — Sun, 08 Nov 2026 10:42:12 +0000

Liquid syntax error: 'raw' tag was never closed

NVIDIA Nemotron 3.5 Bolt E4B CPU: Run Locally with Ollama

KAMAL KISHOR — Sat, 07 Nov 2026 10:37:06 +0000

TL;DR: This guide explains how to get NVIDIA Nemotron 3.5 Bolt E4B running on your laptop's CPU using Ollama. You'll learn the setup and commands to enable local multimodal AI vision capabilities without needing a dedicated GPU.

Running advanced AI models often feels like an exclusive club, requiring expensive GPUs and specialized hardware. But what if you want to experiment with a recently released vision-language model, like NVIDIA's Nemotron 3.5 Bolt E4B, right on your standard laptop's CPU? This is where Ollama steps in, making powerful multimodal AI accessible locally.

Getting Ready

To run NVIDIA Nemotron 3.5 Bolt E4B locally, you'll need a machine running Linux, macOS, or Windows with Ollama installed. The key here is Ollama version 0.40.15 or newer, as this release added support for the Nemotron 3.5 Bolt E4B model. Since we're targeting CPU-only execution, adequate RAM is important; while exact figures can vary based on context size, aiming for 16GB or more is a good starting point for a smooth experience.

As someone who leans towards free, local AI tools when their quality is good enough, this kind of development is exactly what I appreciate. My past experiences, including running models like Llama 3.2 on Android with Ollama, have shown me the value of having these capabilities without relying on cloud APIs or dedicated hardware.

Setup and Running the Model

First, ensure you have Ollama installed. If you don't, you can grab it directly from the official Ollama website. The installation process is straightforward, typically involving a single command for Linux/macOS or a dedicated installer for Windows.

For macOS:

curl -fsSL https://ollama.com/install.sh | sh

For Linux:

curl -fsSL https://ollama.com/install.sh | sh

For Windows, download the installer from the Ollama website.

Once Ollama is installed and running, you can pull the Nemotron 3.5 Bolt E4B model. This command will download the model weights to your local machine:

ollama pull nemotron-3-5-bolt-e4b

After the model is downloaded, you can run it. Nemotron 3.5 Bolt E4B is a vision-language model, meaning it can process both text and images. You'll interact with it via the command line, providing a prompt and optionally an image path.

Here's a quick example. Let's say you have an image file named my_image.jpg in your current directory.

ollama run nemotron-3-5-bolt-e4b
>>> What is in this image? /path/to/my_image.jpg

The model will then process the image and your text prompt, providing a textual response describing the image content. You can continue the conversation, asking follow-up questions about the image or entirely new text-based queries. If you want to understand more about how Ollama handles multimodal inputs, I've covered the general concept previously in Ollama Multimodal Models: Run Vision AI Locally.

Common Issues and Considerations

Running vision-language models like Nemotron 3.5 Bolt E4B on a CPU comes with inherent trade-offs, primarily in performance. While it's fantastic for local experimentation and development, don't expect real-time responses comparable to GPU-accelerated inference. Depending on your CPU's power and the size of the image or complexity of the prompt, responses might take several seconds or even minutes.

Another common issue is insufficient RAM. Even without a dedicated GPU, LLMs and VLMs consume significant system memory. If your system frequently crashes or the ollama run command fails, check your memory usage. You might need to close other applications or upgrade your RAM.

This approach is particularly valuable for developers who need to integrate multimodal AI into applications where privacy is critical, or for those in environments with limited or no internet access. However, for production systems requiring high throughput, low latency, or processing very large batches of data, investing in GPU hardware or cloud-based GPU instances would still be the more practical route. For local development and testing, especially when iterating on prompts or fine-tuning application logic, CPU-only execution with Ollama is a robust and cost-effective solution.

Sources

Ollama v0.40.15 Release Notes: "Added nemotron-3-5-bolt-e4b" (https://github.com/ollama/ollama/releases/tag/v0.40.15)
Ollama Installation Guide (https://ollama.com/download)

Written by **Kamal Kishor* (@koolkamalkishor) — Sr. Frontend Developer building AI agents (React, TypeScript, LangChain). LinkedIn · GitHub · Portfolio*

Published: 07 Nov 2026. This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

llm-d 1.0: Scalable LLM Inference for Production

KAMAL KISHOR — Fri, 06 Nov 2026 11:22:51 +0000

TL;DR: llm-d 1.0 is a new open-source project under the CNCF designed to make LLM inference scalable and cost-effective in production. It aims to reduce operational costs by 5-10x while handling complex agentic and RAG workloads.

Getting Large Language Models into production reliably and affordably is still a major headache for many teams. The challenges compound when you're dealing with diverse workloads, trying to serve multiple users from shared infrastructure, or facing unpredictable bursts of activity. Standard microservice patterns often break down here because LLM inference is highly stateful and computationally intensive, with costs soaring as demand increases.

What llm-d 1.0 Is

llm-d 1.0 is a significant open-source release that tackles these exact problems for production-scale LLM inference. Announced on October 20, 2026, by project contributors including IBM, Red Hat, Google Cloud, CoreWeave, and NVIDIA, it marks a major step towards making distributed LLM inference a standard, cloud-native capability. The project, founded in 2025, is governed by the Cloud Native Computing Foundation (CNCF) and was accepted into the CNCF Sandbox maturity level on March 12, 2026.

Essentially, llm-d 1.0 provides a hardened core for distributed inference, validated for day-2 production deployments. It helps organizations improve AI serving efficiency and scale enterprise workloads, with reported cost reductions of 5-10x compared to all-proprietary configurations in some use cases. In one IBM Research example, using open frontier models and llm-d enabled an 11.5 times lower inference cost. This project is a vendor-neutral alternative aimed at enterprises seeking high-performance inference without being locked into a single provider or hardware platform.

How llm-d 1.0 Works

llm-d 1.0 is engineered to bring sustained performance to the realities of running inference in production. It supports serving multiple tenants, balancing competing priorities, providing zero-downtime features, and handling emerging agentic workloads within a co-designed suite of components.

At its core, llm-d's architecture revolves around three concepts: the Router, the InferencePool, and the Model Server. The llm-d Router acts as the intelligent entry point for inference requests, providing LLM-aware load balancing, request queuing, and policy enforcement. It includes an Endpoint Picker (EPP) that scores and selects model server pods based on real-time metrics, KV-cache affinity, and configured policies.

A key aspect is its comprehensive ecosystem for managing and reusing the KV cache across the inference pool. This includes prefix-cache aware routing to maximize cache hits, KV-cache indexing for tracking cache state, and KV offloading to tiered storage (CPU, SSD) to extend capacity. For latency-tolerant traffic, llm-d 1.0 also introduces llm-d-async-processor, which routes requests from queues alongside real-time traffic, using queue and pool-dispatch gates to enforce priorities and quotas.

It supports proactive, SLO-aware autoscaling using KEDA and EPP metrics, which track signals like queue depth and token backlog. Furthermore, it handles "disaggregated serving," where an inference request is split into phases (like prefill and decode) handled by specialized workers, orchestrated by the Router. This design helps prevent bursty agent loops from starving interactive users by intelligently queuing and prioritizing requests.

Who Needs to Care

If you're building or operating AI agents and RAG (Retrieval-Augmented Generation) systems at scale, you should pay close attention to llm-d 1.0. The project directly addresses the cost and performance bottlenecks often faced when operationalizing LLMs, especially with context-heavy, bursty agentic workloads. As a Sr. Frontend Developer currently building an AI-powered SDR agent platform and learning RAG architectures, I find the focus on multi-tenant QoS and efficient handling of complex, dynamic workloads particularly relevant. Most AI agent projects struggle in production due to issues like weak validation, but the underlying inference infrastructure's stability and cost are equally critical. Projects like llm-d that enable robust, cost-effective inference directly impact the feasibility of deploying advanced agentic systems.

Organizations that need to deploy inference stacks across various hardware (NVIDIA, AMD, Intel, Google TPUs) or cloud environments will find llm-d appealing due to its vendor-neutral approach. If you're looking for an open-source alternative to managing complex AI infrastructure that emphasizes efficiency, scalability, and predictable performance, llm-d 1.0 is worth exploring.

A Practical Consideration

While llm-d 1.0 offers substantial benefits for scaling LLM inference, it's not a magic bullet that eliminates all operational complexity. Being a Kubernetes-native project, adopting it means embracing or deepening your team's expertise in Kubernetes orchestration, custom resources, and cloud-native practices. The 5-10x cost reduction is compelling, but achieving it requires careful configuration, understanding of its architecture, and integrating it into your existing CI/CD and monitoring pipelines. For smaller projects or those with very low inference volume, the overhead of setting up and managing a distributed inference network like llm-d might outweigh the cost savings of using a simpler, managed LLM API. The greatest benefits will likely be seen by enterprises and teams with significant, diverse, and scaling LLM inference requirements.

Sources

Written by **Kamal Kishor* (@koolkamalkishor) — Sr. Frontend Developer building AI agents (React, TypeScript, LangChain). LinkedIn · GitHub · Portfolio*

Published: 06 Nov 2026. This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

Constrained Direct Answering: Get Precise LLM Outputs

KAMAL KISHOR — Thu, 05 Nov 2026 11:25:40 +0000

TL;DR: This article explains Constrained Direct Answering, a technique for getting precise, non-reasoning responses from LLMs. Readers will learn how to implement it to achieve more accurate and reliable AI outputs.

You've probably seen it: you ask an LLM a simple question, expecting a single word or a concise phrase, and it responds with a paragraph of context, a brief essay on the topic, or even a polite disclaimer. This "thinking inertia," where the model feels compelled to elaborate, is a common headache when you need truly precise, direct answers for automated systems or data extraction. It costs tokens, adds latency, and complicates downstream parsing.

This is where Constrained Direct Answering comes in. It's a technique to explicitly guide an LLM to deliver only the requested information, free from any preamble, explanation, or conversational fluff. It essentially tells the model to bypass its typical reasoning output and go straight for the target.

Why Direct Answers Matter

For many applications, especially in AI agent systems, you don't need a conversational partner; you need a data point. When I'm building AI-powered SDR agent platforms, for example, for tasks like lead qualification or personalizing outreach, I often need a specific entity, a sentiment score, or a binary "yes/no" answer. Extra text in these scenarios isn't just annoying; it can break your parsing logic or inflate API costs. I've found that weak validation and error handling on these verbose outputs are often why AI agent projects falter in production, not the models themselves. This technique helps shore up that initial output quality.

Implementing Constrained Direct Answering

The core idea is to be brutally explicit in your prompt, leaving no room for the LLM to interpret your request broadly. You're effectively boxing the model into a very specific response format.

Here’s a step-by-step approach you can use today:

State Your Intent Clearly: Begin your prompt by explicitly telling the LLM to output only the answer and nothing else.
- "Answer ONLY with..."
- "Respond with the [X] value and nothing else."
- "Your entire response must be [Y]."
Define the Output Format: If you need a specific structure, like a JSON object, a comma-separated list, or a single integer, specify it.
- "Return a JSON object with a single key 'city' and its value."
- "List the items as a comma-separated string."
Add Negative Constraints: Tell the LLM what not to do. This reinforces the directness.
- "Do NOT provide any explanation."
- "No preamble, no conversational filler, no concluding remarks."
- "Do not apologize or offer additional information."

An Illustrative Sketch (not from any project's docs):

Here's how this might look in a simple Python snippet using a hypothetical LLM API call:

import os
# Assume 'llm_api_client' is an initialized client for your LLM service
# and 'LLM_MODEL' is your target model, e.g., "gemini-1.5-flash"

def get_direct_answer(question: str, model_name: str = os.getenv("LLM_MODEL", "your_default_model")):
    prompt = (
        f"QUESTION: {question}\n\n"
        "INSTRUCTIONS: Your entire response must be ONLY the direct answer to the question. "
        "Do NOT provide any explanations, preamble, or conversational text. "
        "Do NOT apologize or offer additional information. Respond concisely."
    )

    try:
        # This is a placeholder for your actual LLM API call
        # Example using a hypothetical client. Replace with your LangChain, OpenAI, Anthropic, etc., call.
        response = llm_api_client.generate(model=model_name, prompt=prompt, max_tokens=50) # Keep max_tokens low for brevity
        return response.text.strip()
    except Exception as e:
        print(f"Error calling LLM: {e}")
        return None

# Example Usage:
print(get_direct_answer("What is the capital of France?"))
# Expected: Paris
print(get_direct_answer("Is 17 a prime number? Answer ONLY with 'Yes' or 'No'."))
# Expected: Yes

By keeping max_tokens low, you can also implicitly encourage brevity, though direct instruction is always more reliable.

When Constrained Direct Answering is Not Worth It

This technique shines when you need objective, non-reasoning responses: extracting specific facts, classifying inputs, or performing quick lookups. For instance, when validating agent outputs with explicit pass conditions, getting a "True" or "False" directly is far more useful than an explanation.

However, if your task requires the LLM to think aloud, explain its reasoning, or generate creative and varied outputs, then Constrained Direct Answering is counterproductive. Don't use it for summarizing complex articles, generating marketing copy, or when you actually want the LLM's thought process visible for debugging or audit. It's about precision, not deep thought.

Sources

Written by **Kamal Kishor* (@koolkamalkishor) — Sr. Frontend Developer building AI agents (React, TypeScript, LangChain). LinkedIn · GitHub · Portfolio*

Published: 05 Nov 2026. This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

Ollama Local TTS: Audio with LFM3-Audio on Linux & Windows

KAMAL KISHOR — Wed, 04 Nov 2026 11:26:33 +0000

TL;DR: This guide demonstrates how to run local text-to-speech using Ollama and the LFM3-audio model from Liquid AI on Linux and Windows. Readers will learn the exact commands to get high-quality speech output directly on their machines without cloud dependencies.

Dealing with cloud text-to-speech (TTS) APIs can be a real pain. You're constantly thinking about latency, data privacy, and the monthly bill that creeps up. Building multimodal agents that need to speak without an internet connection, or just wanting to add an accessible voice feature to a local app, often means a compromise: either use lower-quality local options or pay for external services.

Ollama's recent v0.40.13 update changes that, bringing Liquid AI's LFM3 family of models, including lfm3-audio, to Linux and Windows users. This extends the high-performance local inference capabilities previously available only on Apple Silicon, making robust text-to-speech accessible on standard workstations. This is a compact, fully-featured 2B parameter audio model that can run locally on laptops and edge devices. It takes text or audio as input and can respond with text, speech, or both. As someone who prefers free, local AI tools when the quality is good enough, this update is a big win for offline development and reducing cloud reliance.

Getting Started with LFM3-audio

First, ensure you have Ollama installed and updated to at least version 0.40.13. You can check your version by running ollama --version. If you need to update, the process is straightforward:

# If installed via Homebrew (macOS)
brew update && brew upgrade ollama

# If installed via script (Linux)
curl -fsSL https://ollama.com/install.sh | sh

# For Windows, download the latest installer from ollama.com/download

Once Ollama is up to date, pulling the lfm3-audio model is a single command:

ollama pull lfm3-audio

This command downloads the 2.1 GB model to your local machine.

Running Text-to-Speech

With the model downloaded, you can now generate speech directly from your terminal. The ollama run command plays the audio through your computer's speakers.

To generate speech from a simple text prompt:

ollama run lfm3-audio "Hello from your local text-to-speech model!"

The model can handle various inputs and tasks, including multilingual speech synthesis in 18 languages. To specify a language and gender for the voice, you can use a system prompt:

ollama run lfm3-audio "Perform TTS in English using a female voice. Hello, how are you today?"

The lfm3-audio model also supports streaming replies in text and speech for voice chat applications.

You can also save the audio output to a WAV file instead of playing it directly. This is particularly useful for integrating TTS into applications or when working on a server without speakers:

ollama run lfm3-audio "This text will be saved to a file." > speech.wav

For programmatic access, Ollama provides an OpenAI-compatible /v1/audio/speech endpoint. You can interact with it using curl to generate audio files:

curl http://localhost:11434/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{ "model": "lfm3-audio", "input": "This is an example from the API.", "voice": "default" }' \
  --output api_speech.wav

Common Considerations

While running local TTS with Ollama and LFM3-audio is powerful, there are a few things to keep in mind. The lfm3-audio model handles up to 32K tokens for context length (input + output). Longer texts might need to be split, though ollama run often handles this automatically at paragraph and sentence breaks. Also, be aware that numbers, prices, and URLs are often spoken in expanded form, and markdown symbols or emojis are typically skipped.

A potential trade-off with local models is the initial download size and the computational resources required. While lfm3-audio is a compact 2B parameter model, a 2.1 GB download might take a little time depending on your internet connection. On the resource front, it's designed to run on laptops and edge devices, but its performance will still scale with your hardware, especially if you're running other models concurrently. For agentic coding or complex multi-step reasoning, LFM3-Audio-2B is not recommended due to its small size; it's better suited for local agents, tool calling, structured outputs, and domain-specific fine-tuning.

Sources

Ollama v0.40.13: LFM3 Support for Windows & Linux | Laravel - Richness. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHBXXOtztFnnwqO0rHuU3SSgDBx5aRvMsHMQQbW4TPtngLAaUBVsqX3wjCqY51DBKn3Z7gUQ-_GfsdTGqapMHenmu61th7Ux6CAumvoRcLlEL5hRIf55cHLox8qYBpFo-7qUQ6oAmkjP9ozbGXY41cxWmoe9naurlergXpmGtzufCc=
LFM3-Audio-2B - Liquid AI Docs. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEV-zYGdK8LbCKGJofCjRwGsyEAmjWCTpAl7NmM-TGLJ8kjN89v0gd9GKCj82J4TH4MBTDkkl2WDCw1NB6jVpK6D8qUu1nTK-HVFeqNHD_NQORrtBk5fyC4SjsTdIEYgv8G3uKFvQMsK2zF
lfm3-audio - Ollama. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHyai-ylXSU4fyjLPYvtSCsou5pQ8SrxttGGrC4KvZWKk4FEQIcFyaevKHD9g7kGGphMBv5F-xKwVWpAVcRPtwBqkIQpNKrjn5YPh-dovgqqnFhXePuCt9cVL20Y25-VIM=
Ollama 0.40.13: Speech Endpoint, d3 Models and LFM3 - ThreatFrontier.com. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEGBMHXBcI7Jr0akcbuU2cwG4pq3rIUCCdRIUp7n7VL98V6y3Oi00X4Wax0cz-xsl7vgwNSiJG0udBFQ_sJ0PQeIEj8pJ_LUnmr33_dhJ6j6JpLj8uOwh1TAQYWV-d_wD_da88zUr_yh7MlsiUChwOakQqJNkl7r2inZrigJuhlvQ3MOraGhDzoWWNaS1I=
Ollama Release Notes - November 2026 Latest Updates - Releasebot. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHuk_AL28rZMGb2LVRjhOd1YhwBoSSCzkTuXfI84djIjfQcnGFSoq35wahviPAOz9Nt-dWf53_T0EuBROTWtHBKWFDNyGxQznzAUvM2mTXYJPZnB88mpb_DJxWxeXVi0g==
LiquidAI/LFM3-Audio-2B - Hugging Face. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGxJG4idjl9X24S5pe6qipsTKLFEHZMeJDgU-8k0S0IpXVnKAgvQBnl-PIye2bJgBfInx7DlU12r2igT_dekrBEVtBOwEFtlX-WZKJ-VduuaUcwPIykGWIp1bFgNATOvYSNElthwrBe_g==
New audio model: LFM3-Audio-2B : r/LocalLLaMA - Reddit. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHTkWPuGTpvwK3hUOBhtfipAW7K5xvuI16Rv7_WE9VYg6wwkRJUyaoYkg1cpqu5NOcPjVrn3hx3bbmH2RpbMukpnMJ52nSDmjYsaHRgzsGL0YHkGo2F8XwvSpBnQYFMTWT3IxoQQ_c9MaM8AyOtv16jBFUUvMpykkgg6nqVN6c83i2fRY4VR3wKJyXZ-A==
LFM3-Audio-2B on Ollama: Better Speech, the Same Ears - OrcaRouter. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEd7S_nlNcf-8Pwd-zZwixTa0De-ofd3GiSEns1oHLdIzuY51_DFIuBK2S9iXBxziVqC0iQKodDFpNmEG9oGvAMECHvRTRD7sH3-AGWRjiWLWZxrDPu_rxVTmgZ9m9CEoLXT89RlgIcLJALMnHZPNTsNOfXEken4-ABy8=
Ollama Speaks! How to Use the New Local Text-to-Speech AI Models in Ollama. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGACGaTAqEC9XH3iG7O_BwnZ4OrtjmAfSbhdYlq09AHEV51pxnINeTPCLAVsSAq_4-1BfX_qTRJrTMSo1sviM9Kh6KbTjS3xc1dM1GrDoJ7lvr6C2MCNrJCXvimWT8nxTnPjG7co8=
Speech - Ollama documentation. https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFRO5UOaqGH7nzO1J4AwwKsFol-R6fJccb4NbB0OTzLd6hi251LHw0QKel99C-VyC3oBSSrYA0gwNCV6lWQbJdiILZshTB-izBV-mPSafi_443iLJrY4Z9VoSoAme4CYmm6P-G7co8=

Written by **Kamal Kishor* (@koolkamalkishor) — Sr. Frontend Developer building AI agents (React, TypeScript, LangChain). LinkedIn · GitHub · Portfolio*

Published: 04 Nov 2026. This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

Anthropic FlashProve: Verifiable AI Outputs for Developers

KAMAL KISHOR — Tue, 03 Nov 2026 11:14:58 +0000

TL;DR: Anthropic FlashProve is an open-source research prototype that uses zero-knowledge proofs (zkSNARKs) to cryptographically verify that an AI model's output came from a specific set of committed weights, without revealing those weights. This allows developers to build AI applications with auditable an

When you build an AI application, especially one that makes consequential decisions, how do you prove to your users that the output they receive actually came from the model you claim it did? It's a fundamental trust problem: a routing service could swap in a cheaper, less capable model, or an attacker could subtly alter the output stream. Simply getting a "good" answer doesn't guarantee its origin. We need a way to cryptographically attest to an AI's output.

What FlashProve Does

Anthropic's FlashProve, an open-source research prototype unveiled on November 2, 2026, directly tackles this challenge. It provides a method to cryptographically verify that a large language model's response originated from a specific, committed set of model weights without revealing the weights themselves. This means you can prove the provenance of an AI-generated output, moving beyond simple content detection to a verifiable guarantee of its origin.

As a Senior Frontend Developer building an AI-powered SDR agent platform, I know firsthand the pain points around validation and error handling in AI systems. Most AI agent projects fail in production not because of the model's intelligence, but due to a lack of robust verification that the system is operating as intended. FlashProve offers a way to establish that foundational trust.

The system is built on zkSNARKs (zero-knowledge succinct non-interactive arguments of knowledge), a type of cryptographic proof. A zkSNARK allows one party (the prover, in this case, the AI model provider) to convince another party (the verifier, your client application or an auditor) that a computation was performed correctly, without revealing the private inputs behind it. For AI, this means proving an output came from a specific model without exposing proprietary model weights or intermediate computation values. It's designed specifically for mixture-of-experts (MoE) language models, leveraging their architecture for proof generation.

How the Proof Works

FlashProve operates in two main phases:

Offline Commitment: The model provider first publishes a small cryptographic commitment to an exact set of neural network weights. This commitment, roughly 6.5 MB, is generated once and takes about 39 minutes. This initial step effectively "fingerprints" the model being used.
Online Proof Generation and Verification: For each prompt and response pair, the system generates a succinct proof that the output tokens were indeed produced by the committed model. This proof is approximately 19 MB for a 1,024-token request. Anyone with the commitment, the prompt, and the response can then check this proof on a standard CPU, typically within 17 to 21 seconds, without needing access to the actual model weights or a GPU.

The underlying principle ensures completeness (an honest proof is accepted), soundness (a false statement cannot be certified without breaking cryptographic assumptions), and zero-knowledge (the proof reveals nothing about the weights or internal computation beyond the correctness of the output). Anthropic tested FlashProve on Meta's Llama 4 Maverick, a 400.7 billion-parameter MoE model, achieving proof generation at about 110 tokens per second on a single NVIDIA H100 GPU.

Who Should Care

Developers working in domains where the integrity and auditability of AI outputs are non-negotiable should pay close attention. This includes:

Legal Tech: Verifying that AI-generated legal summaries or document analyses come from approved models.
Finance: Ensuring AI-driven trading recommendations or fraud detection outputs are from certified systems.
Secure Agents: Building autonomous AI agents where proving an action was performed by a specific, trusted agent model is critical. For instance, in an agent system responsible for lead enrichment and personalized outreach, being able to cryptographically prove that a specific message was crafted by a known, qualified agent model offers a layer of trust that simple logging cannot. This aligns with challenges in Validating AI Agent Outputs with Explicit Pass Conditions.
Regulatory Compliance: Meeting future regulations that demand verifiable AI systems.

A Practical Limitation

While FlashProve represents a significant leap, it's essential to acknowledge its current overhead. Generating these cryptographic proofs is computationally expensive. Anthropic's benchmarks show that proving requires roughly 70 to 95 times the GPU time compared to serving the model without proofs, using vLLM for inference. This high proving cost means FlashProve, in its current research prototype form, isn't practical for verifying every single token in high-throughput production environments. It's more suited for critical audits or spot checks where the cost is justified by the need for verifiable trust. The proof size, around 19 MB per 1,024-token request, is also substantial for continuous, real-time verification at scale. For now, FlashProve demonstrates the possibility of verifiable AI inference, with the next challenge being to significantly reduce this computational burden.

Sources

Written by **Kamal Kishor* (@koolkamalkishor) — Sr. Frontend Developer building AI agents (React, TypeScript, LangChain). LinkedIn · GitHub · Portfolio*

Published: 03 Nov 2026. This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

MAF Go: What Go Developers Should Know

KAMAL KISHOR — Tue, 03 Nov 2026 08:20:00 +0000

Building reliable AI agents for production is rarely about just chaining a few LLM calls together. As requirements grow, you quickly run into issues like managing conversational state across multiple turns, orchestrating complex multi-agent interactions, handling tool usage, and ensuring graceful recovery from failures. This is especially true for backend systems where Go is often the language of choice for its performance and concurrency.

Microsoft has released the Agent Framework for Go (MAF Go), an official implementation of their open, multi-language framework designed for building and orchestrating production-grade AI agents and multi-agent workflows. It offers Go developers a structured approach to move beyond simple prompt engineering to create more sophisticated, resilient agentic applications.

What MAF Go Is

MAF Go is part of the broader Microsoft Agent Framework, which also has .NET and Python implementations. It's positioned as the successor to frameworks like Semantic Kernel and AutoGen, consolidating their capabilities into a single, comprehensive platform for agent development. The Go SDK specifically provides packages, examples, and APIs tailored for building agents and workflows in Go. It's currently in public preview, with version v0.2.0 introducing provider-independent embedding generation and vector storage for RAG and memory capabilities.

The framework focuses on providing foundational building blocks for complex agent systems, including model clients for various LLMs (like Microsoft Foundry, Azure OpenAI, OpenAI, Anthropic, Gemini, and Ollama), an agent session for state management, and context providers for agent memory.

How MAF Go Works

MAF Go provides a structured way to define agents, their tools, and how they interact within workflows. The core idea is to give you explicit control over agent execution paths, moving beyond stateless chat loops.

Key operational aspects include:

Agents: These are individual units that use LLMs to process inputs, call tools, and generate responses. MAF Go supports an "Agent Harness," which is an opinionated agent with capabilities for multi-step tasks like planning, todo tracking, context compaction, file access, and observability.
Tools and Middleware: Agents can discover and use domain-specific tools or knowledge bases. Middleware allows you to intercept agent actions for custom logging, validation, or other processing, offering a crucial layer for control and monitoring.
Workflows: MAF Go excels at orchestrating multiple agents through graph-based workflows. These workflows can support various patterns:
- Sequential: Agents execute tasks in a defined order, passing results between them.
- Concurrent: Agents work in parallel to maximize efficiency.
- Group Collaboration: Agents can collaborate on a single task, often with a manager agent overseeing the interaction.
- Conditional Routing: Logic can be applied to determine the next agent or step based on certain conditions.
- Checkpointing and Human-in-the-Loop: Workflows can be paused and resumed, and human intervention points can be integrated, which is vital for long-running processes and validation in production environments.
Observability: The framework integrates with OpenTelemetry for distributed tracing, allowing you to monitor and debug agent runs, tool invocations, and workflow steps. This visibility is essential for identifying bottlenecks and errors in complex agentic systems, as covered in Agentic AI: How Autonomous Agents Are Transforming Enterprise Workflows.

Installation is straightforward, you can add it to your Go project using:

go get github.com/microsoft/agent-framework-go

Who Should Care?

If you're a Go developer tasked with building production-grade AI agents or complex multi-agent systems, MAF Go is worth your attention. It's particularly useful for scenarios requiring robust orchestration beyond simple single-prompt interactions or stateless chat loops. This includes applications where you need graph-based workflow patterns, checkpointing for durability, detailed observability, governance, or human oversight.

For teams already invested in Go for backend services, CLIs, workers, or cloud-native applications, MAF Go offers a way to integrate advanced AI capabilities natively within that ecosystem. I’m currently building an AI-powered SDR agent platform myself using other frameworks, and my experience confirms that a lot of AI agent projects tend to stumble in production due to insufficient validation and error handling, rather than issues with the model itself. A framework like MAF Go, with its focus on robust orchestration and observability, directly addresses these common failure points, providing a more stable foundation for agentic applications. This echoes the importance of structured approaches to agent task decomposition, as I discussed in LLM Agent Task Decomposition for Reliable Workflows.

A Concrete Limitation

While MAF Go offers a powerful foundation, it's important to remember that the Go SDK is still in public preview. This means that certain features available in the .NET and Python implementations, such as declarative agents, CodeAct (for generating and executing code), and some advanced functional workflows, are not yet implemented or fully available in the Go version. Additionally, while it supports embedding generation and vector stores for RAG, the more comprehensive RAG capabilities like automatic retrieval before agent runs might still be evolving. It also requires Go 1.26 or later. As with any preview software, APIs may evolve, and it might not be suitable for immediate, critical production deployments that rely on these specific missing features without careful consideration and workaround planning.

Sources

https://github.com/microsoft/agent-framework-go

This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

Daily AI notes on LinkedIn — Kamal Kishor

Muse Glimmer 30B Local: Run Agentic Workflows on Your Laptop

KAMAL KISHOR — Wed, 21 Oct 2026 08:49:48 +0000

If you've been exploring AI agents, you know the struggle: building complex workflows often means relying on cloud APIs, which quickly add up in cost and introduce latency. For local development or scenarios demanding data privacy, this becomes a blocker. This is where models like Meta's Muse Glimmer 30B step in, offering a powerful, open-source solution designed to run efficiently on your own machine.

I'm always on the lookout for free, local AI tools that deliver solid quality, especially for my work building AI-powered SDR agent platforms. It's why I even swapped out a paid coding assistant for a local alternative, as I shared in I Replaced Cursor with a Free Local AI Coding Assistant (And Saved $240/Year). Muse Glimmer fits this philosophy perfectly, promising powerful multimodal agentic capabilities without the cloud dependency.

What You Need to Get Started

To run Muse Glimmer 30B locally, you'll need a few prerequisites:

Python: Version 3.10 or higher is recommended for compatibility with modern AI libraries.
PyTorch: The underlying deep learning framework. You'll need the CUDA version if you have an NVIDIA GPU for accelerated performance, or the CPU-only version if you plan to run it exclusively on your CPU.
Unsloth Library: This is the key. Unsloth optimizes models like Glimmer to run significantly faster and with less memory, making a 30B model feasible on consumer hardware.
Hardware: While Muse Glimmer 30B can run on CPU-only systems, a GPU (even a consumer-grade one) with at least 16GB VRAM will provide a much smoother and faster experience. Expect slower inference times without a dedicated GPU.
RAM: Even with Unsloth's optimizations, a 30B model will still consume a fair amount of system RAM, especially if you're loading it in higher precision or without 4-bit quantization. Aim for 32GB or more system RAM if possible.

Setting Up Your Environment

First, ensure you have Python 3.10+ installed. Then, you'll install the Unsloth library, which includes PyTorch as a dependency. The exact command depends on whether you have a CUDA-enabled GPU:

For systems with an NVIDIA GPU (recommended for speed):

pip install unsloth[cuda-torch]

For CPU-only systems:

pip install unsloth[torch]

This command installs Unsloth along with the necessary PyTorch backend. Unsloth handles much of the complexity behind the scenes, allowing you to load large models efficiently.

Running a Quick Glimmer Example

Once Unsloth is installed, you can load and interact with the Muse Glimmer 30B model. Here's how to run a basic text generation example using the 4-bit quantized version, which is optimized for lower memory usage:

from unsloth import FastLanguageModel
import torch

# Configure model loading
max_seq_length = 2048 # Adjust as needed, Unsloth handles RoPE scaling
dtype = None # Auto detects. Use torch.float16 for older GPUs, torch.bfloat16 for newer ones (Ampere+)
load_in_4bit = True # Enables 4-bit quantization for reduced memory usage

# Load the Muse Glimmer 30B model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/muse-glimmer-30b-bnb-4bit", # Uses the 4-bit quantized version
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

# Define your prompt for the agent
prompt = "As an AI agent, outline a step-by-step plan to research and summarize the latest advancements in quantum computing for a non-technical audience."
inputs = tokenizer(
    [prompt], return_tensors = "pt"
).to("cuda" if torch.cuda.is_available() else "cpu") # Move to GPU if available, else CPU

# Generate the agent's response
outputs = model.generate(
    input_ids = inputs.input_ids,
    attention_mask = inputs.attention_mask,
    max_new_tokens = 512, # Control the length of the generated response
    use_cache = True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

This script first sets up configuration for loading the model, prioritizing 4-bit quantization to minimize memory footprint. It then loads the unsloth/muse-glimmer-30b-bnb-4bit model and its tokenizer. We've crafted a prompt that simulates an agentic task to see how Glimmer responds. The model.generate call processes this input and produces a coherent, contextually relevant output.

Things to Watch Out For

While Unsloth makes running large models like Muse Glimmer 30B more accessible, there are a few practical considerations.

First, performance will still vary significantly based on your hardware. Running a 30B model on a CPU-only system, even with Unsloth's optimizations, will be noticeably slow for any serious agentic workflow. This is the main trade-off: incredible local capability, but at the expense of speed without a decent GPU. For interactive agentic tasks, you really want GPU acceleration.

Second, dependency conflicts can always creep in. If you're running into installation issues, try isolating your environment with venv or conda to prevent conflicts with other Python projects.

Finally, remember that the model is only one part of a robust AI agent system. In my experience, most AI agent projects break in production not because of the underlying model, but due to weak validation and error handling in the surrounding agentic logic. Make sure you're building in robust checks and fallback mechanisms around the model's outputs, especially when designing for complex, multi-step workflows.

Sources

Unsloth Documentation: Muse Glimmer.

This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

Daily AI notes on LinkedIn — Kamal Kishor