DEV Community: Jangwook Kim

nanobot: Build AI Agents in 4,000 Lines You Can Actually Read

Jangwook Kim — Sat, 25 Apr 2026 08:25:43 +0000

Why This Matters

There is a recurring complaint in developer forums about modern AI agent frameworks: you spend more time understanding the framework than building your actual agent. LangGraph's dependency graph, OpenClaw's 430,000+ lines of code, CrewAI's layered abstractions — these are powerful, but they impose a learning cliff that slows down experimentation.

nanobot, released February 2, 2026 by the Data Intelligence Lab at the University of Hong Kong (HKUDS), takes the opposite bet. The entire core agent loop — message routing, LLM calls, memory management, tool execution, cron scheduling — fits in roughly 4,000 lines of Python. You can read it in an afternoon. You can fork it by lunch.

That constraint is not a limitation. It is the design. nanobot implements around 90% of OpenClaw's core capabilities with 99% less code. By April 2026 it had accumulated over 34,000 GitHub stars, making it one of the fastest-growing open-source agent frameworks of the year.

This guide walks through what nanobot actually does, how to get it running, and where it fits compared to heavier alternatives like smolagents and OpenClaw.

What Is nanobot? Core Architecture

nanobot is a personal AI agent that you deploy as a long-running process. It listens on one or more messaging channels (Telegram, Discord, WhatsApp, Slack, and others), routes incoming messages to an LLM, executes tool calls via MCP or custom skills, and persists memory between sessions.

The architecture is deliberately flat:

Incoming message
        ↓
  Channel adapter (Telegram / Discord / WhatsApp / ...)
        ↓
    Message bus
        ↓
   Agent loop
   ├── LLM call (11+ providers supported)
   ├── Tool execution (MCP stdio / HTTP)
   └── Memory read/write (session + Dream)
        ↓
  Response dispatch

Each of these layers is a small, standalone module. There is no hidden orchestration engine, no graph traversal, no pre-built DAG. The agent loop itself is the kind of code you can step through in a debugger in ten minutes.

The project requires Python 3.11 or higher and is licensed under MIT. The latest release at time of writing is v0.1.5.post2 (April 21, 2026), which added Windows and Python 3.14 support, Office document reading, SSE streaming for the OpenAI-compatible API endpoint, and improved session reliability.

Getting Started: Install in Under 5 Minutes

Three installation paths are available. PyPI is recommended for most users:

pip install nanobot-ai

Or with uv for faster installs:

uv tool install nanobot-ai

To track the latest development branch:

git clone https://github.com/HKUDS/nanobot
cd nanobot
pip install -e .

Minimal YAML Configuration

nanobot is configured entirely via a YAML (or JSON) file. A minimal setup with Telegram and Claude:

llm:
  provider: anthropic
  model: claude-sonnet-4-6
  api_key: YOUR_ANTHROPIC_KEY

channels:
  - type: telegram
    token: YOUR_TELEGRAM_BOT_TOKEN

Save this as ~/.nanobot/config.yaml and run:

nanobot start

That is the entire setup. nanobot will start listening on Telegram and responding with Claude Sonnet 4.6. No Docker, no Kubernetes, no environment config beyond the YAML file.

Supported LLM Providers

nanobot ships with adapters for 11+ providers out of the box:

Cloud APIs: Anthropic (Claude), OpenAI (GPT), Google Gemini, DeepSeek, Moonshot, Groq, AiHubMix
Aggregators: OpenRouter, DashScope, Zhipu (智谱)
Self-hosted: vLLM (for local models on your own GPU)

Switching providers is one line in the config. You can also configure multiple providers and route specific channels to specific models.

The Dream Memory System

One of the more interesting engineering choices in nanobot is its two-tier memory architecture.

Session history stores the raw conversation turns for each active chat thread in JSON files under sessions/. This is the short-term buffer — what the agent knows right now, in this conversation.

Dream is the long-term consolidation layer. It runs as a background process that periodically reads session history and extracts durable facts, summaries, and user preferences into a MEMORY.md file. Think of it as the agent sleeping on what it learned and writing notes before waking up.

The underlying storage for Dream is git-versioned, which means every memory state is recoverable. You can roll back to any earlier memory checkpoint with a standard git checkout. This is an elegant solution to a real problem: long-running agents accumulate incorrect or stale memories, and having a full audit trail makes debugging far less painful.

Dream's behavior is configured via DreamConfig in your YAML:

memory:
  dream:
    enabled: true
    interval_minutes: 60
    max_facts: 200

For agents running multi-day workflows — the kind described in guides on Temporal durable execution patterns — the Dream system fills a gap that most lightweight frameworks ignore entirely.

MCP Integration: External Tools Without the Overhead

nanobot connects to MCP (Model Context Protocol) servers via two transport modes:

stdio — for local MCP servers running as child processes
HTTP — for remote servers with optional custom authentication headers

Tools from connected MCP servers are auto-discovered and registered at startup. The agent can call any exposed tool the same way it would call a built-in skill, with no additional plumbing required.

Example YAML configuration for an MCP server:

mcp_servers:
  - name: filesystem
    transport: stdio
    command: npx
    args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]

  - name: brave-search
    transport: http
    url: https://your-mcp-server.example.com
    headers:
      Authorization: "Bearer YOUR_KEY"

This direct MCP support is meaningful in 2026, when the MCP ecosystem has crossed 97 million monthly SDK downloads. You can drop in any of the thousands of public MCP servers — GitHub, Brave Search, databases, home automation — without modifying the agent's core logic.

Multi-Platform Messaging

nanobot connects to 8+ messaging platforms through channel adapters:

Telegram — most mature, best supported
Discord — full slash command support
WhatsApp — via WhatsApp Business API
Slack — workspace bot
Feishu / DingTalk — enterprise Chinese platforms
Email — IMAP/SMTP polling
QQ — via Lagrange bridge

Each channel runs as an independent adapter. You can run multiple channels simultaneously and isolate production from testing environments by spinning up separate instances on the same machine.

This breadth matters for teams building internal AI assistants. Your sales team uses Slack, your ops team uses DingTalk, and nanobot can serve both from a single config file.

Cron Scheduling and Subagents

nanobot includes a cron system built on apscheduler for time-based automation:

# Schedule a daily briefing at 9am
nanobot cron add --name "morning" --message "Summarize today's GitHub activity" --cron "0 9 * * *"

# Check a webhook every hour
nanobot cron add --name "monitor" --message "Check deployment status" --every 3600

Cron jobs can also be defined in YAML:

cron_jobs:
  - name: daily_digest
    schedule: "0 18 * * *"
    message: "Prepare and send daily digest to team Telegram"

Subagents allow the main agent to spin up specialized child agents for scoped tasks. Subagents work in CLI mode and communicate via the internal message bus. A common pattern is a routing agent that delegates research to a subagent, code editing to another, and aggregates their results before responding to the user.

Skills: Extending nanobot With Pre-Built Behaviors

Beyond MCP tools, nanobot has a skills system. Skills are Markdown files that describe repeatable behaviors — think of them as stored prompts with light tool wiring. The agent loads skills from a skills/ directory and can invoke them by name.

nanobot ships with pre-bundled skills for GitHub, weather, system commands, and general task management. Community skills can be discovered and installed through the ClawHub skill registry:

nanobot skill search web-scraper
nanobot skill install web-scraper

Unlike the massive Hermes Agent skill ecosystem (118 bundled skills, self-improving closed learning loop), nanobot keeps the default skill set minimal. That is intentional — you install exactly what you need, and the codebase stays readable.

nanobot vs The Alternatives

Feature	nanobot	smolagents	LangGraph	OpenClaw
Core codebase	~4,000 lines	~1,000 lines	50,000+ lines	430,000+ lines
License	MIT	Apache 2.0	MIT	MIT
Messaging platforms	8+ built-in	None built-in	None built-in	20+
MCP support	Yes (stdio + HTTP)	Yes (limited)	Partial	Yes (full)
Long-term memory	Dream (git-versioned)	External only	Checkpoints	Full memory graph
Cron scheduling	Built-in (apscheduler)	No	External	Built-in
Self-host difficulty	Low	Very low	Medium	Medium
Best for	Personal agents, hackable infra	Code-first prototyping	Complex multi-agent graphs	Production multi-platform

The choice comes down to what you're optimizing for. smolagents is the right tool when you need a minimal code-execution agent fast. LangGraph wins on complex stateful multi-agent graphs with human-in-the-loop requirements. OpenClaw has the widest platform coverage and a large community skills ecosystem.

nanobot's sweet spot is the developer who wants a persistent personal agent — one that lives on a cheap VPS, watches multiple channels, handles cron jobs, and grows with you — without reading 430,000 lines of someone else's code to understand what's happening.

Common Mistakes

Running without a persistent process manager. nanobot is a long-running daemon. Run it with systemd, supervisord, or at minimum nohup. Crashes in Telegram or Discord adapters will disconnect your channels and you won't notice until someone complains.

Over-engineering the skill system early. Skills are for repeatable, well-defined tasks. Using them as a workaround for prompt quality issues just adds indirection. Fix the prompts first.

Ignoring session history growth. The sessions/ directory grows indefinitely if Dream consolidation is disabled. A 3-month-old agent on an active Telegram group can accumulate gigabytes of JSON. Configure Dream with a reasonable interval_minutes and cap max_facts.

Misconfiguring MCP server lifecycles. Stdio MCP servers are child processes of nanobot. If nanobot crashes, those servers crash with it. HTTP MCP servers survive independently. Structure your architecture accordingly when uptime matters.

Running multiple instances against the same config directory. Session and memory files are not designed for concurrent writes. Use separate config directories for each instance.

FAQ

Q: Does nanobot work with local LLMs?

Yes. Configure vLLM as the provider with a local endpoint URL and model name. Any OpenAI-compatible API server works:

llm:
  provider: openai_compatible
  base_url: http://localhost:8000/v1
  model: llama-4-scout
  api_key: none

Q: How does nanobot compare to OpenClaw in terms of stability?

OpenClaw is more battle-tested in high-volume production deployments with community skill ecosystems. nanobot is newer (February 2026) but has been moving fast — v0.1.5.post2 added Windows support and SSE streaming in April 2026. For a personal agent or small team, nanobot's stability is adequate. For enterprise scale, OpenClaw or LangGraph are safer bets today.

Q: Can I run nanobot without connecting it to a messaging platform?

Yes. nanobot ships with a CLI interface — run nanobot chat to interact directly in the terminal without configuring any channel adapter. This is useful for testing skills and memory behavior before deploying to Telegram or Discord.

Q: How does the Dream memory consolidation handle incorrect facts?

Dream stores memory as Markdown files under git version control. To remove or correct a fact, edit MEMORY.md directly and commit the change. The agent picks up the updated file on the next session. There is currently no automated conflict resolution — incorrect memories require manual intervention.

Q: What is the minimum VPS spec to run nanobot?

nanobot's own footprint is small — approximately 100MB RAM for the process itself. Add the memory of whichever cloud LLM provider you use (network only, no local inference), plus any MCP server processes. A $4/month VPS with 512MB RAM runs nanobot comfortably. For local LLM inference via vLLM, the hardware requirements of the model dominate.

Key Takeaways

nanobot is a ~4,000-line MIT-licensed Python agent framework from HKUDS, released February 2, 2026
Install with pip install nanobot-ai, configure with a single YAML file, and start receiving messages in under five minutes
Supports 8+ messaging platforms, 11+ LLM providers, MCP (stdio and HTTP), cron scheduling, subagents, and a two-tier Dream memory system
The entire core codebase is readable in an afternoon — this is a feature, not a constraint
Best suited for personal agents and developer experiments where understanding and forking the code matters more than enterprise-scale throughput
For production multi-agent orchestration with complex state, LangGraph or OpenClaw remain stronger choices

Bottom Line

nanobot delivers a genuinely useful personal AI agent in a codebase you can actually read, fork, and understand. If the opacity of larger frameworks has been your reason for not deploying an agent yet, nanobot removes that excuse. Ship it to a VPS, wire up Telegram, and extend it from there.

Prefer a deep-dive walkthrough? Watch the full video on YouTube.

MCP vs A2A vs Open Responses — AI Agent Communication Protocols in 2026: What to Actually Use

Jangwook Kim — Sat, 25 Apr 2026 06:40:00 +0000

Since late 2025, AI agent standards have been arriving in a cluster. Anthropic donated MCP to the Linux Foundation, Google announced A2A, and OpenAI published the Open Responses spec. That's great news for the ecosystem — but it's also confusing as hell. What does each one do? Are they competing? Can they coexist?

My first reaction was "another protocol war." Then I built a few MCP servers myself, read through the A2A spec, and my view changed. These three protocols aren't competing — they occupy different layers. The confusion comes from the fact that all three sound like "agent communication standards" when you read the names.

In this post I'll break down each protocol and give my honest take on when to use what.

MCP: Giving Agents Hands

MCP (Model Context Protocol) was published by Anthropic in late 2024 and donated to the Linux Foundation's Agentic AI Initiative (AAIF) in December 2025. Its core purpose is singular: standardize how AI models access external tools and data.

The "USB-C for AI" analogy holds up. Before USB-C, every laptop had a different charging port. Before MCP, Claude's tool connections and GPT's tool connections were separate implementations. MCP created a common connector.

What MCP standardizes:

Tools: Functions or actions an agent can invoke (file reads, API calls, code execution)
Resources: Data the agent can read (documents, DB records, file system)
Prompts: Reusable prompt templates the server provides

As of April 2026, there are over 5,000+ MCP servers — GitHub Actions, Notion, PostgreSQL, Brave Search, browser automation, and almost every major tool you can think of.

When I first wired MCP into this blog's automation system, the thing that surprised me most was framework agnosticism. MCP servers I built for Claude Code worked in other MCP-compatible clients without modification. In practice there are edge cases where client feature sets differ, but the direction is sound.

What the 2026 MCP Roadmap Focuses On

The most important item on the 2026 MCP roadmap is solving horizontal scaling. Current Streamable HTTP transport maintains stateful sessions — which fights with load balancers. When requests get routed to different server instances, sessions break. The roadmap aims to make MCP servers genuinely stateless.

The second priority is discovery standardization via .well-known. Right now you have to connect to an MCP server to know what it offers. The goal is to serve capability metadata without a live connection.

My earlier post on WebMCP gets into how MCP server implementation works under the hood, if you want a concrete picture.

A2A: Agents Talking to Each Other

A2A (Agent2Agent) was announced by Google in April 2025 and donated to the Linux Foundation in June 2025. The purpose is different from MCP: standardize how AI agents discover, communicate with, and delegate to each other.

If MCP is "agent ↔ tool," A2A is "agent ↔ agent."

The problem A2A solves: suppose you have a travel booking agent, a hotel search specialist agent, and a flight search specialist agent. How does the booking agent delegate tasks to the specialists? MCP doesn't handle this. That's A2A's domain.

A2A v1.0 Core Concepts

A2A v1.0, released in early 2026:

Agent Card: A JSON document where an agent advertises its capabilities. When a client agent needs to find the right specialist, it reads Agent Cards.

Task-based communication: Interactions are oriented around Tasks. Tasks can complete immediately or run long, with state synchronization built in.

Signed Agent Cards (the v1.0 headline feature): Cryptographic signatures allow receiving agents to verify that an Agent Card was issued by the domain owner. This makes decentralized agent discovery viable — you can filter fake agents.

By April 2026, 150+ organizations have adopted A2A, with production deployments at Microsoft, AWS, Salesforce, SAP, and ServiceNow.

Honest take: when I first read the A2A spec, I was skeptical about practical safety. Agents delegating directly to other agents sounds elegant, but the trust model gets complicated fast. v1.0's Signed Agent Cards are moving in the right direction, but I'd want to see more production validation before treating it as battle-hardened infrastructure.

A separate post covers A2A + MCP production hybrid architectures — specifically how to layer these two protocols without creating a mess.

Open Responses: OpenAI's Bet on API Compatibility

Open Responses is an open spec published by OpenAI in February 2026. It operates at a different level from MCP and A2A. Those two address how agents communicate; Open Responses addresses how to standardize agentic workflow APIs.

The spec is built on OpenAI's Responses API — the successor to Chat Completions — and the pitch is: let's open this standard so that other model providers can offer the same interface. If you write agentic code against the Responses API, it should run against Hugging Face models, local inference, or any other compliant provider without rewriting your integration.

Ecosystem support: Hugging Face, Vercel, OpenRouter have signed on. Ollama, vLLM, and LM Studio support it for local inference. The spec documentation and conformance testing tools are at openresponses.org.

My honest take: Open Responses is complementary to MCP and A2A, not competing. But I don't see a compelling reason to prioritize it in most production stacks right now. Large-scale production validation is thin. The bet that other vendors will adopt OpenAI's API design as a universal standard is real, but unproven at scale.

Side-by-Side Comparison

	MCP	A2A	Open Responses
Purpose	Agent ↔ Tool connectivity	Agent ↔ Agent collaboration	Agentic API loop standardization
Analogy	USB-C (universal connector)	HTTP (for agent networks)	REST API design standard
Origin	Anthropic → AAIF	Google → Linux Foundation	OpenAI
Current version	2025-11-25	v1.0 (early 2026)	Beta
Ecosystem maturity	High (5,000++ servers)	High (150+ orgs)	Low (early stage)
Transport	Streamable HTTP, stdio	JSON-RPC, gRPC	WebSocket, HTTP
Security model	OAuth, per-server auth	Signed Agent Cards	Under specification
When to use	Any time tool access is needed	Multi-agent task delegation	OpenAI-compatible workflows

The most important thing to understand: MCP and A2A are AND, not OR. Most production multi-agent systems in 2026 use both. Each agent connects to its own tools via MCP; agents coordinate via A2A.

How They Layer in Practice

A concrete architecture example:

Scenario: Automated research system

Orchestrator Agent
├── (A2A) → Web research specialist agent
│   └── (MCP) → Brave Search MCP server
│   └── (MCP) → Web scraping MCP server
├── (A2A) → Document analysis specialist agent
│   └── (MCP) → File system MCP server
│   └── (MCP) → PDF processing MCP server
└── (MCP) → Results storage MCP server

The orchestrator delegates via A2A; each specialist accesses its own tools via MCP. Open Responses could sit at the orchestrator's external API interface if you need OpenAI-compatible endpoint exposure.

My breakdown of Claude Code agentic workflow patterns goes deeper on implementing this kind of layered architecture.

What to Learn Right Now

Practical priority order:

Learn immediately: MCP

If you're building agents, start here. Reasons:

5,000++ server ecosystem already exists
Claude Code, OpenAI Agents SDK, LangGraph all support it natively
Streamable HTTP is the settled standard; spec is stable enough for production
Anthropic's Agent Skills standard builds directly on MCP, creating increasingly powerful patterns

Medium-term: A2A

If you're planning multi-agent production systems, study A2A. 150 org adoption, Linux Foundation governance, v1.0 stability — it's ready. But I'd still want to see more validated production case studies before relying on the Signed Agent Cards trust model for anything security-critical.

Monitor: Open Responses

Unless you have a specific OpenAI-compatibility requirement today, there's no urgency. Subscribe to updates; don't architect around it yet.

One more thing worth noting: both MCP and A2A are now under the Linux Foundation. This isn't a standards war — it's the same foundation solving two different layers of the same problem. That's the clearest signal that 2026 is different from 2024.

My Take

MCP is the tool to use right now. It's the layer that gives agents access to the external world, and the ecosystem is mature. A2A is worth learning seriously if you're thinking about multi-agent systems — v1.0 is production-ready in most respects. Open Responses is worth following but not yet worth building around.

Stop thinking about these as competing standards. They solve different problems and most serious systems need all three eventually. My working heuristic: MCP first, A2A when you need multi-agent delegation, Open Responses when the ecosystem catches up.

And your choice of AI agent framework is tightly coupled to this — different frameworks have significantly different levels of MCP and A2A support.

Meta Llama Stack: Deploy Llama 4 With OpenAI-Compatible API

Jangwook Kim — Sat, 25 Apr 2026 04:21:19 +0000

Why Llama Stack Changes the Open-Source Deployment Story

Running open-source LLMs in production has always had a catch: you pick a backend (Ollama, vLLM, llama.cpp), write your integration code against that backend's specific API, and then find yourself locked in. Swap the backend for performance or cost reasons and you're rewriting client code.

Meta Llama Stack solves exactly this problem. It's an open-source AI application server that sits in front of any backend and exposes a single, OpenAI-compatible API layer. The same /v1/chat/completions call that works against your local Ollama instance in development routes to vLLM or AWS Bedrock in production — with zero application-code changes.

As of April 2026, the repository has over 8,200 GitHub stars and is under active development by Meta's open-source team. It ships with native support for Llama 4 Scout and Llama 4 Maverick, alongside older Llama 3.x models. If you're running or planning to run open-weight Llama models in production, Llama Stack is the infrastructure layer worth knowing.

What Llama Stack Actually Is

Most frameworks for LLM deployment focus on one thing — inference serving. Llama Stack goes wider. It provides a unified API layer covering seven concerns:

Inference — run Llama models against any supported backend
RAG — vector store integration for retrieval-augmented generation
Agents — multi-step agent orchestration with tool use
Tools — web search, code interpreter, custom tool registration
Safety — Llama Guard integration for prompt/output filtering
Evals — built-in evaluation harness
Telemetry — OpenTelemetry-based tracing and logging

The architecture has two layers: a distribution (a pre-configured bundle of provider implementations) and the Llama Stack server (a single process that routes API calls to whichever providers are configured in that distribution).

Your application only ever talks to the Llama Stack server. Swapping backends, safety models, or vector stores is a config change, not a code change.

Distributions: The Core Concept

A distribution is the unit of deployment in Llama Stack. It bundles together one provider for each API component and packages them into a runnable server.

Meta ships several official distributions out of the box:

Distribution	Inference Backend	Best For
`ollama`	Ollama	Local development, CPU/Apple Silicon
`vllm`	vLLM	GPU production servers
`tgi`	HuggingFace TGI	HuggingFace-native stacks
`together`	Together AI	Managed API, no GPU needed
`fireworks`	Fireworks AI	Low-latency managed inference
`bedrock`	AWS Bedrock	AWS-native production
`openai`	OpenAI API	Hybrid open/closed LLM routing

The pattern: develop with ollama, deploy with vllm or a managed service. The API your application uses doesn't change.

You can also build custom distributions if you need to mix providers — for example, using Fireworks for inference but self-hosted ChromaDB for vector storage.

Installation and Quick Start

Install the Client

pip install llama-stack-client

Run the Server with Ollama (Local Development)

First, make sure Ollama is running and has pulled the model:

ollama pull llama3.3

Then start the Llama Stack server pointing at the Ollama distribution:

export INFERENCE_MODEL="llama3.3"
export LLAMA_STACK_PORT=8321

docker run -it \
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
  -v ~/.llama:/root/.llama \
  -e INFERENCE_MODEL=$INFERENCE_MODEL \
  llamastack/distribution-ollama:latest

The server is now running at http://localhost:8321 and exposes standard OpenAI-compatible endpoints.

Your First API Call

Use the OpenAI Python client directly — point it at the Llama Stack server:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8321/v1",
    api_key="not-required"  # Llama Stack handles auth separately
)

response = client.chat.completions.create(
    model="llama3.3",
    messages=[
        {"role": "user", "content": "Explain attention mechanisms in one paragraph."}
    ]
)

print(response.choices[0].message.content)

That's it. Any existing code that uses the OpenAI Python SDK can point at a Llama Stack server instead, with no further changes.

Or Use the Native Llama Stack Client

from llama_stack_client import LlamaStackClient

client = LlamaStackClient(base_url="http://localhost:8321")

response = client.inference.chat_completion(
    model_id="llama3.3",
    messages=[{"role": "user", "content": "What are the Llama 4 model sizes?"}]
)

print(response.completion_message.content)

The native client exposes additional Llama Stack-specific APIs (agents, memory, safety) that aren't part of the OpenAI SDK interface.

Production Deployment with vLLM

For GPU production deployments, swap the distribution from ollama to vllm.

Start the vLLM Backend

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tensor-parallel-size 2

Start Llama Stack Pointed at vLLM

export INFERENCE_MODEL="meta-llama/Llama-4-Scout-17B-16E-Instruct"
export VLLM_URL="http://localhost:8000"
export LLAMA_STACK_PORT=8321

docker run -it \
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
  -e INFERENCE_MODEL=$INFERENCE_MODEL \
  -e VLLM_URL=$VLLM_URL \
  llamastack/distribution-vllm:latest

Now your application code is unchanged — it still talks to http://your-server:8321/v1. The only thing that moved was the backend.

Llama 4 Model Options

Llama Stack supports the full Llama 4 family. The two currently available production models:

Model	Parameters	Active Params	Context	Best For
Llama 4 Scout	109B total / 17B active	17B	10M tokens	Single GPU, balanced tasks
Llama 4 Maverick	400B total / 52B active	52B	10M tokens	Multi-GPU, high-quality output

Both are MoE (Mixture of Experts) models under Meta's open-weight license. Scout runs on a single A100 80GB; Maverick requires 2–4 GPUs depending on quantization.

Agents and Tool Use

Llama Stack's agents API goes well beyond basic chat completion. An agent maintains a session, executes multi-step plans, and calls tools.

from llama_stack_client import LlamaStackClient

client = LlamaStackClient(base_url="http://localhost:8321")

# Create an agent with web search enabled
agent_config = {
    "model": "llama3.3",
    "instructions": "You are a research assistant. Use search when you need current information.",
    "tools": [{"type": "brave_search", "api_key": "YOUR_BRAVE_API_KEY"}],
    "max_infer_iters": 5
}

agent = client.agents.create(**agent_config)
session = client.agents.sessions.create(
    agent_id=agent.agent_id,
    session_name="research-session"
)

# Turn 1
response = client.agents.turns.create(
    agent_id=agent.agent_id,
    session_id=session.session_id,
    messages=[{"role": "user", "content": "What are the latest Llama 4 benchmarks?"}],
    stream=True
)

for chunk in response:
    if hasattr(chunk, "event") and chunk.event.payload.event_type == "turn_complete":
        print(chunk.event.payload.turn.output_message.content)

The agent automatically decides when to call the search tool, reads the results, and synthesizes a final answer — all within Llama Stack's orchestration layer.

Built-in tools include:

brave_search — web search via Brave Search API
wolfram_alpha — math and science queries
code_interpreter — sandboxed Python execution
Custom tools via function registration

Safety: Llama Guard Integration

Every Llama Stack distribution can run Llama Guard as the safety provider, filtering both inputs and outputs against configurable policy categories.

# Check a response against safety policy
safety_response = client.safety.run_shield(
    shield_id="meta-llama/Llama-Guard-3-8B",
    messages=[{"role": "assistant", "content": response_text}]
)

if safety_response.violation:
    print(f"Safety violation: {safety_response.violation.user_message}")
else:
    print(response_text)

Safety categories can be tuned per deployment. Production use cases often enable all defaults; internal developer tools might relax some restrictions.

Additional safety capabilities in Llama Stack:

Prompt injection detection
Output validation before streaming
Rate limiting (configurable per session or API key)
Human-in-the-loop controls for high-stakes actions
Audit logging for compliance

Memory and RAG

The Memory API supports four storage types for different use cases:

Type	Best For
Vector (FAISS, ChromaDB, Weaviate)	Semantic similarity search, RAG
Key-Value (Redis, PostgreSQL)	Session state, structured lookup
Keyword (BM25)	Exact-match and hybrid search
Graph (Neo4j)	Relationship-based retrieval

Adding RAG to an agent is a configuration change:

# Create a memory bank (vector store)
memory_bank = client.memory_banks.register(
    memory_bank_id="my-docs",
    params={
        "memory_bank_type": "vector",
        "embedding_model": "all-MiniLM-L6-v2",
        "chunk_size_in_tokens": 512,
        "overlap_size_in_tokens": 64
    }
)

# Insert documents
client.memory.insert(
    bank_id="my-docs",
    documents=[
        {"document_id": "doc-1", "content": "Llama 4 Scout has a 10M token context window..."}
    ]
)

In production, PostgreSQL is the recommended backend for both vector storage and key-value persistence, replacing in-memory FAISS for durability across restarts.

Telemetry and Observability

Llama Stack ships a complete OpenTelemetry-native telemetry system. Traces, spans, and events flow from the server to any OTEL-compatible backend (Jaeger, Grafana Tempo, Datadog, etc.).

# Enable OTEL tracing in your distribution config
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
OTEL_SERVICE_NAME=llama-stack-prod

Every inference call, agent step, tool invocation, and safety check becomes a traced span. This gives you:

Per-request token counts and latency
Agent tool-call traces with reasoning steps
Safety shield evaluation times
Provider-level error attribution

For teams already using Langfuse or other LLM observability tools, Llama Stack's OTEL output integrates cleanly with existing dashboards.

Common Mistakes

Using the wrong distribution for your hardware. The ollama distribution works fine on CPU and Apple Silicon, but for A100/H100 servers, vllm gives 3–5x better throughput. Don't use CPU-tier distributions for GPU production workloads.

Not setting model IDs consistently. The model identifier in your API call must match exactly what the provider backend has loaded. With vLLM this is usually the full HuggingFace path (meta-llama/Llama-4-Scout-17B-16E-Instruct); with Ollama it's the short tag (llama3.3). Mismatches return a 404 that looks like a server error.

Skipping Safety in development. Llama Guard evaluation adds ~50ms latency. Developers sometimes disable it locally to speed up iteration, then forget to re-enable it before production. Treat safety configuration as part of your deployment checklist, not a late addition.

Ignoring session management for agents. Agent sessions accumulate context across turns. For production services that handle many concurrent users, set session_ttl and clean up sessions explicitly, or you'll see memory growth over time.

Mounting volumes incorrectly for model weights. The Docker images expect model weights at specific paths. If the volume mount doesn't match, the container downloads models on startup — slow, expensive, and fragile in autoscaling environments. Pre-pull weights and mount them at the documented path.

FAQ

Q: Can I use Llama Stack with non-Llama models?

Yes. Llama Stack supports any model available through its provider backends. The openai distribution lets you route to GPT-4o or GPT-6, the anthropic provider connects to Claude, and the vllm distribution serves any HuggingFace-compatible model. The "Llama" branding is about the defaults, not a hard constraint.

Q: How does Llama Stack compare to LiteLLM?

LiteLLM focuses on unified API routing to managed providers (OpenAI, Anthropic, Azure, etc.) with cost tracking and fallback logic. Llama Stack is broader: it includes self-hosting, agent orchestration, safety, RAG, and evaluation. For a team running managed cloud providers, LiteLLM is simpler. For teams self-hosting Llama models who need agents and safety, Llama Stack adds significant value beyond what LiteLLM offers.

Q: Is Llama Stack production-ready?

The core inference and OpenAI-compatible endpoints are stable and used in production deployments. The agent, memory, and evaluation APIs are under more active development. As of version 0.2.x (April 2026), production use is most reliable for inference + safety use cases. Agents work well but have more API surface area that can change between minor versions.

Q: What's the minimum hardware for running Llama 4 Scout with Llama Stack?

Llama 4 Scout (17B active parameters) requires approximately 35GB VRAM in BF16 or 20GB with 4-bit quantization. A single A100 40GB handles it comfortably. For Apple Silicon, M4 Pro (48GB unified memory) or M4 Max runs it at reduced throughput. Maverick needs 2–4 A100/H100 GPUs depending on quantization level.

Q: Can I run Llama Stack without Docker?

Yes. Install via pip: pip install llama-stack and run llama-stack start --config path/to/config.yaml. Docker is recommended for production for isolation and reproducibility, but the Python package works for development and custom deployments.

Key Takeaways

Meta Llama Stack is the cleanest path from local Llama model experimentation to production deployment. Its distribution model — develop with Ollama, deploy with vLLM, never change your API client — removes the most common painful rewrite in open-source LLM adoption.

The OpenAI compatibility layer is the practical unlock: teams already using the OpenAI Python SDK can switch to self-hosted Llama 4 by changing one line (base_url). Combined with built-in agents, safety, RAG, and telemetry, Llama Stack positions itself as a full infrastructure layer, not just a model server.

For the current state of production deployments: inference and safety are solid; agents and RAG are functional with active API evolution. A reasonable approach is to start with inference + safety in production, then evaluate the agents API for lower-stakes workloads while it stabilizes.

Bottom Line

Llama Stack is the best available open-source infrastructure layer for Llama 4 production deployments. The OpenAI-compatible API and swappable distribution model eliminate the usual vendor lock-in trade-off of self-hosting — you get full control without rewriting your application code when you scale up or change backends.

Prefer a deep-dive walkthrough? Watch the full video on YouTube.

DeepSeek V4-Pro and V4-Flash: Migration Guide and API Setup

Jangwook Kim — Sat, 25 Apr 2026 00:19:37 +0000

DeepSeek dropped two new models on April 24, 2026: V4-Pro, a 1.6-trillion-parameter MoE flagship, and V4-Flash, a 284-billion-parameter workhorse optimized for throughput. Both support a one-million-token context window, dual Thinking/Non-Thinking modes, and an OpenAI-compatible API — available immediately on DeepSeek's platform and across third-party providers.

There's a deadline attached. The legacy deepseek-chat and deepseek-reasoner model names are being retired on July 24, 2026, 15:59 UTC. If your application targets either of those strings, you have roughly three months to update a single line of code.

This guide covers what changed architecturally, how V4-Pro and V4-Flash differ, how the benchmarks and pricing compare to frontier alternatives, and exactly how to migrate — with copy-paste code examples.

Why This Matters

DeepSeek's V4 release lands at a moment when the frontier pricing war has broken wide open. GPT-5.5 output tokens cost $30/M; Claude Opus 4.7 costs $25/M. V4-Pro output is $3.48/M — roughly one-seventh the price of GPT-5.5 — while scoring within single-digit percentage points of both models on most coding and reasoning benchmarks.

For cost-sensitive production deployments — high-volume agents, RAG pipelines, code review bots, document analysis systems — the V4 release changes the default calculus. The question is no longer "is open-source good enough?" but "which workloads still justify the closed-source premium?"

The migration urgency adds a second dimension: any team still using deepseek-chat or deepseek-reasoner in production needs to act before July 24.

Model Overview: V4-Pro vs V4-Flash

Spec	DeepSeek V4-Pro	DeepSeek V4-Flash
Total Parameters	1.6T (49B active)	284B (13B active)
Architecture	MoE + Hybrid Attention	MoE + Hybrid Attention
Context Window	1,000,000 tokens	1,000,000 tokens
Reasoning Modes	Thinking + Non-Thinking	Thinking + Non-Thinking
Input Pricing (cache miss)	$1.74/M tokens	$0.14/M tokens
Input Pricing (cache hit)	$0.145/M tokens	$0.028/M tokens
Output Pricing	$3.48/M tokens	$0.28/M tokens
License	Apache 2.0	MIT
Weights on Hugging Face	Yes	Yes
Best For	Complex reasoning, agents, coding	High-throughput, cost-sensitive tasks

Both models also receive a 50% discount during Beijing off-peak hours — relevant for batch jobs that don't need real-time response.

V4-Pro is designed for tasks where quality ceiling matters: complex multi-step agents, competitive programming, research-grade reasoning, long-document analysis across the full 1M context.

V4-Flash replaces both deepseek-chat (non-thinking mode) and deepseek-reasoner (thinking mode) in the transition mapping. At $0.28/M output tokens, it's the right default for classification, summarization, extraction, customer-facing chat, and high-volume pipelines where V4-Pro's quality headroom goes unused.

What's New: The Hybrid Attention Architecture

The architectural upgrade that makes V4's 1M context practical is the Hybrid Attention Architecture (HAA) — a combination of two complementary attention strategies applied layer by layer.

Compressed Sparse Attention (CSA)

CSA first compresses KV caches along the sequence dimension (compression rate 4 in V4), then applies DeepSeek Sparse Attention. A "lightning indexer" selects the top-k most relevant compressed KV entries per query: V4-Pro selects the top 1,024; V4-Flash selects the top 512.

This gives the model a high-precision view of the most relevant context chunks — similar to how a search index retrieves only the best-matching documents rather than scanning everything.

Heavily Compressed Attention (HCA)

HCA applies a much more aggressive compression rate of 128, then performs dense attention over that smaller representation. Every layer gets a cheap, global view of distant tokens — the model always knows roughly what happened 800K tokens ago, even if it can't recall exact details.

The Combined Effect

By routing between CSA and HCA at every depth, V4 avoids the standard memory explosion of full attention at 1M tokens. The result:

27% of single-token inference FLOPs compared to DeepSeek-V3.2 at equivalent context
10% of KV cache memory compared to V3.2
Usable 1M context on standard inference hardware rather than requiring specialized memory configurations

DeepSeek also trained V4 on 32T+ tokens using FP4 + FP8 mixed precision (MoE experts at FP4, most parameters at FP8), which contributes to the efficiency advantage over V3.2.

Manifold-Constrained Hyper-Connections (mHC)

A secondary architectural addition: mHC strengthens conventional residual connections to improve signal propagation stability across the model's many layers. The practical effect is more stable training and better performance on tasks requiring deep multi-step reasoning.

Benchmarks: How V4-Pro Stacks Up

Benchmark	V4-Pro	Claude Opus 4.7	GPT-5.5	Gemini 3.1 Pro
SWE-bench Verified	80.6%	~80.4%	—	—
LiveCodeBench	93.5	88.8	—	91.7
Codeforces Rating	3206	—	3168 (GPT-5.4)	—
BrowseComp (agentic search)	83.4%	—	—	—
MCPAtlas (tool orchestration)	73.6%	—	—	—
HMMT 2026 math	95.2%	96.2%	97.7% (GPT-5.4)	—
SimpleQA factual recall	57.9%	—	—	75.6%
Output cost / M tokens	$3.48	$25.00	$30.00	—

Where V4-Pro leads: coding. LiveCodeBench 93.5 puts it ahead of both Gemini 3.1 Pro (91.7) and Claude (88.8). On real-world competitive programming via Codeforces rating (3206), it beats GPT-5.4 (3168). SWE-bench Verified (80.6%) lands within 0.2 points of Claude Opus 4.6.

Where it trails: factual knowledge retrieval. SimpleQA-Verified at 57.9% versus Gemini's 75.6% is a meaningful gap for applications that need reliable factual recall (knowledge base Q&A, citation-heavy document work). Advanced math competition problems (HMMT 2026) show Claude (96.2%) and GPT-5.4 (97.7%) pulling ahead.

The agentic benchmarks are the headline: BrowseComp 83.4% and MCPAtlas 73.6% suggest V4-Pro is genuinely competitive at autonomous multi-step tasks — not just raw text generation.

Migration Guide: Updating from deepseek-chat and deepseek-reasoner

The migration is a one-line change in most codebases. DeepSeek kept the base URL and request/response shapes identical. Only the model parameter changes.

Legacy model mapping

Old model name	New equivalent	Mode
`deepseek-chat`	`deepseek-v4-flash`	Non-Thinking
`deepseek-reasoner`	`deepseek-v4-flash`	Thinking

During the transition window (until July 24), the legacy names are already silently routing to V4-Flash. After the deadline, they will return errors.

Python migration (OpenAI SDK)

Before:

from openai import OpenAI

client = OpenAI(
    api_key="your-deepseek-key",
    base_url="https://api.deepseek.com"
)

response = client.chat.completions.create(
    model="deepseek-chat",  # ← retiring July 24
    messages=[{"role": "user", "content": "Review this code: ..."}]
)

After (drop-in replacement):

from openai import OpenAI

client = OpenAI(
    api_key="your-deepseek-key",
    base_url="https://api.deepseek.com"
)

response = client.chat.completions.create(
    model="deepseek-v4-flash",  # ← updated
    messages=[{"role": "user", "content": "Review this code: ..."}]
)

To upgrade to V4-Pro for higher-quality outputs:

response = client.chat.completions.create(
    model="deepseek-v4-pro",  # ← flagship model
    messages=[{"role": "user", "content": "Review this code: ..."}]
)

Enabling Thinking mode

Both V4-Pro and V4-Flash support explicit Thinking mode. Pass thinking in the model field suffix or use the reasoning_effort parameter — DeepSeek supports three levels:

# Thinking mode: for complex reasoning chains
response = client.chat.completions.create(
    model="deepseek-v4-pro",
    messages=[{"role": "user", "content": "Solve this step by step: ..."}],
    extra_body={"thinking": True}  # enable extended reasoning
)

# Non-Thinking mode (default): for faster, cheaper completions
response = client.chat.completions.create(
    model="deepseek-v4-pro",
    messages=[{"role": "user", "content": "Summarize this document: ..."}]
    # thinking defaults to False
)

If you were previously using deepseek-reasoner specifically for its chain-of-thought behavior, migrate to deepseek-v4-flash with "thinking": True — that maps directly to the same reasoning capability at the same price tier.

TypeScript / Node.js migration

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.DEEPSEEK_API_KEY,
  baseURL: "https://api.deepseek.com",
});

// V4-Flash: fast, cheap, suitable for most tasks
const response = await client.chat.completions.create({
  model: "deepseek-v4-flash",
  messages: [{ role: "user", content: "..." }],
});

// V4-Pro: best quality for coding/agent tasks
const proResponse = await client.chat.completions.create({
  model: "deepseek-v4-pro",
  messages: [{ role: "user", content: "..." }],
});

Environment variable approach for easy switching

If you centralize your model name:

import os

# Set in .env or deployment config:
# DEEPSEEK_MODEL=deepseek-v4-pro
# DEEPSEEK_MODEL=deepseek-v4-flash

model = os.getenv("DEEPSEEK_MODEL", "deepseek-v4-flash")

response = client.chat.completions.create(
    model=model,
    messages=[...]
)

This makes it easy to A/B test Pro vs Flash across deployments without touching application code.

Third-Party Providers

DeepSeek V4 is available on multiple inference providers — useful if you need lower latency, specific geographic regions, or pay-per-use billing alternatives:

Provider	Models Available	Notes
DeepSeek API	V4-Pro, V4-Flash	Official, cheapest at peak pricing
Together AI	V4-Pro	`deepseek-ai/DeepSeek-V4-Pro`
OpenRouter	V4-Pro, V4-Flash	Unified key across providers
DeepInfra	V4-Pro	Low-latency EU and US endpoints
APIYI	V4-Flash	5-minute migration guide available
Hugging Face	Both (weights)	Self-host via vLLM or TGI

For most teams, the DeepSeek API is the default. But if you're already using Together AI or OpenRouter for provider routing, both models are available there without a separate key.

Self-Hosting Options

Both models have weights published on Hugging Face:

deepseek-ai/DeepSeek-V4-Pro (Apache 2.0)
deepseek-ai/DeepSeek-V4-Flash (MIT)

V4-Flash is the realistic self-host option for most teams. At 284B total parameters with 13B active per forward pass, it runs on multi-GPU setups in FP4/FP8 with reasonably sized hardware. V4-Pro at 1.6T is a data-center-scale deployment — full-scale self-hosting requires significant infrastructure, though quantized versions via Unsloth (unsloth/DeepSeek-V4-Pro) reduce that burden.

vLLM added native support for V4's Hybrid Attention Architecture shortly after release, making it the preferred inference framework for self-hosted deployments. DeepSeek also noted close integration with Huawei's Ascend chips for organizations running on Chinese cloud infrastructure.

Cost Comparison: When to Use Which Tier

At $0.28/M output tokens, V4-Flash is the right default for:

High-volume classification and extraction pipelines
Customer support chatbots
Real-time summarization
Any task where deepseek-chat was already sufficient

At $3.48/M output tokens, V4-Pro makes sense when:

You're building coding agents that need reliable SWE-bench-level performance
Tasks require multi-step agentic reasoning (MCPAtlas-class tool orchestration)
Documents approach 100K+ tokens and you need deep contextual understanding
Long-context retrieval across 500K–1M token windows

Even V4-Pro at $3.48/M is a significant discount from frontier alternatives. For a team generating 100M output tokens/month:

V4-Flash: $28/month
V4-Pro: $348/month
Claude Opus 4.7: $2,500/month
GPT-5.5: $3,000/month

The cash-per-quality tradeoff genuinely favors V4-Pro for most mid-complexity workloads that aren't specifically HMMT-level math or heavy factual recall.

Common Mistakes to Avoid

Migrating to the wrong tier: deepseek-reasoner → deepseek-v4-flash (not v4-pro). V4-Flash with thinking mode is the direct functional equivalent of deepseek-reasoner. You don't need to upgrade to V4-Pro just because you were using the reasoning model.

Not setting cache_control breakpoints: V4-Pro cache-hit input is $0.145/M versus $1.74/M for cache-miss — a 12x difference. For agent loops that repeat the same system prompt and tool definitions, prompt caching can cut input costs by 90%. Structure your messages to keep the cacheable prefix stable (system prompt → tools → documents → conversation history → current user message).

Ignoring the 50% off-peak discount: If you're running batch jobs, scheduling them during Beijing off-peak hours halves the cost. For teams in UTC±8 time zones this is trivial to configure; for US/EU teams, a simple cron job targeting the overnight batch window captures the discount without user-facing latency impact.

Assuming Thinking mode is always better: V4-Pro and V4-Flash both support Thinking mode, but enabling it adds latency and cost. Use it for complex multi-step problems where the reasoning chain genuinely helps — not for simple extraction or summarization tasks where it adds overhead without quality benefit.

Testing only on your original benchmark: SimpleQA-Verified at 57.9% is a real gap. If your application depends on factual knowledge retrieval (especially for niche or recent information), test V4-Pro against your specific dataset before committing — Gemini 3.1 Pro may outperform here even at higher cost.

FAQ

Q: Is July 24, 2026 a hard cutoff for deepseek-chat and deepseek-reasoner?

Yes. DeepSeek has stated that deepseek-chat and deepseek-reasoner will be fully retired and inaccessible after July 24, 2026, 15:59 UTC. Any request using those model names after that time will return an error. Plan your migration with time to test — migrating in the final week is risky for production systems.

Q: Do I need to change my base_url when migrating?

No. The base URL (https://api.deepseek.com) remains the same. Only the model parameter in your request body changes. The request and response shapes are unchanged, so existing parsing code requires no modification.

Q: What is the difference between V4-Pro and V4-Flash in thinking mode?

Both models support Thinking and Non-Thinking modes. The difference is capability ceiling: V4-Pro has 49B active parameters versus V4-Flash's 13B, which gives it substantially better performance on complex reasoning and coding tasks even in thinking mode. V4-Flash thinking mode is appropriate for moderately complex problems at lower cost; V4-Pro thinking mode is for tasks where you need the highest quality available.

Q: Can I self-host DeepSeek V4-Flash without special hardware?

V4-Flash at 284B total parameters (13B active per forward pass) is feasible on multi-GPU servers with 80GB+ VRAM total. Quantized variants via Unsloth lower the memory requirements further. V4-Pro self-hosting requires significantly more resources due to the 1.6T total parameter count.

Q: Is DeepSeek V4 suitable for agentic workflows with tool calling?

Yes, this is one of V4-Pro's demonstrated strengths. MCPAtlas score of 73.6% measures tool orchestration performance; BrowseComp at 83.4% covers autonomous search-and-retrieval agents. V4-Pro is competitive with frontier closed-source models on these benchmarks while costing 7-9x less per token.

Q: Does V4 support multimodal inputs?

Currently, both V4-Pro and V4-Flash are text-only. DeepSeek stated they are "working on incorporating multimodal capabilities," but no release date has been announced. For vision tasks, Gemini 3.1 or GPT-5.5 remain the options.

Key Takeaways

Migrate before July 24, 2026: deepseek-chat → deepseek-v4-flash, deepseek-reasoner → deepseek-v4-flash with thinking mode enabled. One line of code change.
V4-Flash is the default upgrade: cheaper than any frontier alternative at $0.28/M output, with 1M context and dual thinking modes built in.
V4-Pro for coding and agents: SWE-bench Verified 80.6%, LiveCodeBench 93.5, Codeforces 3206 — leads the field on coding benchmarks at $3.48/M output (1/7th of GPT-5.5).
Hybrid Attention Architecture makes 1M context practical: 27% of FLOP and 10% of KV cache vs V3.2, enabling long-context retrieval at inference cost that previously required far more hardware.
Self-hosting is viable for Flash: Apache 2.0 (V4-Pro) and MIT (V4-Flash) licenses, weights on Hugging Face, vLLM support. Flash's 13B active parameters make it runnable on a multi-GPU server.
Watch for factual recall gaps: SimpleQA at 57.9% means V4-Pro isn't the right choice for factual knowledge-heavy applications. Test on your specific dataset before committing.

Bottom Line

DeepSeek V4-Pro is the best cost-per-quality option for coding and agentic workloads in April 2026, period. V4-Flash replaces deepseek-chat at the same price tier with a much larger context window. Migrate both before July 24 — it's one line of code and there's no reason to wait.

Prefer a deep-dive walkthrough? Watch the full video on YouTube.

GPT-5.5 Spud: Unified Multimodal API — Developer Integration Guide

Jangwook Kim — Fri, 24 Apr 2026 08:27:56 +0000

OpenAI shipped GPT-5.5 on April 23, 2026 — six weeks after GPT-5.4 and one week after Anthropic released Claude Opus 4.7. The internal codename is "Spud," and the model is something genuinely different from the incremental updates that preceded it.

GPT-5.5 is the first fully retrained base model OpenAI has released since GPT-4.5. Every prior 5.x release was a tuned derivative of the same underlying architecture. Spud is not. It processes text, images, audio, and video inside a single unified system — no Whisper call for transcription, no DALL-E endpoint for image generation, no separate pipeline stitching the modalities together. One model, one API endpoint, all four modalities.

For developers building production applications, that architectural shift matters more than the benchmark numbers. Here is what you need to know.

Why GPT-5.5 Is a Different Kind of Release

The GPT-5.x lineage started with GPT-5 (August 2025), then cycled through 5.1, 5.2, 5.3, 5.4, and now 5.5. Most of those were targeted improvements — better reasoning in 5.2, faster latency in 5.3, stronger computer-use in 5.4. They shared the same fundamental architecture.

GPT-5.5 breaks from that pattern. OpenAI rebuilt the token embedding layer to unify all four modalities at the representation level. Text, audio frames, image patches, and video keyframes are projected into the same vector space from the start. Previous OpenAI models encoded modalities separately and fused them at a later layer; Spud does not. The result is that the model reasons across modalities rather than translating between them.

The practical consequence: you can send an audio file, a screenshot, and a text question in a single request, and the model understands the relationships between all three without any preprocessing on your side. If you have been building pipelines that pass audio through Whisper first, then feed the transcript to GPT, you have a genuine opportunity to simplify your stack.

Benchmark Performance: Where Spud Leads and Where It Doesn't

GPT-5.5 landed well on agent-oriented benchmarks and general knowledge, but the picture is more nuanced when you look at code quality head-to-head against Claude Opus 4.7.

Benchmark	GPT-5.5	Claude Opus 4.7	GPT-5.4
Terminal-Bench 2.0	82.7%	69.4%	~71%
SWE-Bench Pro	58.6%	64.3%	54.1%
Expert-SWE	73.1%	—	—
OSWorld-Verified	78.7%	78.0%	—
MCP-Atlas	75.3%	79.1%	—
MMLU	92.4%	—	89.1%
Hallucination Rate	60% lower vs 5.4	—	baseline
Best For	Agentic workflows	Precision code review	Lower cost

Terminal-Bench 2.0 measures complex command-line workflows — multi-step operations that require planning, tool invocation, iteration, and recovery from failed steps. GPT-5.5's 82.7% vs Opus 4.7's 69.4% is a meaningful gap for developers building autonomous agents that operate over CLI tools, file systems, or APIs.

SWE-Bench Pro is the inverse story. Claude Opus 4.7 holds a 5.7-point lead (64.3% vs 58.6%) on solving real GitHub issues. That benchmark rewards careful, precise code generation — the kind of output you want when a human will review the PR.

The practical read: GPT-5.5 is the stronger choice for autonomous, long-running agentic tasks. Opus 4.7 remains stronger for code generation where precision matters and a human reviews the output. OpenAI's own Agents SDK is built to run on GPT-5.5 by default.

Pricing and Tier Structure

GPT-5.5 is priced higher than GPT-5.4 but offset by significantly better token efficiency — OpenAI reports the model completes equivalent Codex tasks with fewer tokens and fewer retries.

Tier	Input (per 1M tokens)	Output (per 1M tokens)
GPT-5.5 Standard	$5.00	$30.00
GPT-5.5 Pro	$30.00	$180.00
Batch / Flex	$2.50	$15.00
Priority	$12.50	$75.00

For context: GPT-5.4 runs at $2.50/$15 standard, so GPT-5.5 doubles the per-token rate. OpenAI's position is that the token efficiency gain offsets the price increase in most workloads — and for multimodal tasks that previously required multiple API calls (Whisper + GPT + DALL-E), the consolidation often results in lower total cost.

GPT-5.5 Pro is aimed at scientific research, complex analysis, and the highest-stakes production tasks. For most teams, standard GPT-5.5 or batch mode will be the right starting point.

Context Window and Available Modalities

The context window is 1 million tokens in ChatGPT and the upcoming Responses/Chat Completions API. In Codex CLI, the window is fixed at 400K tokens across all subscription plans.

GPT-5.5 handles four modalities natively:

Text: Standard token-based processing, same API interface as GPT-5.4
Image: Send base64-encoded or URL-referenced images; the model generates images natively as output (no separate DALL-E call)
Audio: Send audio files directly; transcription and speech synthesis are handled within the same request
Video: Pass video files or frame sequences; the model analyzes temporal content and can describe, summarize, or reason about video

One million tokens of context is substantial. For reference, a typical 100K-line codebase fits in roughly 700K tokens. You can now send your entire medium-sized repository in a single API call — relevant for automated code review, architecture analysis, or repo-level refactoring tasks.

API Integration Guide

As of April 24, 2026, full Responses API and Chat Completions access is staged — currently available via Codex sign-in for developers, with the general API rollout described as "very soon." The model IDs to watch for are gpt-5.5 and gpt-5.5-pro.

Text-Only Request (Chat Completions)

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {"role": "system", "content": "You are a senior software engineer."},
        {"role": "user", "content": "Review this function for edge cases."}
    ],
    max_tokens=2048
)

print(response.choices[0].message.content)

Multimodal Request: Image + Text

import base64

with open("screenshot.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{image_data}"
                    }
                },
                {
                    "type": "text",
                    "text": "What errors are visible in this UI? List them with severity."
                }
            ]
        }
    ]
)

Multimodal Request: Audio Transcription + Analysis

with open("meeting.mp3", "rb") as f:
    audio_data = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "audio": {
                        "data": audio_data,
                        "format": "mp3"
                    }
                },
                {
                    "type": "text",
                    "text": "Summarize the action items from this recording."
                }
            ]
        }
    ]
)

Note that the audio modality interface follows the same pattern as image input — no separate Whisper API call needed. The transcript, analysis, and any follow-up reasoning happen in a single model inference.

Batch Processing for Cost Reduction

For high-volume workloads where latency is not critical, the Batch API cuts costs by half:

import json

batch_requests = [
    {
        "custom_id": f"request-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-5.5",
            "messages": [{"role": "user", "content": document}],
            "max_tokens": 1024
        }
    }
    for i, document in enumerate(documents)
]

with open("batch_requests.jsonl", "w") as f:
    for req in batch_requests:
        f.write(json.dumps(req) + "\n")

batch_file = client.files.create(
    file=open("batch_requests.jsonl", "rb"),
    purpose="batch"
)

batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

Batch results arrive within 24 hours at $2.50 per million input tokens — the most cost-efficient path for document processing, classification, or any workload that tolerates latency.

Agentic Use Cases Where GPT-5.5 Excels

The Terminal-Bench 2.0 lead is not an accident. GPT-5.5 was trained specifically for multi-step autonomous task completion. Three categories stand out:

1. Long-running CLI agent workflows. Tasks that involve: read a file, decide what to do, invoke a shell command, observe the output, retry on failure, and produce a final result. GPT-5.5 handles the iteration loop with significantly fewer stalls than 5.4. OpenAI reports it "uses significantly fewer tokens to complete the same Codex tasks" — meaning the model is better at deciding when it has enough information to act rather than asking for clarification.

2. Multimodal investigation pipelines. Security analysts reviewing screenshots, logs, and audio recordings in a single context window. Accessibility auditors comparing UI screenshots to specs. QA engineers sending screen recordings and asking the model to identify regressions. These workloads become dramatically simpler without the orchestration layer between modalities.

3. Code + documentation cross-referencing. Sending an entire codebase (within the 1M context) alongside its documentation and asking the model to identify inconsistencies. At GPT-5.4's context limit, this required chunking or vector search. At 1M tokens, many medium repos fit directly.

For Claude Code-style agentic coding, GPT-5.5 via Codex is OpenAI's answer. The Terminal-Bench lead suggests it is competitive for autonomous task completion; SWE-Bench Pro results suggest human-in-the-loop code review still favors Opus 4.7.

Common Mistakes When Integrating GPT-5.5

Assuming audio input replaces structured data. Sending audio files works, but for structured extraction (dates, numbers, names), the transcription quality benefits from an explicit instruction in the text part of the message. Don't assume the model will automatically apply structured output constraints to audio-derived content without being told.

Ignoring batch mode for classification tasks. If you are running GPT-5.5 on thousands of documents for classification, tagging, or summarization, and latency is not a constraint, not using the Batch API means you are paying 2× unnecessarily. Batch mode is half the price with no quality trade-off.

Over-estimating token efficiency gains for simple tasks. The token efficiency improvement is most pronounced on complex, multi-step tasks. For simple Q&A or single-turn completions, GPT-5.4 at half the price is often the better call. Reserve GPT-5.5 for workloads where the capability improvement justifies the cost.

Using Chat Completions when Responses API is available. The Responses API is the unified interface OpenAI is investing in going forward. It supports tool calling, multimodal output, and streaming in a more consistent way than Chat Completions. For new production integrations, build against Responses API from the start.

Sending full videos when frame sampling suffices. Video input works natively but large video files consume tokens proportionally to their duration. For most use cases, sampling key frames (every 5-10 seconds) and sending those as image inputs gives similar analysis quality at a fraction of the token cost.

FAQ

Q: When will GPT-5.5 be available in the general API?

OpenAI's announcement states it will be in the Responses and Chat Completions APIs "very soon." As of April 24, developers can access it through Codex. Watch the OpenAI API changelog for the rollout announcement.

Q: Is GPT-5.5 better than GPT-6 for everyday tasks?

GPT-6 (released April 14, 2026) sits above GPT-5.5 in OpenAI's model hierarchy with a 2M token context window and Symphony architecture. For the highest-stakes tasks, GPT-6 is the more capable option. For most production workloads — agentic coding, document analysis, multimodal pipelines — GPT-5.5 offers a better price-to-capability ratio.

Q: Does GPT-5.5 replace the Whisper API for audio transcription?

For most use cases, yes. GPT-5.5 handles audio input natively with equivalent transcription quality. The Whisper API remains available for dedicated transcription-only workloads where you do not need subsequent language model reasoning on the output.

Q: What is the model ID for GPT-5.5?

The model identifier is gpt-5.5 for standard and gpt-5.5-pro for the Pro tier. These identifiers are confirmed in the OpenAI changelog and will be available once the general API rollout completes.

Q: How does GPT-5.5 handle the Claude Mythos comparison?

VentureBeat's Terminal-Bench 2.0 results show GPT-5.5 at 82.7% narrowly ahead of Claude Mythos Preview. Mythos remains in gated access (approximately 50 organizations). For most developers, the relevant comparison is GPT-5.5 vs Claude Opus 4.7, where the trade-off is agentic tasks (GPT-5.5) vs precision code generation (Opus 4.7).

Key Takeaways

GPT-5.5 is a genuine architectural rebase, not an incremental fine-tune. It is the first fully retrained OpenAI model since GPT-4.5.
Native omnimodal processing means text, audio, image, and video share a single embedding space. One API call, no preprocessing pipelines.
Benchmark positioning is task-dependent: Terminal-Bench 2.0 (82.7%) and MMLU (92.4%) favor GPT-5.5; SWE-Bench Pro (58.6% vs 64.3%) still favors Claude Opus 4.7.
API pricing: $5/$30 per million tokens standard; half that on Batch. Worth the upgrade for complex multimodal or long-running agentic tasks; overkill for simple Q&A.
1M token context window opens up whole-repo analysis that previously required RAG or chunking.
General API access is staged — currently through Codex, full Responses/Chat Completions rollout imminent.

Bottom Line

GPT-5.5 is the right choice for developers building autonomous agents that operate over extended workflows, multimodal inputs, or large codebases. The unified omnimodal architecture genuinely simplifies application code that previously required multiple specialized API calls. For precision-focused code generation with human review, Claude Opus 4.7's SWE-Bench Pro advantage is still worth considering.

Prefer a deep-dive walkthrough? Watch the full video on YouTube.

>-

Jangwook Kim — Fri, 24 Apr 2026 06:50:05 +0000

Yesterday (April 23), OpenAI released GPT-5.5. One sentence in the announcement stood out.

"This is the first GPT flagship model designed as an agent runtime, not a chat assistant."

Whether that is marketing copy or a genuine architectural shift is hard to judge right away. GPT-5.1 through 5.4 were iterative improvements on the GPT-5 base — fine-tuning and RLHF layered on top of the same foundation. GPT-5.5, according to OpenAI, is the first fully retrained base model since GPT-4.5. MMLU 92.4%, SWE-bench 88.7%, Terminal-Bench 2.0 82.7% — those are the numbers they shipped alongside the announcement.

Setting the claim aside, April has been an unusually busy month for AI agents. Anthropic launched Claude Managed Agents into public beta on April 8, followed by the Claude Advisor Tool on April 9. GitHub Copilot Agent Mode reached GA in Q1. Cursor 3.0 Glass dropped in early April. In the span of a few weeks, every major AI coding and agent platform shipped a significant update. In that context, GPT-5.5 is worth examining carefully — particularly how it stacks up against Claude.

The Verdict: A Real Shift, But Not One That Demands Immediate Action

GPT-5.5 is a meaningful update. But the conclusion "every developer should switch right now" is wrong. Three reasons.

First, the API is not available yet. Right now, only ChatGPT Plus/Pro/Business/Enterprise users can access it. The API comes "after additional cybersecurity guardrail review," with no specific date given. Anyone saying they've "tested GPT-5.5 and it's great" has tested it through the ChatGPT interface — not in their own agent pipelines with tool calls and custom system prompts.

Second, the price doubled. Justifying that requires performance gains that clearly outweigh the cost increase. That cannot be verified independently until the API ships and third-party benchmarks appear.

Third, Anthropic's concurrent releases — Managed Agents and the Advisor Tool — are not just model improvements. They are infrastructure-layer upgrades: checkpointing, credential management, scoped permissions, long-running sessions, multi-agent coordination. "A smarter model" and "more reliable agent infrastructure" serve different needs, and the right choice depends on what problem your team is actually solving.

That said, GPT-5.5 is not being overhyped by nothing. SWE-bench 88.7% clears a threshold that no widely available model has crossed before on that benchmark, and a 6-week release cadence signals OpenAI is taking this competition seriously. Once production data accumulates post-API release, this assessment may change. For now, it is provisional.

What Actually Changed from Previous Models

To understand GPT-5.5, it helps to understand what GPT-5.1 through 5.4 were.

They were variations on a theme: take the GPT-5 base, apply targeted fine-tuning and reinforcement learning, push specific capabilities higher. Faster reasoning, more stable multimodal processing, better domain accuracy. This approach ships improvements quickly, but has a fundamental ceiling. Fine-tuning cannot fully instill patterns that require the base model to have internalized them during pre-training: complex multi-step tool call sequences, self-correction loops, long-context coherence.

GPT-5.5 retrains from the ground up. Two key changes per the announcement:

Pre-training data composition optimized for agentic tasks. Rather than predicting text in isolation, the model learned from substantially more multi-step tool-call sequences and self-correction trajectories. OpenAI did not release the exact data mix, but said "agentic workflow data representation was significantly increased relative to prior generations." What that actually means in practice — code execution traces, API call/response pairs, error recovery loops — is not public.

Speed and capability improved simultaneously. Response latency matches GPT-5.4 while benchmark scores are higher. This is OpenAI's claim from their announcement materials; independent latency measurement awaits API availability. Simple scaling would not typically achieve this — it likely reflects architectural optimization. The specific mechanism is outside my expertise, and I won't pretend otherwise.

The timing matters too. Six weeks from GPT-5.4. For context, previous major GPT releases were spaced 2–4 months apart. Anthropic announced Managed Agents and the Advisor Tool days before this. That is not a coincidence. The release cadence across the industry is compressing.

Why the Benchmark Numbers Deserve Skepticism

SWE-bench 88.7% is genuinely impressive. But using it to conclude "GPT-5.5 is way better at coding than Claude" is premature.

MMLU 92.4% — a knowledge recall benchmark. It measures how much a model has memorized, not how effectively it acts on multi-step problems. High MMLU and strong agentic performance are correlated but not the same thing. An agent that knows a lot but hallucinates tool arguments is still a bad agent.

SWE-bench 88.7% — more directly relevant. But the comparison point often cited is Claude Sonnet 4.6 plus the Opus advisor at 74.8% on SWE-bench Multilingual. GPT-5.5's 88.7% is on the original English SWE-bench. These are different test distributions. Comparing them is not an apples-to-apples exercise. Fair comparison requires identical evaluation conditions, which we do not have.

Terminal-Bench 2.0 at 82.7% — the most credible datapoint for the "agent runtime" claim. This benchmark measures what actually matters for agents: execute a command, interpret output, decide what to do next. Scoring 82.7% on this is consistent with the positioning and represents a meaningful capability for CLI-based agents and CI/CD integrations. That said, this is still an OpenAI self-reported number, and independent replication has not happened yet.

GDPval 84.9% — OpenAI's own benchmark, which most developers had not heard of before yesterday. Self-designed benchmarks can favor the model that designed them. Cite the source when you cite the number.

This pattern is familiar from the LLM API pricing comparison I did earlier this year: every major lab leads with the benchmarks that favor them. The situation has gotten worse, not better.

The Price Doubled — Who Can Absorb That?

This is the part that bothers me most about this release.

GPT-5.4: $2.50/1M input, $15/1M output

GPT-5.5: $5/1M input, $30/1M output

Exactly 2x. "Performance goes up, price goes up" sounds reasonable in the abstract. But agentic workflows disproportionately generate output tokens — multi-step reasoning chains, tool call results being processed, intermediate state being recorded, final responses being synthesized. Output costs hit agents harder than they hit chat applications.

GPT-5.5 Pro: $30/1M input, $180/1M output. That is the highest output token price among any major publicly available LLM. For high-stakes reasoning tasks that might justify it — but for most teams, this tier is not realistically accessible.

Let me make this concrete. Assume 500 agent tasks per day, 8,000 output tokens each:

GPT-5.4: 500 × 8,000 × $15/1M = $60/day, ~$1,800/month
GPT-5.5: 500 × 8,000 × $30/1M = $120/day, ~$3,600/month

That is a $1,800/month difference. Whether that is worth it depends on whether the task success rate improvement, the reduction in error handling costs, and the faster iteration time add up to more than $1,800/month of value. That calculation is team-specific and requires real production data — not benchmark comparisons.

Compare this to Anthropic's Claude Managed Agents at $0.08/session-hour plus standard token costs. Time-based billing is more predictable for long-running agent tasks, where token consumption can be hard to estimate in advance.

Claude vs. GPT-5.5: Which One to Use

There is no universal answer. But the decision factors are concrete enough to be useful.

When GPT-5.5 makes more sense. If your team is already deep in the OpenAI ecosystem — Azure OpenAI, Vercel AI SDK on OpenAI, GitHub Copilot integration — switching costs are lower. If raw coding performance on SWE-bench style tasks is your primary metric, GPT-5.5 has the higher self-reported numbers. If you are building a product on top of ChatGPT, GPT-5.5 aligns with what your end users already have access to through their ChatGPT subscription.

When Claude still makes more sense. As covered in Claude Code's 5 agentic workflow patterns, Claude's tool-use behavior is more granular and context handling is more stable across long sessions. Claude Managed Agents combined with the Advisor Tool offers meaningful cost efficiency: Sonnet 4.6 as the executor plus Opus as the advisor improves task success rates while reducing per-task cost by 11.9%, according to Anthropic's data. For agent workflows that run for minutes or hours, Claude's infrastructure layer — checkpointing, credential management, scoped permissions — makes a practical difference that model benchmarks do not capture.

The bigger factor: ecosystem and workflow integration. A few percentage points on a benchmark matters less than which SDK your existing codebase depends on and which tooling your team already understands. Switching models is not an API key swap. Prompt engineering, error handling patterns, tool schema design, retry strategies — all of these are model-specific and require rework. I have seen teams underestimate this and lose days to re-engineering what was already working.

For my own projects, I am staying with Claude for now. The Vercel AI SDK + Claude streaming agent work I did recently confirmed that Claude's streaming behavior is stable even when tool calls are interleaved with generation — a common requirement in production agents that is easy to underestimate until it breaks. Once GPT-5.5 API access opens up, I plan to run the same tasks and compare directly.

Decision Framework

Existing codebase deeply tied to OpenAI SDK? → Consider GPT-5.5
Agent infrastructure (checkpointing, long sessions, multi-agent) is the key need? → Claude Managed Agents
Cost predictability matters? → Claude Managed Agents' time-based billing is more stable
Cannot wait for independent benchmark validation? → Claude API is available now
Planning to compare when GPT-5.5 API ships? → Run on Claude now, evaluate later
Coding agents are the primary use case and cost is manageable? → Worth experimenting with GPT-5.5 when API opens

The real question is not "which model is better" — it is "which tool fits my team's current constraints and technical context." Both ecosystems are evolving fast. Assessments 3–6 months from now will look different.

Agent Model vs. Agent Infrastructure — Not the Same Problem

This is the part of the GPT-5.5 announcement I find most frustrating.

OpenAI positioned GPT-5.5 as an "agent runtime." But what Anthropic shipped with Managed Agents is a different layer entirely. Anthropic's answer is agent infrastructure, not just a better agent model: checkpointing, credential management, scoped permissions, multi-agent coordination, long-running session support — all provided at the platform level.

If GPT-5.5 is "a model optimized for agent runtimes," Managed Agents is "the infrastructure for running agents." Smarter engine versus more reliable rails. Both matter, but conflating them is a category error.

My read — and I hold this with appropriate uncertainty — is that whoever establishes the infrastructure standard for production agents will have a durable advantage over whoever just has the best benchmark scores. Benchmarks are quarterly. Infrastructure lock-in is much stickier. As I explored in the 2026 AI agent framework comparison, the agent ecosystem is converging toward integrated framework-plus-infrastructure stacks, and early positioning there tends to be self-reinforcing.

The Questions This Release Leaves Open

Several things remain unclear after the announcement.

API availability timeline is vague. "After additional cybersecurity guardrail review" provides no schedule. Positioning a model as an agent runtime while keeping it inaccessible to API developers creates friction between the marketing claim and the developer experience. Claude Managed Agents was available to API users from day one.

The "agent runtime" claim lacks infrastructure specifics. The benchmarks support the claim at the model performance layer. But what does "agent runtime design" mean for checkpointing? For long-running sessions? For credential handling? The announcement is heavy on numbers and light on architectural detail.

Pro tier pricing is difficult to justify without independent data. $180/1M output is the highest in the mainstream LLM market. Justifying that requires demonstrably superior task completion rates — which we cannot evaluate until the API ships and production data accumulates.

One more thing worth noting: if GPT-5.5 is heavily optimized for agentic tasks, it may show little improvement over GPT-5.4 for simple conversation use cases. For users who interact with ChatGPT as a writing assistant or search tool rather than an agent platform, GPT-5.5 is likely to feel like a more expensive GPT-5.4.

GPT-5.5 is a real release that deserves attention. The benchmark numbers, the positioning pivot toward agent runtimes, the accelerated release cadence — taken together, they signal that the competition for the agentic AI platform is intensifying faster than most predicted.

But there is no reason to move production workloads to GPT-5.5 today. The API is not available. The price doubled. Independent evaluation is pending. And Anthropic is not standing still — they are building infrastructure that complements the model layer in ways GPT-5.5's announcement did not match.

Who wins the production agent standard war will not be decided by a benchmark press release. Developer experience, pricing, reliability, and infrastructure depth will be the actual differentiators. That race is still very open.

When GPT-5.5 API access opens, I intend to test the same agent workflows against the Claude Managed Agents stack — same prompts, same tasks, same success criteria. That comparison will be more informative than anything in today's announcement. My conclusion will follow from what actually happens when the rubber meets the road.

Cursor 2.0: 8 Parallel AI Agents and Visual Editor Bridge

Jangwook Kim — Fri, 24 Apr 2026 04:22:43 +0000

Cursor 2.0: 8 Parallel AI Agents and Visual Editor Bridge

8.6
/ 10



  <span>Parallel Agent Performance</span>

  <span>9.0</span>


  <span>Composer Model Speed</span>

  <span>8.8</span>


  <span>Visual Editor Usability</span>

  <span>8.2</span>


  <span>Value for Money</span>

  <span>8.5</span>


  <span>Workflow Integration</span>

  <span>8.7</span>

Cursor 2.0 launched on October 29, 2025, and it drew a hard line in AI-assisted development. For the first time, Anysphere shipped its own proprietary coding model — Composer — and rebuilt the agent interface around one idea: multiple agents working in parallel, each in its own isolated environment, without stepping on each other.

By April 2026, when Cursor 3 arrived with its full Agents Window redesign, the foundation was already established in 2.0. If you want to understand why Cursor's parallel agent architecture works the way it does — and how to use it effectively — 2.0 is where it started. This review covers what Cursor 2.0 introduced, how it evolved into 2.2's Visual Editor Bridge, and whether the approach holds up.

What Is Cursor 2.0?

Cursor is a VS Code fork that puts AI agents at the center of the development workflow. Unlike GitHub Copilot, which suggests code inline, or Claude Code, which runs in your terminal, Cursor embeds the agent directly into the IDE with access to your entire codebase.

Cursor 2.0 was the first version to ship a parallel agent UI alongside Anysphere's own model. Before 2.0, agents ran one at a time — you had to wait for one task to finish before starting another. The 2.0 release broke that constraint entirely.

Three things defined the 2.0 milestone:

Composer — Anysphere's first proprietary coding model, built specifically for low-latency agentic work
Parallel Agents UI — a new interface that makes running up to 8 agents simultaneously feel manageable rather than chaotic
Git Worktrees integration — the under-the-hood mechanism that makes true isolation possible

Everything in Cursor 3 — the Agents Window, Design Mode, cloud agents — built on top of what 2.0 proved out.

The Composer Model: Cursor's First Proprietary AI

Until Cursor 2.0, the IDE routed all requests to third-party models: Claude, GPT-4o, Gemini. Composer changed that. It is Anysphere's own frontier model, trained specifically for agentic coding tasks inside Cursor.

The headline numbers: Composer generates at 250 tokens per second and completes most agent turns in under 30 seconds. In benchmarks comparing average task completion time, Composer finished in 62 seconds against GitHub Copilot's 89 seconds. The trade-off is accuracy — Composer sits at around 51.7% success rate on SWE-Bench style tasks compared to Copilot's 56.5%. Speed wins over correctness in many interactive development loops, where the developer is reviewing and steering the agent anyway.

What makes Composer different from simply routing to a fast model is that it was trained with codebase-aware tools: semantic search across the full repo, file reading, diff generation, and test execution. It is not a general-purpose model given a system prompt — it is a purpose-built model that understands the specific operations a coding agent needs.

In practice, Composer is the default for agent tasks in Auto mode. You can still choose Claude Sonnet 4.6, GPT-6, or any other model from the model picker, but doing so draws from your monthly credit pool. Composer in Auto mode is unlimited on all paid plans.

Running 8 Agents in Parallel

The parallel agent interface is the feature most developers asked about after the 2.0 release. Here is how it actually works.

When you open the Agents panel in Cursor 2.0, you see a list view of active agents, each with their task description, status, and current diff. You can start a new agent with Cmd+Shift+A (macOS) or Ctrl+Shift+A (Windows/Linux). Each agent gets its own conversation thread and its own working environment.

You can run up to 8 agents simultaneously. In practice, the most useful pattern is task specialization: one agent handles a feature branch, another fixes failing tests, a third does documentation updates. They work simultaneously and independently.

Starting a parallel agent session looks like this:

# Agent 1: Add feature
Refactor the UserService class to support OAuth token refresh. 
Tests are in tests/user_service_test.py.

# Agent 2: Fix tests
The integration tests in tests/api/ are failing with a 401 
after the recent middleware changes. Diagnose and fix.

# Agent 3: Update docs
Update the API reference in docs/api.md to reflect the 
/auth/refresh endpoint added in PR #412.

Each agent runs in parallel. You switch between them in the sidebar like switching browser tabs.

How Git Worktrees Enable True Isolation

The reason parallel agents do not conflict with each other is git worktrees. This is the technical foundation that makes 2.0's parallel agents actually safe.

A git worktree creates a separate working copy of your repository that shares the same .git directory — same commit history, same branches — but with independent file state. When Cursor spawns an agent, it creates a dedicated worktree for that agent. Changes the agent makes exist only in that worktree until you explicitly merge them.

You can also create a worktree manually:

# Cursor creates these automatically, but you can manage them directly
git worktree add ../cursor-agent-feature feature/oauth-refresh
git worktree add ../cursor-agent-fix-tests fix/integration-401
git worktree list

Output:

/Users/you/myproject          abc1234 [main]
/Users/you/cursor-agent-feature  def5678 [feature/oauth-refresh]
/Users/you/cursor-agent-fix-tests  789abcd [fix/integration-401]

Each agent sees a clean state. If agent 2 modifies src/auth.ts, agent 1 does not see that change. When each agent finishes, you review the diff and merge — or discard — independently. Cursor manages worktree creation and cleanup automatically when agents start and stop.

The /best-of-n Command

Cursor 2.0 also shipped /best-of-n, a command that runs the same task across multiple models simultaneously. You invoke it in the agent chat:

/best-of-n Write a function that validates JWT tokens and handles 
clock skew within a 30-second window

Cursor spins up N agents (default: 3, configurable), each using a different model — Composer, Claude Sonnet 4.6, GPT-6 — and runs them in parallel in separate worktrees. When they finish, you get a side-by-side comparison of the three results and choose which to keep.

For harder tasks where correctness matters more than speed, best-of-n materially improves output quality. Anysphere found that comparing results across models and picking the best one increased success rates by roughly 20% on complex coding tasks.

The Visual Editor Bridge

Cursor 2.2, released after the initial 2.0 launch, added the Visual Editor for Cursor Browser — commonly called the "visual editor bridge." This feature connects the agent's code output to a visual representation of the running UI.

When you have Cursor's built-in browser open (accessible via Cmd+Shift+B), the Visual Editor lets you:

Click on any element in the rendered UI to select it
Annotate that element with a note (e.g., "move this button 8px to the right")
Add the annotation directly to the agent conversation

Instead of describing UI changes in text ("the submit button in the footer needs more padding"), you click the button, add a comment, and the agent receives precise element-level context. It reads the annotation, locates the relevant component in the codebase, and makes the change.

This is particularly useful for front-end work where the gap between "I see the problem" and "I can describe the problem in words" costs time. The visual editor bridge closes that gap.

Note: In Cursor 3, this concept expanded significantly into Design Mode, which works directly in the Agents Window and supports drag-select annotations, keyboard shortcuts for targeting elements, and canvas visualizations. But the visual editor bridge in 2.2 established the pattern.

Strengths
<ul>
  <li>Up to 8 true parallel agents via git worktrees — no file conflicts between agents</li>
  <li>Composer generates at 250 tokens/sec, completing most turns in under 30 seconds</li>
  <li>/best-of-n improves output quality on complex tasks by comparing across models</li>
  <li>Visual editor bridge removes the verbosity of describing UI changes in text</li>
  <li>Auto mode is unlimited on all paid plans — Composer has no credit cost</li>
</ul>


Limitations
<ul>
  <li>Composer accuracy (51.7%) trails some third-party models — matters for critical tasks</li>
  <li>Managing 8 agents simultaneously is cognitively demanding without a clear task system</li>
  <li>Visual editor requires Cursor's built-in browser — doesn't work with external browsers</li>
  <li>Credit pool system can be confusing: manual model selection draws from pool, Auto doesn't</li>
</ul>

Pricing Breakdown

Cursor's credit system, introduced in June 2025, ties your monthly usage allowance to your plan price in dollars. Auto mode (using Composer) is unlimited and costs zero credits on all paid plans. Manually picking a premium model draws from your pool at rates that vary by model.

Plan	Price	Credit Pool	Best For
Hobby	Free	Limited	Trying it out
Pro	$20/mo	$20/mo	Solo developers
Pro+	$60/mo	$60/mo	Power users, heavy parallel agent use
Ultra	$200/mo	$200/mo	Intensive daily use, 20x Pro usage
Teams	$40/user/mo	Per user	Engineering teams, SSO, admin controls
Enterprise	Custom	Pooled org usage	Large organizations, compliance needs

For most individual developers who rely primarily on Composer in Auto mode, Pro at $20/month covers the vast majority of work. The credit pool only matters when you deliberately choose premium third-party models. If you run /best-of-n frequently with GPT-6 or Claude Sonnet 4.6, Pro+ makes more financial sense.

Annual billing saves 20% on Pro ($16/month instead of $20).

Who Should Use Cursor 2.0?

Good fit

Full-stack developers who regularly switch between features, tests, and documentation — parallel agents let you delegate each to a separate agent simultaneously
Teams moving from GitHub Copilot who want a more complete agentic experience beyond inline suggestions
Developers who value speed over perfect first-pass accuracy — Composer is fast and Cursor's review interface makes iterating on agent output quick
Front-end developers who want to give agents visual UI feedback without writing long textual descriptions

Less ideal

Developers who need maximum accuracy on complex tasks and cannot afford iteration — in that scenario, routing manually to Claude Sonnet 4.6 or GPT-6 (with the credit cost) makes more sense than defaulting to Composer
Vim or Neovim users who do not want to leave their editor — Cursor is VS Code-based
Projects with strict data residency requirements that haven't yet evaluated Cursor's enterprise data handling policies

FAQ

Q: How does Cursor 2.0 differ from Cursor 3?

Cursor 2.0 introduced parallel agents, the Composer model, and git worktrees for isolation. Cursor 3 (April 2, 2026) rebuilt the entire interface into the Agents Window, added Design Mode for precise UI annotations, and introduced cloud agents and the /best-of-n improvements. The 2.0 features are all still present in 3.0 — 3.0 is an evolution, not a replacement architecture.

Q: Does using 8 parallel agents multiply my credit consumption?

Only if you are using premium third-party models manually. Composer in Auto mode costs zero credits, so 8 parallel agents running in Auto mode are effectively free of per-token cost. If you use /best-of-n with Claude or GPT-6, each model run draws from your pool. For heavy /best-of-n users, Pro+ at $60/month is the practical entry point.

Q: Can I use Cursor 2.0's parallel agents on monorepos?

Yes, and monorepos are actually a strong use case. Because each agent runs in a git worktree, they share the repository's full history and branch state but maintain independent file copies. Multiple agents can work on different packages in the monorepo simultaneously without conflicts. Cursor handles worktree creation and cleanup automatically.

Q: Does the visual editor bridge work with React, Vue, and other frameworks?

The visual editor works with any framework that renders in Cursor's built-in Chromium browser. React, Vue, Svelte, Angular — if it renders in the browser, the visual editor can target it. The agent uses the DOM structure and component metadata to locate the relevant code in your project.

Key Takeaways

Cursor 2.0 changed the fundamental unit of AI-assisted development from a single conversation to a parallel team of agents. The Composer model provided the speed necessary to make that practical — waiting 4 minutes for a single response makes parallel agents useless; waiting 62 seconds makes them very useful.

The git worktree architecture is the part that makes it real rather than theoretical. True file isolation means you can genuinely run 8 agents without fear of them corrupting each other's work. The visual editor bridge, arriving in 2.2, added a dimension of feedback precision that text prompts cannot match.

If you are evaluating Cursor today in April 2026, you are looking at Cursor 3's interface on top of this foundation. But understanding what 2.0 built explains why the platform is structured the way it is — and how to use parallel agents effectively rather than just spawning them and hoping for the best.

Bottom Line

Cursor 2.0 earned its reputation by making parallel agents actually work — not as a demo, but as a daily workflow tool backed by git worktrees and a purpose-built model fast enough to justify the parallel overhead. If your development workflow involves managing more than one concern at a time (and whose doesn't?), this is the release that made Cursor worth serious consideration.

Prefer a deep-dive walkthrough? Watch the full video on YouTube.

Llama 4 Maverick: 400B MoE Model — Self-Hosting and API Guide

Jangwook Kim — Fri, 24 Apr 2026 00:18:51 +0000

Why Llama 4 Maverick Matters

In April 2025, Meta shipped the Llama 4 family and reset what open-weight models can do. Llama 4 Maverick — the mid-tier model in the three-model lineup (Scout, Maverick, Behemoth) — packs 400 billion total parameters into a mixture-of-experts design that only activates 17 billion at inference time. That combination delivers near-frontier multimodal performance at a fraction of the compute cost compared to running a dense 400B model.

What makes Maverick uniquely interesting for infrastructure teams: the weights are free. You can run it inside your own VPC, own the full data pipeline, and still get a model that beats GPT-4o and Gemini 2.0 Flash on most multimodal benchmarks. For teams that cannot send data to a third-party API — finance, healthcare, defense — Maverick is the strongest open-weight option available.

This guide covers everything you need to put Maverick into production: the architecture, the real hardware costs, step-by-step vLLM setup, which managed API to pick if you don't want to self-host, and how benchmarks stack up against proprietary alternatives.

The Architecture: MoE in Practice

Mixture of Experts — the active parameter trick

Llama 4 Maverick uses an alternating dense and MoE layer architecture. In every MoE layer, each token activates:

One shared expert (always active)
One of 128 routed experts (selected per token by a learned router)

The router is a small linear layer that picks which expert processes each token. At inference time, only around 17 billion of the 400 billion parameters do actual computation. The remaining ~383B parameters sit in VRAM, cold.

This creates the central hardware trade-off: you pay in memory for all 400B parameters, but you pay in compute only for 17B. For batch inference with large batches, this is a significant throughput advantage. For latency-sensitive, small-batch workloads, the advantage shrinks because you still need all weights loaded.

Early fusion multimodality

Unlike previous Llama releases that handled vision through a separate projection head bolted onto the text model, Llama 4 uses early fusion: image patches and text tokens are encoded into the same embedding space from the first transformer layer. There is no separate visual encoder like a CLIP tower. The practical effect is more natural interleaving of visual and textual reasoning — the model can reference specific image regions mid-sentence rather than treating vision as a separate step.

Context window: Maverick vs Scout

A common point of confusion: Maverick supports up to 1 million tokens. Scout — the smaller sibling with 16 experts instead of 128 — is the one that reaches 10 million tokens. The trade-off is quality: Maverick's 128-expert depth produces significantly stronger scores on reasoning and coding benchmarks. For most production workloads under 500K tokens, Maverick is the right pick. If you genuinely need multi-million-token context, Scout is the tool.

In practice, hardware determines how close to Maverick's 1M ceiling you can operate:

8× H100 80GB (BF16): approximately 430K tokens
8× H200 141GB (BF16): full 1M tokens

Hardware Requirements

Self-hosting Maverick is not a single-GPU affair. Here are the real numbers:

Format	VRAM Required	Minimum Hardware
BF16 (full precision)	~800 GB total	8× H100 80GB or 10× A100 80GB
FP8 quantized	~400 GB total	4× H100 80GB or 4× H200
INT4 quantized	~200 GB total	2× H200 or 8× A40 48GB

Important: VRAM requirements above cover model weights only. Add 20–50% headroom for the KV cache at production context lengths.

For most teams, FP8 Maverick on 4× H100 80GB is the practical entry point. Benchmark quality versus BF16 on your specific use case before committing — FP8 typically shows near-identical MMLU and HumanEval scores with roughly half the memory cost.

On cloud hardware, an 8× H100 node (AWS p4de.24xlarge) runs approximately $32/hour on-demand. FP8 on 4× H100s cuts that by half. If you expect sustained load, reserved instances reduce the effective rate to $12–16/hour.

Self-Hosting with vLLM

vLLM v0.8.3 introduced full Llama 4 Maverick support, including FP8 quantization and multimodal inputs.

Step 1: Install vLLM

pip install "vllm>=0.8.3"

Step 2: Download weights from Hugging Face

Accept Meta's Llama 4 Community License at huggingface.co before downloading:

huggingface-cli login
huggingface-cli download meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --local-dir ./llama4-maverick

Step 3: Launch the inference server

BF16 on 8× H100:

vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --tensor-parallel-size 8 \
  --max-model-len 400000 \
  --dtype bfloat16 \
  --trust-remote-code

FP8 on 4× H100 (recommended starting point):

vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 200000 \
  --dtype fp8 \
  --quantization fp8 \
  --trust-remote-code

--trust-remote-code is required. Maverick's MoE routing uses custom code that vLLM cannot load without it.

Step 4: Test with the OpenAI-compatible endpoint

vLLM exposes a /v1/chat/completions endpoint. Any library built for the OpenAI SDK works without changes:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="none"
)

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct",
    messages=[
        {"role": "user", "content": "Explain MoE routing in under 100 words."}
    ],
    max_tokens=200
)

print(response.choices[0].message.content)

Multimodal requests

Pass image URLs alongside text in the content array — no separate endpoint needed:

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/architecture-diagram.png"}
                },
                {
                    "type": "text",
                    "text": "What components does this architecture diagram show?"
                }
            ]
        }
    ]
)

Early fusion means the model handles interleaved text and images in a single pass with no additional latency overhead from a separate vision model.

API Providers (No Self-Hosting Required)

If 8× H100s are not in your budget, four managed providers offer production Maverick access:

Provider	Input (per M tokens)	Output (per M tokens)	Max Context	Best For
Groq	$0.50	$0.77	128K	Lowest latency, dev/test
Fireworks AI	$0.22	$0.88	1M	Production, full context
Together AI	$0.27	$0.85	1M	Inference + fine-tuning
OpenRouter	$0.15	$0.60	1M	Cost-optimized routing

Groq uses LPU inference hardware that delivers near-instant token generation for short responses. The 128K context cap is the main constraint — workloads requiring long context need a different provider.

Fireworks AI was among the first to serve Maverick (minutes after Meta published the weights) and has a well-tested production configuration. Full 1M context, competitive pricing, strong reliability track record.

Together AI is the best pick if you plan to fine-tune Maverick later. They offer both inference and fine-tuning on the same platform, so you keep the full model lifecycle in one place.

OpenRouter routes requests dynamically to the cheapest available backend. Lowest floor price, but you sacrifice control over which provider handles each request.

All four expose OpenAI-compatible endpoints. Change base_url and api_key and your existing code runs without modification.

Benchmarks

Meta released Llama 4 Maverick on April 5, 2025. Here are the official benchmark scores from Meta's release announcement (0-shot, temperature=0, no majority voting):

Benchmark	Llama 4 Maverick	Notes
MMLU Pro	80.5	Knowledge reasoning
GPQA Diamond	69.8	Graduate-level science
HumanEval	82.4%	Code generation
DocVQA (test)	94.4	Document understanding
ChartQA	90.0	Chart comprehension
MMMU	73.4	Multimodal understanding
Multilingual MMLU	84.6	12-language coverage

At launch, Maverick outperformed GPT-4o and Gemini 2.0 Flash on the multimodal benchmarks (DocVQA, ChartQA, MMMU) while scoring comparably to DeepSeek V3 on coding. In 2026, newer frontier models (GPT-5.4, Gemini 3.1 Pro) have raised the ceiling on reasoning benchmarks, but Maverick's multimodal document understanding scores remain competitive with commercial APIs.

The comparison that matters for most cost-conscious teams: at $0.50/$0.77 on Groq, Maverick delivers strong multimodal capability at 9–23x lower per-token cost than comparable proprietary alternatives. If your workload is document processing, visual analysis, or multilingual generation rather than cutting-edge math competitions, Maverick's price-performance ratio is difficult to match with any closed-source model.

Common Mistakes

Loading all weights onto one GPU: 400B parameters never fit a single GPU at any practical precision. Always specify --tensor-parallel-size matching your GPU count.

Confusing Maverick and Scout context windows: Scout reaches 10 million tokens; Maverick reaches 1 million. If you need to ingest multi-million-token codebases, use Scout. For most production workloads under 500K tokens, Maverick's stronger reasoning quality wins.

Skipping FP8 without benchmarking: FP8 Maverick on 4× H100 80GB achieves near-identical MMLU and HumanEval scores versus BF16 for most tasks. Defaulting to BF16 without checking doubles your hardware cost.

Underestimating KV cache memory: A 400K token context in BF16 requires roughly 160GB of KV cache alone, on top of model weights. Budget VRAM with KV cache included, not just model size.

Omitting --trust-remote-code: Maverick's custom routing code will not load without this flag. The server silently fails model initialization if you forget it.

FAQ

Q: Does Llama 4 Maverick support function calling?

Yes. The Instruct variant was trained with tool use. The format mirrors the OpenAI function calling spec — pass a tools array in the chat completion request. Groq, Fireworks AI, and Together AI all support it in their Maverick deployments.

Q: What's the practical difference between Llama 4 Scout and Maverick?

Scout has 16 experts (Maverick has 128), a 10M token context window (Maverick has 1M), and runs on a single H100 80GB node. Maverick scores 4–8% higher on reasoning and coding benchmarks. Use Scout when context length is the bottleneck; use Maverick when quality is.

Q: Can I fine-tune Llama 4 Maverick?

Yes, under the Llama 4 Community License. Fine-tuning with LoRA at FP8 requires at least 4× H100 80GB. Together AI and Fireworks AI both offer managed fine-tuning pipelines. A full fine-tune on a 50K sample dataset typically takes 3–7 days on that configuration.

Q: Is Llama 4 Maverick production-ready in 2026?

Yes. Fireworks AI reports Maverick as one of their five most-requested models. Multiple organizations are running it in production. The main operational concern is node-level failure: if one H100 in an 8-GPU setup fails, the full model instance goes down. Plan for multi-node redundancy if you have uptime SLA requirements.

Q: How does Maverick perform on long-context tasks?

At 1M token context, retrieval quality degrades noticeably past ~400K tokens on needle-in-a-haystack benchmarks. For RAG workloads, semantic retrieval to a 200K or smaller context window still outperforms stuffing 800K raw tokens into the model. Use full 1M context for tasks where the model needs to reason across the entire corpus, not for general production RAG.

Key Takeaways

Architecture: 400B total / 17B active parameters, 128 experts, alternating dense and MoE layers. Early fusion handles text and images in a single model.
Context window: 1M tokens for Maverick (Scout is the 10M variant).
Hardware floor: 4× H100 80GB at FP8, 8× H100 80GB at BF16. Multi-GPU is mandatory.
vLLM setup: v0.8.3 or later, --tensor-parallel-size, --trust-remote-code required.
Managed APIs: Groq for speed, Fireworks and Together for full 1M context, OpenRouter for lowest cost.
Benchmark position: Beats GPT-4o on multimodal document tasks; slightly behind on pure math and coding. 9–23x cheaper per token on managed APIs.

Bottom Line

Llama 4 Maverick is the strongest open-weight multimodal model available and the only practical choice for teams that need frontier-class performance with full data ownership. The 4× H100 FP8 entry point is real infrastructure investment, but it buys you a model that beats GPT-4o on document understanding at a fraction of the API cost. If self-hosting is off the table, Fireworks AI or Groq give you managed access at 9–23x lower per-token pricing than comparable proprietary models.

Prefer a deep-dive walkthrough? Watch the full video on YouTube.

Databricks Unity AI Gateway: MCP Agent Governance Guide

Jangwook Kim — Thu, 23 Apr 2026 08:26:31 +0000

Enterprise AI adoption has hit a governance wall. Organizations that rushed to deploy LLM-powered applications now face an uncomfortable reality: dozens of agents making API calls across multiple providers, MCP servers accessing sensitive data without proper audit trails, and no unified way to track what any of it costs. Databricks calls this "agent sprawl," and in April 2026 they shipped a direct answer: Unity AI Gateway.

This guide covers what Unity AI Gateway actually does, how its MCP governance model works in practice, and where it fits in the broader enterprise AI infrastructure stack.

Why This Matters: The Agent Sprawl Problem

The shift to agentic AI workflows created a governance gap that earlier tooling wasn't designed to handle. A single production agent might:

Call three different LLM providers in the same session
Invoke five external MCP servers to access Slack, GitHub, and internal databases
Run as a shared service account with broader permissions than any human would be granted
Generate costs that get attributed to a catch-all "AI budget" line item

Traditional cloud IAM controls weren't designed for this pattern. You can restrict what a service account can do at the infrastructure level, but you can't easily say "this agent can use Claude for reasoning tasks but must route code generation to GPT-6, and can only access the GitHub MCP server if the requesting user has write access to that repo."

That's the problem Unity AI Gateway is designed to solve—not by adding another governance layer on top of your existing stack, but by extending the Unity Catalog permission model you may already use for data governance directly into your AI layer.

What is Unity AI Gateway?

Unity AI Gateway is the AI governance component of Databricks' Unity Catalog, extended to cover LLM endpoints, MCP servers, and coding agents. It was previously branded as Mosaic AI Gateway—the April 2026 rename to "Unity AI Gateway" signals the deeper integration with Unity Catalog's existing access control and audit infrastructure.

The core architecture positions AI Gateway as a proxy layer that sits between your agents and the external systems they call. Every request—whether it's an LLM completion from Anthropic's API or a tool call to a GitHub MCP server—passes through AI Gateway, where it's evaluated against access policies, monitored for compliance, and logged to a centralized audit table.

From a developer perspective, this is similar to how API gateways work in microservices architectures, but with two enterprise-specific additions: identity propagation (so the gateway knows who initiated the request, not just which service is making it) and Unity Catalog integration (so permissions are expressed in the same terms your data teams already use).

MCP Governance: The Key Differentiator

The most significant April 2026 addition is first-class MCP server governance. Model Context Protocol has gone from a research curiosity to standard infrastructure—97 million monthly SDK downloads as of March 2026—and most enterprise AI deployments now involve agents that use MCP servers to access internal systems.

The problem is that MCP servers are typically authenticated with service account credentials, which means every agent that connects gets the same access level regardless of who initiated the request. An agent helping a junior analyst might access the same financial data that a senior analyst would.

Unity AI Gateway addresses this with on-behalf-of (OBO) execution: when an agent calls an MCP server through AI Gateway, the server receives the requesting user's identity and permissions, not the agent's service account. The MCP server then enforces Unity Catalog permissions based on that user identity.

Every MCP server accessible through the workspace is registered in Unity Catalog as a catalog object. This means:

Discovery: Teams can browse available MCP servers in the same interface they use to find datasets and tables.
Access control: Admins grant or revoke MCP server access with the same GRANT and REVOKE syntax used for table permissions.
Audit logging: Every MCP call logs the requesting identity, connection name, HTTP method, and OBO status to a centralized audit table queryable via SQL.

This last point matters more than it might seem. When your compliance team asks "which agents accessed the customer data MCP server last quarter, and on whose behalf?", the answer is a SQL query rather than a multi-week log analysis project.

Managed vs. External MCP Servers

Databricks distinguishes between two server types:

Managed MCP servers are hosted by Databricks and pre-integrated with Unity Catalog. The initial set includes:

Genie: Natural language queries against your Databricks data
Vector Search: Semantic retrieval from indexed documents
UC Functions: Custom tools registered as Unity Catalog functions
DBSQL: Direct SQL execution against Unity Catalog tables

Managed servers inherit Unity Catalog permissions automatically—there's no additional configuration needed to enforce row-level security or column masking policies that already exist on your tables.

External MCP servers are third-party or self-hosted (GitHub, Slack, internal APIs). These are registered in Unity Catalog with a connection definition, and AI Gateway applies OBO auth when routing requests to them. Unity Catalog permissions control which users and service principals can access each external server.

LLM Safeguards: Beyond Simple Rate Limiting

AI Gateway's guardrail system has expanded significantly in 2026. The current feature set covers:

Rate Limiting

Rate limits apply at three granularities:

Endpoint level: Maximum requests per minute across all callers
User level: Per-identity limits to prevent runaway costs from a single misconfigured agent
Group level: Department or team-scoped budgets enforced at the request layer

When a request exceeds a rate limit, it receives a 429 response. Other agents sharing the endpoint are unaffected.

Automatic Failover

AI Gateway supports multi-model endpoints where multiple LLM providers are listed in priority order. When the primary model returns a 429 (rate limited) or 5XX (server error), the gateway automatically routes to the next listed model—no application code changes needed.

This is useful for reliability, but it's also a cost optimization mechanism: you can list an expensive frontier model first and a faster, cheaper model as fallback, catching cases where the premium model is unavailable rather than failing the request entirely.

LLM-Judge Guardrails

The guardrail system uses an LLM-judge approach—configurable with custom models and prompts—to enforce policies that can't be expressed as simple rules. Available checks include:

PII detection and redaction: Identify and mask personal information in inputs or outputs before logging
Content safety: Block or flag outputs that violate configured policies
Prompt injection defense: Detect attempts to override system instructions through user input
Data exfiltration prevention: Flag requests that appear to be extracting bulk data
Hallucination checks: Evaluate output confidence against retrieved context

Each guardrail is independently configurable. Violations result in request rejection or data masking, and all enforcement actions are logged. You can run guardrails on input, output, or both.

End-to-End Observability with MLflow Tracing

Governance without observability is incomplete—you need to know not just whether your policies are enforced but what your agents are actually doing at execution time. MLflow Tracing provides the second half of this picture.

When an agent runs through Databricks, MLflow automatically captures:

LLM calls: Model, prompt, response, token count, latency
MCP tool calls: Which server, which tool, inputs and outputs, execution time
Agent reasoning steps: The sequence of decisions that led to each tool call
Retrieval operations: Documents fetched, similarity scores, chunk boundaries

This trace data is OpenTelemetry-compatible, so it flows naturally into existing observability infrastructure. The Unity Catalog audit logs and MLflow traces complement each other: audit logs answer security and compliance questions ("who accessed what?"), while traces answer debugging and performance questions ("why did this agent make that tool call?").

Cost Attribution

One of the more practical capabilities is request tagging for cost attribution. Teams can attach custom tags to requests—project code, team name, user ID, deployment environment—and the system aggregates costs by those dimensions in Unity Catalog system tables.

This moves AI spend from a catch-all line item to something your finance team can actually work with. Product teams can see their LLM costs broken down by feature. Platform teams can identify which agents are consuming disproportionate resources. Budget alerts can trigger at the team or project level rather than only at the account level.

The DBU rate for Foundation Model API workloads starts at approximately $0.07 per DBU in the 2026 pricing, but the more significant value is the attribution clarity rather than the rate itself.

Practical Setup: Adding MCP Governance to an Existing Agent

Here's how the integration works in practice for a team that already has a Databricks workspace and wants to add governance to an agent that calls external MCP servers.

Step 1: Register External MCP Servers

External MCP servers are registered as Unity Catalog connections. Using the Databricks UI or Terraform:

CREATE CONNECTION github_mcp
TYPE HTTP
OPTIONS (
  host 'https://api.github.com',
  port '443'
);

GRANT USAGE ON CONNECTION github_mcp TO `data-engineering-team`;

Once registered, the server appears in the MCP Servers tab of the Agents workspace and is discoverable by other teams.

Step 2: Configure AI Gateway on LLM Endpoints

Enable AI Gateway on a serving endpoint through the UI or API:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import (
    AiGatewayConfig,
    AiGatewayGuardrails,
    AiGatewayRateLimit,
    AiGatewayUsageTrackingConfig
)

client = WorkspaceClient()

client.serving_endpoints.put_ai_gateway(
    name="production-llm-endpoint",
    ai_gateway=AiGatewayConfig(
        usage_tracking_config=AiGatewayUsageTrackingConfig(enabled=True),
        rate_limits=[
            AiGatewayRateLimit(
                calls=1000,
                renewal_period="minute",
                key="user"
            )
        ],
        guardrails=AiGatewayGuardrails(
            input_safety=True,
            pii_detection=True
        )
    )
)

Step 3: Route Agent Traffic Through AI Gateway

Agents call the AI Gateway endpoint rather than provider APIs directly. The endpoint URL is OpenAI-compatible, so most frameworks require only a base URL change:

from openai import OpenAI

client = OpenAI(
    api_key=databricks_token,
    base_url=f"https://{workspace_host}/serving-endpoints/production-llm-endpoint/v1"
)

# All requests now flow through AI Gateway
response = client.chat.completions.create(
    model="databricks-claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Summarize Q1 sales data"}]
)

Step 4: Query Audit Logs

Audit data lands in Unity Catalog system tables, queryable via standard SQL:

SELECT
    user_identity,
    request_id,
    timestamp,
    mcp_connection_name,
    tool_name,
    on_behalf_of_user,
    response_status_code
FROM system.ai_gateway.mcp_audit_logs
WHERE timestamp > '2026-04-01'
  AND mcp_connection_name = 'github_mcp'
ORDER BY timestamp DESC
LIMIT 100;

Common Mistakes to Avoid

Using service accounts for all agent traffic: OBO auth only works if agents pass user identity through the request chain. If your agent framework authenticates with a shared service account and doesn't propagate user context, all MCP calls will appear to originate from that account in the audit logs. Check that your agent framework supports identity forwarding before deploying.

Configuring guardrails in blocking mode without testing: LLM-judge guardrails have non-zero latency and false positive rates. Start guardrails in monitoring mode to understand the false positive rate on your actual traffic before switching to blocking mode in production.

Skipping rate limit configuration for internal tools: Teams often configure rate limits on external-facing endpoints but skip them for internal tools. A misconfigured internal agent can generate the same runaway costs—set limits everywhere, not just at the perimeter.

Over-permissioning managed MCP servers: The convenience of managed servers can lead to blanket grants ("grant data-team access to all MCP servers") instead of the principle of least privilege. Audit which servers each team actually uses and grant accordingly.

Not tagging requests for cost attribution: Tags need to be set when the request is made—retroactive attribution isn't possible. Establish a tagging convention at project start, not after the first billing surprise.

Bottom Line

Unity AI Gateway is the most complete enterprise AI governance platform available in 2026 for teams already on Databricks. The Unity Catalog integration means you're not adding a separate permission system—you're extending existing data governance to cover LLM calls and MCP tools. For organizations outside the Databricks ecosystem, the switching cost is high; alternatives like LiteLLM or Cloudflare AI Gateway provide a subset of the governance features without the platform lock-in.

How It Compares to Alternatives

Enterprise teams evaluating AI governance platforms typically consider three options besides Unity AI Gateway:

LiteLLM is the open-source alternative—140+ provider support, budget management, semantic caching, and self-hosted deployment. It lacks Unity Catalog integration and the OBO auth model for MCP servers, but it's a strong choice for teams that need multi-cloud LLM routing without vendor lock-in. We covered LiteLLM's setup in detail here.

Cloudflare AI Gateway handles edge caching, rate limiting, and spend controls at the CDN layer—zero code changes for basic observability. The governance model is simpler (no per-user identity propagation), making it better suited for customer-facing applications than internal agent workflows.

Native provider controls (Anthropic's system prompt policies, OpenAI's organization settings) provide some guardrails but don't unify multi-provider deployments and don't address the MCP governance problem at all.

Capability	Unity AI Gateway	LiteLLM	Cloudflare AI Gateway
Multi-provider LLM routing	Yes	Yes (140+ providers)	Yes
MCP server governance	Yes (OBO auth)	No	No
Per-user rate limiting	Yes	Yes	No
LLM-judge guardrails	Yes	Partial	No
End-to-end traces	Yes (MLflow)	Yes (Langfuse/Helicone)	Basic
Unity Catalog integration	Native	No	No
Self-host option	No (Databricks managed)	Yes	No
Best for	Databricks-native enterprises	Multi-cloud / open-source	Edge / customer-facing

FAQ

Q: Does Unity AI Gateway work with agents built outside Databricks?

Yes—any agent that can make HTTP requests to an OpenAI-compatible endpoint can route through AI Gateway. The gateway doesn't require Databricks-native agent frameworks. Identity propagation for OBO auth requires passing a user token in the request header, which most frameworks support via custom headers.

Q: How does OBO auth work when an agent initiates a multi-step workflow without a human in the loop?

For fully automated workflows without an active user session, OBO auth falls back to the service principal identity of the agent. The audit log records this as a service principal call rather than an end-user call. If your compliance requirements mandate user-level attribution for automated workflows, you'll need to either redesign the workflow to include human approval steps or accept service principal attribution for background tasks.

Q: Can I use Unity AI Gateway with Claude Code or Cursor?

Yes, as of April 2026, AI Gateway explicitly supports coding agent governance. The "Governing Coding Agent Sprawl" blog post from Databricks covers this use case in detail—you can route Claude Code and Cursor traffic through AI Gateway to enforce the same policies applied to other agents in your workspace.

Q: What's the latency overhead of routing through AI Gateway?

Databricks hasn't published precise latency benchmarks for AI Gateway overhead. In practice, the proxy layer adds single-digit millisecond overhead for policy evaluation on cached decisions. Guardrail evaluation—particularly LLM-judge checks—adds meaningful latency proportional to the complexity of the check. For latency-sensitive applications, configure guardrails in async monitoring mode rather than synchronous blocking mode.

Q: Is there a free tier for experimenting with Unity AI Gateway?

Unity AI Gateway is available on all Databricks workspace tiers, including the free trial. The Foundation Model API, which is the primary LLM endpoint type, charges at DBU rates that start around $0.07 per DBU. External provider pass-through endpoints (where you bring your own API key for Anthropic, OpenAI, etc.) incur DBU charges for the gateway itself but you pay the provider directly for model usage.

Key Takeaways

Unity AI Gateway is Unity Catalog's governance layer extended to LLM endpoints and MCP servers—the same permissions and audit infrastructure, applied to AI.
MCP governance is the standout April 2026 addition: every MCP server is a Unity Catalog object with fine-grained permissions and full audit logging, with OBO auth ensuring agents act with the requesting user's identity rather than a shared service account.
Rate limiting (endpoint/user/group), automatic failover across providers, and LLM-judge guardrails are all configurable without application code changes.
MLflow Tracing provides the debugging and performance visibility layer; Unity Catalog audit logs provide the compliance layer. They address different questions and are used together.
For teams outside the Databricks ecosystem, LiteLLM covers most of the LLM routing and cost control use cases; the Unity Catalog integration is the primary reason to stay on Databricks AI Gateway specifically.
Cost attribution via request tags is opt-in and must be configured before deployment—retroactive tagging isn't supported.

Prefer a deep-dive walkthrough? Watch the full video on YouTube.

Building a Claude Streaming Agent with Vercel AI SDK

Jangwook Kim — Thu, 23 Apr 2026 06:43:08 +0000

const result = streamText({
  model: anthropic('claude-sonnet-4-6'),
  prompt: 'Hello, streaming test',
});

for await (const chunk of result.textStream) {
  process.stdout.write(chunk);
}

The first time I ran this, I had an odd reaction. Not amazed that Claude was outputting text character by character — surprised that this actually worked in 5 lines. Compared to writing the Anthropic SDK directly, the setup code is less than half.

I first heard about Vercel AI SDK from a Slack link someone shared at work. The usual "AI chat in 10 minutes with Next.js" type of title — and usually these fall apart at the dependency installation step. I tried it skeptically and it actually worked quickly. I've been reaching for it during prototyping ever since.

After using Vercel AI SDK seriously for a while, the tradeoffs became clear. This post explains how to use it while being honest about where it gets complicated. If you've already used it, jump to the "Tool Calling" and "Production Considerations" sections.

Why Vercel AI SDK — A Direct Comparison

I tried the alternatives first. Direct Anthropic SDK, LangChain.js, then Vercel AI SDK.

Direct Anthropic SDK is most flexible, but the boilerplate for getting streaming responses to the frontend is more than expected. You're writing SSE format handling, frontend hooks, and error handling from scratch. The underlying capability is simple, but the code length grows unnecessarily.

// Direct Anthropic SDK streaming setup — gets longer than this
const stream = await anthropic.messages.stream({
  model: 'claude-sonnet-4-6',
  max_tokens: 1024,
  messages: [{ role: 'user', content: prompt }],
});

// Manually construct SSE response
const encoder = new TextEncoder();
const readable = new ReadableStream({
  async start(controller) {
    for await (const chunk of stream) {
      if (chunk.type === 'content_block_delta' && chunk.delta.type === 'text_delta') {
        controller.enqueue(encoder.encode(`data: ${JSON.stringify(chunk.delta.text)}\n\n`));
      }
    }
    controller.close();
  },
});

Add the frontend parsing code and you're looking at real volume. This is right when you need fine-grained control, but it's too much overhead for building a single chat app.

LangChain.js — I gave up on this one. Frequent API changes between versions, documentation that doesn't match actual behavior. Digging through GitHub issues often turns up "this feature has been removed" answers. Might work for complex pipelines, but not right for fast prototyping.

Vercel AI SDK's practical advantages come down to three things:

First, streamText() + useChat() gets server-to-client streaming wired up in under 10 lines. Second, switching between Claude, OpenAI, Gemini, and Mistral is a single provider line change — which turns out to be genuinely useful for comparing model outputs on the same code. Third, generateObject() with Zod schema validation gives you clean structured output handling.

The downsides: it's optimized for Vercel's platform, which creates friction in other deployment environments. When you need fine-grained control over the agent loop, you lose some flexibility compared to the Anthropic SDK directly. More on that later.

Compared to building directly with Claude Managed Agents, Managed Agents are easier to start without infrastructure but hit customization ceilings quickly. Vercel AI SDK sits between the two — more abstracted than raw SDK, more controllable than Managed Agents.

Environment Setup

Prerequisites:

Node.js 20+
Anthropic API key (ANTHROPIC_API_KEY)
Next.js 15 (App Router)

# New project
npx create-next-app@latest my-claude-app --typescript --app
cd my-claude-app

# Core AI SDK packages
npm install ai @ai-sdk/anthropic zod

Add your API key to .env.local:

ANTHROPIC_API_KEY=sk-ant-api03-...

One thing I ran into: @ai-sdk/anthropic installed fine, but a TypeScript type error appeared. The moduleResolution in tsconfig.json needs to be bundler or node16 or higher. create-next-app handles this in the default config.

Directory structure:

app/
├── api/
│   ├── chat/
│   │   └── route.ts      # Streaming chat API
│   └── extract/
│       └── route.ts      # generateObject API
├── page.tsx              # Chat UI
└── components/
    └── Message.tsx

Not complex by design — this structure is actually sufficient for a functional chat app.

Implementing Claude Streaming with streamText

Start with the server-side API route.

app/api/chat/route.ts:

import { streamText } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: anthropic('claude-sonnet-4-6'),
    system: `You are a helpful technical blog assistant.
Answer code questions practically, and be honest when you don't know something.
Keep responses concise but complete.`,
    messages,
    maxTokens: 2048,
    temperature: 0.7,
  });

  return result.toUIMessageStreamResponse();
}

toUIMessageStreamResponse() is the key piece. This single method handles SSE header setup, chunk formatting, and stream termination. Doing this manually with the Anthropic SDK takes 20+ lines minimum.

Frontend app/page.tsx:

'use client';

import { useChat } from 'ai/react';
import { useEffect, useRef } from 'react';

export default function ChatPage() {
  const { messages, input, handleInputChange, handleSubmit, isLoading, error } = useChat({
    api: '/api/chat',
  });
  const bottomRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    bottomRef.current?.scrollIntoView({ behavior: 'smooth' });
  }, [messages]);

  return (
    <div className="flex flex-col h-screen max-w-2xl mx-auto p-4">
      <div className="flex-1 overflow-y-auto space-y-4 pb-4">
        {messages.length === 0 && (
          <p className="text-gray-400 text-center mt-8">
            How can I help you?
          </p>
        )}
        {messages.map((m) => (
          <div
            key={m.id}
            className={`p-3 rounded-lg max-w-[85%] ${
              m.role === 'user' ? 'bg-blue-100 ml-auto' : 'bg-gray-100'
            }`}
          >
            <p className="text-xs text-gray-400 mb-1">
              {m.role === 'user' ? 'You' : 'Claude'}
            </p>
            <div className="whitespace-pre-wrap text-sm">
              {m.content as string}
            </div>
          </div>
        ))}
        {isLoading && (
          <div className="bg-gray-100 p-3 rounded-lg text-gray-400 text-sm">
            Claude is typing...
          </div>
        )}
        {error && (
          <div className="text-red-500 text-sm p-2 bg-red-50 rounded">
            Error: {error.message}
          </div>
        )}
        <div ref={bottomRef} />
      </div>

      <form onSubmit={handleSubmit} className="flex gap-2 mt-4 border-t pt-4">
        <input
          value={input}
          onChange={handleInputChange}
          placeholder="Type a message..."
          className="flex-1 border rounded-lg px-4 py-2 focus:outline-none focus:ring-2 focus:ring-blue-400"
          disabled={isLoading}
        />
        <button
          type="submit"
          disabled={isLoading || !input.trim()}
          className="bg-blue-500 text-white px-6 py-2 rounded-lg disabled:opacity-50"
        >
          Send
        </button>
      </form>
    </div>
  );
}

useChat handles message state, streaming updates, loading state, and error handling. Building this from scratch means writing useState, useRef, AbortController, SSE parsing, and retry logic.

Run npm run dev and the chat works at localhost:3000. Claude's response streams in character by character.

Tool Calling — Making Claude Actually Do Things

To go beyond chat and let Claude call external tools, add the tools option with maxSteps.

import { streamText, tool } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';
import { z } from 'zod';

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: anthropic('claude-sonnet-4-6'),
    system: 'You are an assistant that manages weather info and todo lists.',
    messages,
    maxSteps: 5,
    tools: {
      getWeather: tool({
        description: 'Gets the current weather for a specific city',
        parameters: z.object({
          city: z.string().describe('City name to get weather for'),
          unit: z.enum(['celsius', 'fahrenheit']).default('celsius'),
        }),
        execute: async ({ city, unit }) => {
          // Real implementation would call a weather API
          return {
            city,
            temperature: unit === 'celsius' ? 22 : 72,
            condition: 'Sunny',
            humidity: 65,
            feelsLike: unit === 'celsius' ? 20 : 68,
          };
        },
      }),
      addTodo: tool({
        description: 'Adds a new item to the todo list',
        parameters: z.object({
          title: z.string().describe('Todo title'),
          priority: z.enum(['low', 'medium', 'high']).default('medium'),
          dueDate: z.string().optional().describe('Due date (YYYY-MM-DD)'),
        }),
        execute: async ({ title, priority, dueDate }) => {
          const id = Math.random().toString(36).slice(2);
          return { id, title, priority, dueDate, created: new Date().toISOString() };
        },
      }),
    },
  });

  return result.toUIMessageStreamResponse();
}

maxSteps: 5 matters. Without it, Claude receives tool results but doesn't loop back to generate another response. The SDK handles this loop automatically — maxSteps caps the maximum iterations.

The pattern of agents combining multiple tools to solve problems depends heavily on maxSteps and the quality of each tool's description. Vague descriptions mean Claude can't decide when to use which tool. I had an early version where weather and todos were getting mixed up — adding explicit usage scenarios to the system prompt fixed it.

Real-time tool call progress in the frontend:

{messages.map((m) => (
  <div key={m.id}>
    {m.role === 'assistant' &&
      Array.isArray(m.toolInvocations) &&
      m.toolInvocations.map((ti) => (
        <div key={ti.toolCallId} className="text-xs text-gray-400 italic mb-1">
          {ti.state === 'call' && `⚙ Calling ${ti.toolName}...`}
          {ti.state === 'result' && `✓ ${ti.toolName} complete`}
        </div>
      ))
    }
    <div className="text-sm">{m.content as string}</div>
  </div>
))}

Extracting Structured Output with generateObject

Separate from streaming chat, use generateObject() when you need to pull specific structured data out of Claude's response.

// app/api/extract/route.ts
import { generateObject } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';
import { z } from 'zod';

const ArticleMetaSchema = z.object({
  title: z.string().describe('Post title (under 60 chars)'),
  summary: z.string().max(300).describe('3-4 sentence summary'),
  tags: z.array(z.string()).min(2).max(5).describe('Related technology tags'),
  difficulty: z.enum(['beginner', 'intermediate', 'advanced']),
  estimatedReadTime: z.number().int().describe('Estimated read time in minutes'),
  hasCodeExamples: z.boolean(),
  mainTopics: z.array(z.string()).max(3).describe('Up to 3 main topics'),
});

export async function POST(req: Request) {
  const { content } = await req.json();

  try {
    const { object } = await generateObject({
      model: anthropic('claude-sonnet-4-6'),
      schema: ArticleMetaSchema,
      prompt: `Analyze the following technical blog post and extract metadata.
Primary readers are backend and fullstack developers.

---
${content}
---`,
    });

    return Response.json(object);
  } catch (error) {
    return Response.json({ error: 'Analysis failed' }, { status: 500 });
  }
}

The response comes back as a type-safe object matching the Zod schema. No JSON parse errors or type mismatches to handle separately.

This pattern works well for: automated blog post tagging, user input classification, structured information extraction from documents, and form auto-completion.

I'm using a similar pattern for this blog's category score extraction. Writing good describe() text on Zod schema fields is the key to better output quality. Good context engineering means schema design and prompt quality determine 80% of extraction accuracy.

streamObject() is also available — useful when you want fields in a large schema to appear progressively in the UI without waiting for the full response.

Production Issues I've Encountered

After enough use, a few constraints surface.

Edge Runtime Limitations

Running on Vercel Edge Functions means no Node.js-specific packages. @ai-sdk/anthropic works on Edge, but importing Node.js packages inside tool functions breaks deployment.

// Explicitly declare at top of route.ts
export const runtime = 'nodejs'; // Use Node.js runtime, not Edge

For most cases, setting runtime = 'nodejs' is the practical choice.

Serverless Timeout

Vercel's free tier serverless function timeout is 10 seconds. Long Claude outputs or complex tool loops can exceed this. Pro tier raises it to 60 seconds.

For longer-running tasks, the architecture needs to change. Building a separate MCP server to offload long-running work is one approach.

Context Accumulation Cost

As conversations grow, the full message history accumulates in context, and token costs climb fast. Check usage from the streamText result:

const result = streamText({ ... });

result.usage.then((usage) => {
  const inputCost = (usage.promptTokens / 1_000_000) * 3.0;
  const outputCost = (usage.completionTokens / 1_000_000) * 15.0;
  console.log(`Tokens: ${usage.promptTokens} in, ${usage.completionTokens} out`);
  console.log(`Cost: $${(inputCost + outputCost).toFixed(5)}`);
});

A real service needs a context management strategy. Simplest approach: only pass the last N turns.

const recentMessages = messages.slice(-20); // last 10 turns each direction

const result = streamText({
  model: anthropic('claude-sonnet-4-6'),
  messages: recentMessages,
});

Rate Limits

Hitting Anthropic API rate limits returns 429 Too Many Requests. Multi-user environments need request queuing or backoff logic. The ai package doesn't include retry logic, so you'll build it or add middleware.

When Should You Use This?

Vercel AI SDK fits well when:

You want to add AI chat to a Next.js app quickly
You want to test Claude, OpenAI, and Gemini on the same codebase
You want useChat to handle frontend state management
You're deploying to Vercel and the timeout limits aren't a bottleneck for your scale

Skip it when:

You need full control over the agent loop internals — use Anthropic SDK directly
You have a Python backend — this SDK is TypeScript only
Your service baseline is long conversations with dozens of turns per user — context management becomes a significant architecture concern

Personally, I reach for it when validating new AI feature ideas. Going from idea to working prototype in under 30 minutes is real. Where it falls short is when you need fine-grained production control — at that point, you're either using Anthropic SDK directly or building abstractions on top of the AI SDK.

Vercel AI SDK made a tradeoff between convenience and flexibility. That tradeoff fits a lot of use cases, just not all of them. "Start with this, migrate when needed" is the realistic mental model.

What I want to test next is the human-in-the-loop tool approval flow added in AI SDK 6. The idea is that an agent can pause before calling certain tools and wait for human approval. Whether this is reliable enough for production is something I haven't fully validated yet. Finding the right balance between fully autonomous agents and manual workflows is one of the core challenges in agent development right now.

GitLab 18.11: Agentic AI for Security, CI, and Analytics

Jangwook Kim — Thu, 23 Apr 2026 04:22:35 +0000

GitLab 18.11 landed on April 16, 2026, and it is the most agentic release the platform has shipped. Three separate AI agents — one for security vulnerability remediation, one for CI pipeline configuration, and one for delivery analytics — moved from concept to either general availability or public beta. If you are running GitLab on any tier and have the Duo Agent Platform enabled, at least one of these agents is available to you today.

This review covers what each agent actually does, who can use it, what the limitations are, and whether the hype holds up.

8.4
/ 10



  <span>Security Automation</span>

  <span>9.0</span>


  <span>CI/CD Improvement</span>

  <span>8.0</span>


  <span>Analytics Depth</span>

  <span>8.5</span>


  <span>Tier Accessibility</span>

  <span>7.5</span>


  <span>Cost Controls</span>

  <span>9.0</span>

What Is GitLab 18.11?

GitLab 18.11 is the eleventh monthly release in the GitLab 18.x series. It builds directly on the GitLab Duo Agent Platform, which reached general availability in GitLab 18.8 (January 2026). The Duo Agent Platform is the runtime layer that hosts GitLab's foundational agents — pre-built, domain-specific AI assistants that can take multi-step actions inside the GitLab platform without requiring the developer to orchestrate each step manually.

Prior to 18.11, the platform had agents for planning and for security analysis (Security Analyst Agent). This release adds three more:

Agentic SAST Vulnerability Resolution — automatically generates merge requests that fix confirmed security vulnerabilities
CI Expert Agent — proposes a complete CI/CD pipeline from a natural-language description of your project
Data Analyst Agent — answers natural-language questions about your delivery metrics with visual charts

All three are backed by the same underlying agent infrastructure, which means they share the same Credits consumption model, the same IDE access points (VS Code, JetBrains, GitLab UI), and the same access controls.

Agentic SAST Vulnerability Resolution

This is the headline feature of 18.11, and it earns that billing. Agentic SAST Vulnerability Resolution is now generally available for GitLab Ultimate customers who have the Duo Agent Platform enabled.

How it works

When a SAST scan completes on the main branch, the agent:

Reviews each detected vulnerability and filters out likely false positives
For confirmed High and Critical severity findings, analyzes the root cause using multi-shot reasoning
Generates a context-aware code fix targeting that specific root cause
Opens a merge request with the proposed fix, a confidence score, and a short explanation
Runs the pipeline automatically to validate the fix resolves the issue without introducing regressions

Developers receive a ready-to-review MR in their inbox. They can inspect the diff, see the confidence score, and merge or close it — without ever switching to a separate security dashboard.

Why this matters

The bottleneck in most DevSecOps workflows is not scanning. Scans run automatically. The bottleneck is the gap between "scan found a vulnerability" and "developer fixed it." Security teams file tickets, developers deprioritize them, and vulnerabilities sit unresolved for weeks. This agent collapses that gap by delivering the fix alongside the finding.

The confidence score is an important detail. GitLab is not silently auto-merging fixes — it surfaces a signal that lets the developer make an informed call. A 92% confidence fix for a SQL injection is a different decision than a 61% confidence fix for a complex deserialization issue.

Caveats

This feature is Ultimate only. Teams on Free or Premium do not get auto-remediation. Additionally, the agent currently targets SAST findings — DAST, dependency scanning, and secret detection are not yet covered by auto-remediation. Incremental SAST scanning (which analyzes only changed files rather than the full codebase) is a separate 18.11 improvement that speeds up scans generally, but that is distinct from the agentic remediation feature.

CI Expert Agent (Beta)

The CI Expert Agent is in public beta in 18.11. It is aimed at a specific problem that has blocked teams for years: the blank .gitlab-ci.yml file.

Writing a CI pipeline from scratch requires knowing GitLab's YAML syntax, understanding which stages your project needs, knowing your test runner commands, and figuring out caching and parallelization. Developers who have not configured CI before either copy from an existing project (with mismatches), stitch together docs examples (fragile), or wait for someone who has done it.

What the CI Expert Agent does

The agent inspects your repository — file structure, detected language and framework, existing scripts — and proposes a complete build-and-test pipeline in natural language. It targets a running pipeline in under five minutes without manual YAML authoring.

Beyond initial setup, the CI Expert Agent can:

Debug failing jobs by reading pipeline logs and explaining what went wrong
Suggest optimizations: caching, needs dependencies to start jobs earlier, parallelization
Help migrate from other CI systems

This is a beta feature, which means it works but is not recommended for production pipelines without human review of the generated YAML. GitLab's docs are explicit: test in a fork or staging branch first.

Practical example

You tell the agent: "This is a Python FastAPI project using pytest for tests and Docker for deployment." It reads your repo, identifies the framework, and proposes:

stages:
  - test
  - build
  - deploy

test:
  stage: test
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - pytest --cov=app tests/
  cache:
    paths:
      - .pip_cache/

build-docker:
  stage: build
  image: docker:24
  services:
    - docker:dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

You review it, adjust any project-specific details, and push. The agent handles the structural thinking; you own the configuration.

Data Analyst Agent

The Data Analyst Agent is generally available in 18.11 and is notable for its tier availability: Free, Premium, and Ultimate customers with the Duo Agent Platform enabled can use it.

What it does

You ask questions in natural language. The agent translates them into GitLab Query Language (GLQL) queries, runs them against your project or group data, and returns visual charts or tables.

Example questions you can ask:

"How many merge requests did the backend team close last month?"
"What is our average MR cycle time this quarter compared to last quarter?"
"Which pipelines are failing most often and on which branches?"
"Show me deployment frequency for the production environment over the last 90 days."

The agent covers the four DORA metrics (deployment frequency, lead time, change failure rate, mean time to restore), merge request analytics, pipeline health, and team contribution summaries.

Why this is useful

Engineering managers previously needed to export data from GitLab, load it into a BI tool, and write queries manually. Teams with dedicated analytics tooling (like Grafana or Tableau) could build dashboards, but smaller teams could not justify the overhead.

The Data Analyst Agent brings that capability to anyone on the platform with a Duo subscription. The answers are not static dashboards — you can ask follow-up questions, drill into specific time ranges, or compare teams.

Limitations

The agent is powered by GLQL, which has coverage gaps. Not every data point across the GitLab platform is queryable through natural language yet. Complex cross-project analytics (for example, correlating security findings across five repos) may require falling back to manual GLQL queries or the API.

GitLab Credits Spending Controls

18.11 also ships a practical addition for teams worried about AI cost creep: subscription-level and per-user spending caps for GitLab Credits.

GitLab Credits are the consumption unit for on-demand AI features on the platform. Prior to 18.11, teams had visibility into usage but limited enforcement controls.

Now:

Billing account managers can set a monthly Credits cap at the subscription level. When the cap is hit, Duo Agent Platform features are suspended until the next billing period or until an admin adjusts the cap.
Per-user caps let admins prevent any single user from consuming the full team allocation. If one user hits their cap, only that user is suspended — other team members are unaffected.
Free namespaces have an automatic on-demand cap of $25,000 per calendar month as a safety floor.

This matters because agentic features (particularly the SAST remediation agent, which can open many MRs at once) consume more Credits than a standard Duo Chat query. Having hard stops prevents a single automated scan from generating an unexpected bill.

What's Good and What's Not

Liked
<ul>
  <li>SAST auto-remediation GA — genuinely closes the scan-to-fix gap without requiring a separate security tool</li>
  <li>Confidence scores on generated fixes prevent blind merges</li>
  <li>Data Analyst Agent available on all tiers — not locked behind Ultimate</li>
  <li>Per-user and subscription-level credit caps give CFOs and engineering managers real spend controls</li>
  <li>Incremental SAST scanning reduces pipeline time for large repos independently of the agentic features</li>
  <li>Kubernetes 1.35 support keeps GitLab current with the k8s release schedule</li>
</ul>


Didn't Like
<ul>
  <li>SAST auto-remediation is Ultimate-only — teams on Premium still have to fix vulnerabilities manually</li>
  <li>CI Expert Agent is beta only — generated YAML needs human review before shipping to production</li>
  <li>Auto-remediation covers SAST only; DAST and dependency scanning are not yet included</li>
  <li>Data Analyst Agent has GLQL coverage gaps for complex cross-project queries</li>
  <li>Duo Agent Platform still requires a paid add-on even on GitLab.com free tier</li>
</ul>

Pricing Breakdown

Understanding which features you can access requires mapping your GitLab tier to the Duo Agent Platform availability.

Feature	Free	Premium ($29/user/mo)	Ultimate ($99/user/mo)
Data Analyst Agent	Yes (Duo required)	Yes (Duo required)	Yes (Duo required)
CI Expert Agent (Beta)	Yes (Duo required)	Yes (Duo required)	Yes (Duo required)
Agentic SAST Remediation	No	No	Yes (Duo required)
Credits Spending Caps	Yes ($25K auto-cap)	Yes (manual cap)	Yes (manual cap)
Best Value Tier	Analytics only	Analytics + CI Agent	Full agentic suite

GitLab Duo add-on pricing sits at approximately $19/user/month on top of your base plan. For Ultimate customers, the Duo add-on is bundled starting in certain enterprise arrangements — check your GitLab contract for specifics.

Who It's For — and Who Should Skip

Go all-in on 18.11 if you are:

A security-conscious team on GitLab Ultimate: the SAST auto-remediation agent alone justifies the upgrade if you have a backlog of unresolved vulnerabilities
A mid-size team without a dedicated DevOps engineer: the CI Expert Agent reduces the YAML expertise required to maintain pipelines
An engineering manager who wants DORA metrics without a separate BI tool: the Data Analyst Agent handles most common delivery questions

The upgrade is less compelling if you are:

On Free or Premium with no plans to upgrade: you get the Data Analyst Agent and CI Expert Agent beta, but the highest-value feature (SAST remediation) is out of reach
Running a very small project with a simple pipeline: the CI Expert Agent adds value primarily for teams maintaining complex multi-stage pipelines
Relying on DAST or dependency scanning for your primary security workflow: auto-remediation does not cover those vectors yet

FAQ

Q: Does Agentic SAST Vulnerability Resolution work on self-managed GitLab?

Yes. The Duo Agent Platform is available on GitLab.com, GitLab Self-Managed, and GitLab Dedicated. For self-managed instances, you need to enable the Duo Agent Platform in your admin settings and ensure your instance has network access to GitLab's AI gateway.

Q: Can the CI Expert Agent write pipelines for non-standard languages or frameworks?

The agent identifies language and framework from your repository structure. For common stacks (Python, Node, Go, Java, Ruby, PHP), it performs well. For less common stacks or custom toolchains, the generated YAML provides a reasonable starting point but will likely need manual adjustments. The docs recommend treating it as a first draft, not a final config.

Q: What counts as a GitLab Credit?

Credits are consumed when you use on-demand AI features — Duo Agent Platform agent actions, certain Duo Chat queries, and AI-powered MR summaries. Standard GitLab features (CI minutes, storage, Packages) use separate consumption units and do not draw from your Credits balance. The Credits dashboard in the Admin area shows a per-feature breakdown.

Q: Is the Data Analyst Agent aware of data across multiple projects?

It can query data from projects and groups you have access to. Cross-group analytics (comparing metrics across separate top-level groups) is limited. For complex multi-group rollups, the GitLab Analytics API remains the more powerful path.

Q: How does the SAST agent handle false positives?

The agent filters out likely false positives before generating a fix. It assigns a confidence score to each fix it does generate. Developers are expected to review the MR and decide whether to merge — the agent does not auto-merge anything without human approval.

Key Takeaways

GitLab 18.11 is a meaningful release, not a marketing refresh. The SAST auto-remediation agent solves a workflow problem that security teams have been complaining about for years. The Data Analyst Agent's broad tier availability means most teams can use it today without an upgrade. The CI Expert Agent is beta — useful, but not ready to hand off without review.

The per-user credit caps are an underrated addition. Agentic features are inherently higher-consumption than chat-style AI, and organizations need budget controls before deploying them at scale. GitLab shipping the controls in the same release as the agents is the right sequencing.

The outstanding gap is coverage: SAST auto-remediation is excellent, but teams that depend on DAST or dependency scanning for their primary security findings will not see the same workflow improvement yet. That feels like an obvious 18.12 candidate.

Bottom Line

GitLab 18.11 is the best DevSecOps release of 2026 so far. If you are on Ultimate, enable the Duo Agent Platform and let the SAST remediation agent start clearing your vulnerability backlog. Everyone else gets the Data Analyst Agent for free — that alone is worth turning on Duo.

Prefer a deep-dive walkthrough? Watch the full video on YouTube.

Kimi Code K2.6: Moonshot AI's Coding Model vs Claude Code

Jangwook Kim — Thu, 23 Apr 2026 00:19:51 +0000

8.7
/ 10



  <span>Benchmark Performance</span>

  <span>9.5</span>


  <span>Agentic Capabilities</span>

  <span>9.0</span>


  <span>Cost Efficiency</span>

  <span>9.5</span>


  <span>Instruction Following</span>

  <span>7.2</span>


  <span>Ecosystem &amp; Tooling</span>

  <span>7.5</span>

Moonshot AI shipped Kimi Code K2.6 as generally available on April 20, 2026 — one week after beta testers ran the Code Preview. The release is significant: K2.6 tops SWE-Bench Pro at 58.6%, outscoring GPT-5.4 (57.7%) and Claude Opus 4.6 (53.4%) on the benchmark that comes closest to measuring real-world GitHub issue resolution. It does this while running fully open weights under a Modified MIT License and charging $0.60 per million input tokens — roughly 5x cheaper than Claude Sonnet 4.6.

That combination — top-tier coding benchmarks, open weights, and aggressive pricing — makes K2.6 the most credible challenger to Claude Code that developers have seen in 2026.

What Kimi Code K2.6 Is

Kimi K2.6 is Moonshot AI's flagship model, built from the ground up for agentic software engineering. Architecturally, it uses the same Mixture-of-Experts design as K2.5: 1 trillion total parameters with only 32 billion activated per forward pass. The full architecture details: 384 experts in total, 8 selected per token (plus one shared expert that is always active), 61 layers, an attention hidden dimension of 7,168, and 64 attention heads.

What K2.6 changes from K2.5 is execution depth. Kimi K2.5 could reliably follow 30–50 sequential tool calls before losing coherence. K2.6 extends that to 200–300 calls. Agent swarm capacity grows from 100 to 300 simultaneous sub-agents, each capable of executing across up to 4,000 coordinated steps. Moonshot AI demonstrated the practical implications with a real test: K2.6 autonomously overhauled an 8-year-old financial matching engine over 13 hours, achieving a 185% throughput improvement without human intervention.

That's not a benchmark. That's a production refactoring job that would normally take a senior engineer a week.

If you've been following the AI coding tools landscape in 2026, Kimi K2.6 lands in the tier just below Claude Mythos but well above the open-weight field. It's Moonshot AI's direct answer to Claude Sonnet 4.6 and the Cursor background agent ecosystem.

Benchmarks: Where K2.6 Actually Leads

Numbers first, context after.

Benchmark	Kimi K2.6	GPT-5.4	Claude Opus 4.6	Gemini 3.1 Pro	Kimi K2.5
SWE-Bench Pro	58.6%	57.7%	53.4%	54.2%	50.7%
SWE-Bench Verified	80.2%	—	—	—	—
LiveCodeBench v6	89.6	—	88.8	—	—
HLE-Full (with tools)	54.0	52.1	53.0	51.4	—
DeepSearchQA (F1)	92.5%	78.6%	—	—	—
Terminal-Bench 2.0	66.7%	—	—	—	—
API Input Price	$0.60/M	varies	$3.00/M	varies	$0.60/M

SWE-Bench Pro is currently the most credible coding evaluation because it tests models on real GitHub issues — bugs filed by actual developers, not synthetic problems. K2.6's 58.6% means it correctly resolves more than half of those issues autonomously, placing it ahead of every closed-weight model in this comparison.

The HLE-Full with tools result (54.0) is perhaps more surprising. Humanity's Last Exam tests genuinely hard multi-domain reasoning, and K2.6 leads there too — which suggests that Moonshot AI's improvements to tool call reliability have broader reasoning implications, not just code execution effects.

One important note: BenchLM currently ranks K2.6 as #6 out of 111 models for coding overall, with an average score of 89.9. It is leading the open-weight category by a significant margin.

What's Good

Strengths
<ul>
  <li>

Top SWE-Bench Pro score — 58.6% on real GitHub issues beats every frontier model, including GPT-5.4 and Claude Opus 4.6

Genuine long-horizon execution — 200–300 reliable tool calls, 13-hour autonomous sessions, 300-agent swarm with 4,000 coordinated steps

Price-to-performance ratio — $0.60/M input vs Claude Sonnet 4.6's $3/M input. For batch coding pipelines, this difference compounds quickly

Open weights, actual commercial use — Modified MIT allows commercial deployment without per-token fees, hardware cost aside

Claude Code drop-in — Three environment variables and you're running K2.6 through the Claude Code interface

Multi-language depth — Tested and reliable across Python, Go, Rust, and front-end (HTML/CSS/JS motion generation)

Weaknesses
<ul>
  <li>

English instruction following lags Claude — Complex multi-part English prompts with nuanced constraints show more drift than Claude Sonnet 4.6

Ecosystem is maturing, not mature — IDE integrations, plugin coverage, and community tooling lag Claude Code and Cursor by 12+ months

Self-hosting requires serious hardware — Full weights need H100-class GPUs; GGUF quantizations help but reduce performance noticeably

Revenue credit requirement — Modified MIT's "display Kimi K2.6 credit" clause for $20M+/month revenue companies creates unexpected branding obligations

Pricing Breakdown

Kimi K2.6 is available through four channels with different economics:

Managed API (platform.kimi.ai)

Input: $0.60 per million tokens
Output: $2.50 per million tokens
Zero infrastructure overhead; recommended for teams under 10M tokens/month

OpenRouter (moonshotai/kimi-k2.6)

Slightly higher margin on OpenRouter's standard passthrough
Useful if you're already routing multiple providers through OpenRouter

Microsoft Azure AI Foundry

Available as a managed deployment in Azure infrastructure
Pricing follows Azure AI model marketplace rates; better for enterprises with existing Azure commitments

Self-Hosted (Hugging Face weights)

Zero per-token cost after hardware
Requires transformers ≥4.57.1
Recommended inference: vLLM or SGLang
Community GGUF quantizations (ubergarm) available for lower VRAM configurations
Practical for teams running >50M tokens/month with H100-class access

For context: at $0.60 input / $2.50 output, K2.6 is 5x cheaper on input and 6x cheaper on output than Claude Sonnet 4.6 ($3/$15). Against Claude Opus 4.6 or 4.7, the gap widens further. For agentic pipelines that generate thousands of tool-call roundtrips, this pricing difference translates directly to project economics.

The Modified MIT License allows unrestricted commercial use with one exception: if your product exceeds 100 million monthly active users or $20 million in monthly revenue, you must display a visible "Kimi K2.6" attribution in your user interface. Most developer teams won't hit that threshold, but SaaS companies building on top of K2.6 should check their TOS terms before deploying.

API Integration: Get K2.6 Running in 10 Minutes

Option 1: Direct Kimi API (OpenAI-compatible)

Kimi's API is OpenAI SDK-compatible. If you're already calling OpenAI endpoints, the switch is a base URL change:

from openai import OpenAI

client = OpenAI(
    api_key="your-moonshot-api-key",
    base_url="https://api.moonshot.ai/v1"
)

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "user", "content": "Refactor this Python class to use dataclasses."}
    ]
)
print(response.choices[0].message.content)

Get your API key at platform.kimi.ai/console/api-keys.

Option 2: Run K2.6 Inside Claude Code

This is the integration that's gained the most traction. Set three environment variables and Claude Code's entire interface — slash commands, subagents, CLAUDE.md — runs against K2.6's backend:

# Linux / macOS
export ANTHROPIC_BASE_URL="https://api.moonshot.ai/anthropic"
export ANTHROPIC_AUTH_TOKEN="your-moonshot-api-key"
export ANTHROPIC_MODEL="kimi-k2.6"
export ANTHROPIC_DEFAULT_OPUS_MODEL="kimi-k2.6"
export ANTHROPIC_DEFAULT_SONNET_MODEL="kimi-k2.6"

# Then launch Claude Code normally
claude

Kimi maintains an Anthropic-compatible API endpoint at api.moonshot.ai/anthropic, which means Claude Code's tool call format, context compaction, and session management work without modification. The practical advantage: you get Claude Code's polished UX at K2.6's pricing.

If you're already using Claude Code for advanced workflows, this is the fastest way to evaluate K2.6 without changing your tooling setup.

Option 3: Kimi Code CLI

Moonshot AI ships its own terminal agent built on K2.6:

pip install kimi-cli
kimi /login   # OAuth via browser
kimi          # Start coding session

The CLI includes repository-aware context, MCP tool integration (kimi mcp add), cron scheduling, and shell mode toggle with Ctrl-X. It supports 256K context tuned for repository-scale codebases and outputs at ~100 tokens/second. For teams comfortable with terminal-first AI coding agents, this is the most direct path.

Self-Hosting K2.6 with vLLM

For teams wanting zero per-token cost:

# Install dependencies
pip install vllm transformers>=4.57.1

# Launch vLLM server with K2.6 weights
python -m vllm.entrypoints.openai.api_server \
  --model moonshotai/Kimi-K2.6 \
  --tensor-parallel-size 4 \
  --max-model-len 65536 \
  --dtype bfloat16

Hardware baseline: 4× H100 80GB for the full model in bfloat16. For lower-budget setups, community GGUF quantizations from ubergarm reduce VRAM requirements significantly, though at reduced accuracy on complex reasoning tasks.

The recommended inference stack is vLLM or SGLang. vLLM's MRV2 architecture (released March 2026) handles MoE routing well; SGLang is faster for structured output generation. If you're already running vLLM in production, K2.6 slots in without configuration changes beyond the model path.

Real-World Performance: What Developers Are Reporting

The 13-hour financial engine refactor is the headline, but production reports are more nuanced.

Where K2.6 genuinely wins:

Long refactoring sessions that cross 50+ file touches — K2.6 maintains context coherence that previous open-weight models couldn't sustain
Python and Go codebases — these appear to be the training-data sweet spots, with clean output and minimal hallucinated APIs
Cost-sensitive batch pipelines — teams running nightly code analysis, automated PR review, or large-scale code generation report meaningful cost reductions at K2.6's pricing versus equivalent Claude Sonnet usage

Where Claude Code still has the edge:

Complex English-language system prompts with layered constraints — Claude Sonnet 4.6's instruction following is measurably tighter on prompts with 5+ simultaneous requirements
Sensitive code contexts (security, compliance) — Anthropic's Constitutional AI training shows in how Claude handles edge cases; K2.6 is more willing to generate code that might have subtle issues
IDE integrations — the JetBrains, VS Code, and Cursor ecosystems are built around Anthropic's API; K2.6 works as a drop-in but surface-level polish differences are noticeable

The hybrid workflow gaining traction: K2.6 for code generation and bulk execution, Claude Opus 4.7 for planning, validation, and anything requiring precise instruction adherence. Running K2.6 via the OpenAI-compatible endpoint alongside tools like LiteLLM's proxy makes provider switching transparent to application code.

Who It's For

K2.6 is the right choice if you're:

Running cost-sensitive agentic pipelines at scale (>10M tokens/month where pricing compounds)
Building on open-weight infrastructure where you need weights you actually control
Doing large-scale refactoring, automated PR review, or repository-level code analysis
Evaluating a Claude Code alternative without locking into Anthropic's pricing
Already familiar with MoE model deployment and have H100-class access for self-hosting

Stick with Claude Code if you're:

Writing complex English-language system prompts with nuanced multi-part constraints
Building in an IDE-first workflow where JetBrains or VS Code integrations matter
Prioritizing safety and compliance behavior over raw benchmark performance
A solo developer where the tooling ecosystem difference matters more than per-token costs
Working in domains (legal, medical, security) where Anthropic's safety tuning is a practical requirement

Compare K2.6 alongside other capable open-weight agents like Goose by Block and Hermes Agent if your priority is moving away from proprietary model dependencies entirely.

FAQ

Q: Is Kimi K2.6 actually open source?

The weights are publicly available on Hugging Face under a Modified MIT License. "Modified" because of the revenue/MAU attribution requirement — but for the vast majority of developers and teams, it's functionally open source with commercial use allowed.

Q: Can I use Kimi K2.6 with existing Claude Code projects?

Yes. Set ANTHROPIC_BASE_URL=https://api.moonshot.ai/anthropic and ANTHROPIC_AUTH_TOKEN=<your-kimi-key> and ANTHROPIC_MODEL=kimi-k2.6. Claude Code's UI, slash commands, and CLAUDE.md handling all work against K2.6's backend via Kimi's Anthropic-compatible endpoint.

Q: How does the agent swarm work in practice?

The 300 sub-agent, 4,000 coordinated step architecture is accessible via Kimi Code CLI and the managed API. You define an orchestration prompt describing the overall task; K2.6's planning layer spawns sub-agents for parallelizable work (e.g., different modules or files) and coordinates their outputs. Direct programmatic control over individual sub-agent allocation is not yet exposed in the API — it's handled internally by the model.

Q: What's the context window?

The Kimi Code CLI is tuned for 256K tokens on repository-scale codebases. Via the managed API, current documentation shows 128K. Self-hosted configurations depend on your --max-model-len setting and available VRAM.

Q: How does K2.6 compare to DeepSeek V3.2?

Both are competitive open-weight coding models at aggressive price points. DeepSeek V3.2 has the unique capability of simultaneous thinking + tool use in one API call. K2.6 leads on SWE-Bench Pro and on agent swarm scale. For pure coding throughput and agentic workflows, K2.6 currently has the benchmark edge.

Key Takeaways

Kimi K2.6 posts 58.6% on SWE-Bench Pro, the highest score among publicly listed frontier models as of April 2026
The core improvement over K2.5 is execution reliability: 200–300 sequential tool calls without drift, versus 30–50 previously
API pricing at $0.60/M input is 5x cheaper than Claude Sonnet 4.6 — significant for agentic pipelines at scale
Claude Code integration requires three environment variables; K2.6 runs transparently through Claude's interface
Open weights on Hugging Face under Modified MIT; self-hosting requires H100-class hardware for the full model
Instruction following for complex English prompts remains a gap versus Claude; hybrid workflows mitigate this

Bottom Line

Kimi Code K2.6 is the most capable open-weight coding model available in April 2026, and its pricing makes it a serious Claude alternative for cost-sensitive agentic pipelines. The benchmark lead is real and the Claude Code drop-in integration removes most switching friction. The honest caveat: complex instruction following and ecosystem maturity still favor Anthropic — but for teams primarily doing code generation at scale, K2.6 earns its place in the stack.

Prefer a deep-dive walkthrough? Watch the full video on YouTube.