DEV Community: Shrijal Acharya

7 Best Code Execution APIs for AI Agents in 2026

Shrijal Acharya — Sun, 01 Nov 2026 13:29:37 +0000

Your AI agent just wrote a perfectly reasonable Python script. Nice.

The next question is where you actually run it.

You could technically execute it on your own server, but letting LLM-generated code run next to your application, secrets, and production data can become a pretty damn problem. Believe me when I say it. 🤧

Ref: docker.com

And execution itself is only half the story.

Your agent still needs somewhere to send that code, get stdout, stderr, files, and errors back, maybe install dependencies, and sometimes come back to the same environment later.

That is where code execution APIs come in.

I went through the major options and picked seven I would actually look at for AI agents today.

⚠️ Quick note: This isn't a scientific ranking. These APIs solve different problems, so the best option depends a lot on what your agent is trying to do.

TL;DR: Best code execution APIs for AI agents

#	Provider	I would use it for
1	Tensorlake	Persistent coding agents, RL rollouts, evals, and agent CI
2	E2B	Stateful Python and JavaScript code interpreter workflows
3	Daytona	Shell execution, code interpreters, and long-running development sessions
4	Modal Sandboxes	GPU-backed execution, ML workloads, and parallel compute
5	Vercel Sandbox	TypeScript-first coding agents, builds, and preview environments
6	AWS Bedrock AgentCore Code Interpreter	Managed code execution for agents already running on AWS
7	Claude Code Execution	Model-native execution without managing your own infrastructure

You probably don't need all seven.

The important part is understanding what kind of execution your agent actually needs.

What is a code execution API?

A code execution API gives your application or AI agent a programmatic way to run code or shell commands somewhere else and get the result back.

The basic flow looks like this:

For a simple agent, that might mean sending a small Python script and getting the output back.

For a coding agent, things get a little more involved.

The agent might clone a repository, install dependencies, edit files, run tests, inspect logs, fix the code, start a server, and then continue working inside the same environment.

That is also where the difference between a code execution API and an AI sandbox becomes clear.

💁 The API is how your agent asks for code to be executed.

💁 The sandbox is the isolated environment where that code actually runs.

Here are the ones I would look at.

1. Tensorlake

ℹ️ Best when your agent needs to keep working inside the same environment over time.

Most code execution APIs are built around a fairly simple flow: create an environment, run some code, get the result back, and shut the environment down when you're finished.

That works perfectly fine for a lot of agents.

It starts getting weird when your agent installs dependencies, changes files, starts processes, goes idle for a while, and then needs to come back and continue from the same environment.

That's where Tensorlake gets interesting.

You can create isolated Linux sandboxes and run commands through its Python SDK, TypeScript SDK, CLI, or REST API.

A basic Python execution looks like this:

from tensorlake.sandbox import Sandbox

sandbox = Sandbox.create(
    name="my-agent",
    cpus=1.0,
    memory_mb=1024
)

result = sandbox.run(
    "python",
    ["-c", "print('Hello from Tensorlake')"]
)

print(result.stdout)

The command execution itself is pretty straightforward.

The bigger difference is what Tensorlake lets you do with the environment around it.

Named sandboxes can suspend while preserving the filesystem, memory, and running processes. So if your agent comes back later, it can resume the same environment instead of rebuilding everything from scratch.

Tensorlake also supports background processes and PTY sessions, which is useful when the agent needs to interact with something longer-running than a single command.

Then there are snapshots.

You can take a snapshot of a running environment and create multiple copies from the same state.

Something like this:

Install dependencies
        ↓
Build project
        ↓
Reach known-good state
        ↓
Snapshot
   ↙    ↓    ↘
Agent  Agent  Agent
run 1  run 2  run 3

This becomes pretty handy for RL rollouts, evals, CI shards, or best-of-N agent runs.

Instead of preparing the same environment ten times, you can prepare it once and branch from there.

Tensorlake also combines sandboxes with its own hosted Git and versioned filesystem.

So the agent's work doesn't necessarily disappear when the sandbox does. Files can live in durable versioned storage, workspaces can be restored, and coding agents can keep their Git state around across runs.

What it does

Executes commands through run()
Supports background processes and interactive PTY sessions
Runs CPU sandboxes inside isolated microVMs
Supports ephemeral and persistent named sandboxes
Preserves memory, files, and running processes across suspend and resume
Supports filesystem and memory snapshots
Lets you copy a running or suspended sandbox into multiple environments
Provides hosted Git for coding-agent workflows
Provides a versioned POSIX filesystem with snapshots and restore
Supports network allow and deny rules
Has Python, TypeScript, CLI, and REST interfaces

💡 When I would pick it

I would use Tensorlake when execution is only one step inside a much longer agent workflow.

Coding agents are probably the clearest example.

If an agent installs dependencies, modifies a repository, runs tests, goes idle, comes back later, tries several approaches in parallel, and eventually needs to preserve its work, rebuilding the whole environment for every call starts becoming annoying pretty quickly.

RL rollouts and evals are another good fit because you can prepare one known state and fan it out into multiple independent sandboxes.

If all you want is to run a Python snippet and get a chart or DataFrame back, E2B will probably feel more natural.

The free tier is also fairly limited at one concurrent sandbox.

But if your agent needs the whole environment to survive, not just the result of one execution call, Tensorlake is the one I would look at first.

Resources

Tensorlake
Tensorlake Sandboxes
Sandbox lifecycle
Snapshots

2. E2B

ℹ️ Probably the first one I would try for a traditional AI code interpreter.

E2B is one of the names that comes up almost immediately when you start looking for code execution infrastructure for AI agents.

And there is a good reason for that.

Its Code Interpreter API is very close to the workflow most people imagine when they say, "I want my agent to write some Python and run it."

A basic example looks like this:

from e2b_code_interpreter import Sandbox

sbx = Sandbox()

execution = sbx.run_code("""
x = 10
print(x * 2)
""")

print(execution.logs)

The nice part is that interpreter state can persist between executions.

So your agent can define a variable in one call, use it again in the next one, and keep working without starting a completely fresh Python environment every time.

E2B also gives you richer results than most command-oriented APIs.

That includes logs, errors, charts, and DataFrame-style outputs, which makes it particularly nice for data-analysis agents.

You can also stream stdout, stderr, and execution results while the code is running.

Underneath all of this, E2B runs the sandbox inside Firecracker microVMs.

Sandboxes can also be paused and resumed, with options for preserving memory alongside the filesystem.

What it does

Runs Python and JavaScript generated by agents
Provides persistent interpreter contexts
Returns logs, errors, charts, and other rich results
Streams stdout, stderr, and execution output
Supports regular shell commands
Uses Firecracker microVMs for isolation
Supports pause and resume
Can preserve filesystem and memory state
Has Python and JavaScript/TypeScript SDKs
Supports parallel isolated code contexts

💡 When I would pick it

If I were building a data-analysis agent, notebook agent, or anything where the loop is mostly:

LLM writes code
      ↓
execute it
      ↓
inspect result
      ↓
write more code

E2B would probably be one of the first APIs I would try.

The interpreter experience is much more natural for that than manually managing shell processes.

Resources

E2B Code Interpreter
E2B persistence
E2B pricing

3. Daytona

ℹ️ A good option when you want several execution styles under one SDK.

Daytona gives you a few different ways to run agent-generated code.

You can execute regular shell commands, use its higher-level code execution APIs, create persistent sessions, or work with interactive processes.

For a normal command, the flow looks like this:

result = sandbox.process.exec(
    "ls -la",
    timeout=5
)

print(result.result)

If you want something closer to a code interpreter, Daytona also provides code_run, which can execute code and return artifacts such as stdout.

For longer-running agent workflows, sessions are probably the more interesting part.

A session lets the agent keep shell state between multiple commands rather than treating every execution as a completely separate process.

You can run commands, inspect their status, and stream logs while they are still running.

Daytona also has fairly broad SDK coverage.

There are SDKs for Python, TypeScript, Ruby, Go, and Java, which is useful if your agent infrastructure isn't built around one language.

What it does

Executes shell commands programmatically
Provides higher-level code execution APIs
Supports long-running command sessions
Streams command logs
Supports PTY workflows
Provides interpreter-style execution contexts
Has SDKs for Python, TypeScript, Ruby, Go, and Java
Supports Linux and Windows environments
Exposes execution errors and exit-code information

💡 When I would pick it

I would use Daytona when I wanted several execution styles in the same platform.

Maybe one agent only needs quick shell commands, another needs a persistent terminal session, and another works better with interpreter-style execution.

Daytona can cover all of those without forcing everything into the same model.

Resources

Daytona process and code execution
Daytona Python process SDK
Daytona V2 process execution

4. Modal Sandboxes

ℹ️ The one I would look at when agent code execution starts becoming a compute problem.

Modal already has a strong presence around serverless compute, GPUs, and ML workloads.

Its Sandbox API gives you isolated command execution inside that same infrastructure.

The basic API is pretty clean:

import modal

sb = modal.Sandbox.create()

process = sb.exec(
    "python",
    "-c",
    "print('hello')"
)

print(process.stdout.read())

You can wait for the process, inspect its return code, or stream output while the command is still running.

This feels a little different from something like E2B.

You are working with processes inside a sandbox rather than a persistent Python notebook-style interpreter.

But that becomes useful when the code your agent is running is actually expensive.

What it does

Creates isolated sandboxes programmatically
Executes processes with Sandbox.exec
Streams process output
Exposes process return codes
Supports configurable CPU and memory
Supports GPU-backed workloads
Supports custom runtime environments
Integrates with Modal's broader compute platform
Supports filesystem snapshots for longer-running workflows

💡 When I would pick it

Modal is what I would look at when the code itself is expensive.

If an agent is launching ML jobs, running GPU-backed code, evaluating models, or creating lots of compute-heavy executions in parallel, Modal fits pretty naturally.

For a tiny Python analysis script, it might be more infrastructure than you really need.

There is also a 24-hour sandbox lifetime limit, so that's something to consider.

But for GPU-heavy execution, ML agents, and workloads where compute matters more than workspace persistence, Modal is one of the stronger options here.

Resources

Modal Sandboxes
Modal Sandbox API

5. Vercel Sandbox

ℹ️ A very natural fit for coding agents that build web applications.

Vercel Sandbox is pretty much what you would expect from Vercel.

It focuses heavily on application-development workflows.

Your agent can create an isolated environment, run commands, install dependencies, build an application, start a server, and expose the result through a preview.

The JavaScript SDK gives you runCommand:

const command = await sandbox.runCommand("npm", ["run", "build"]);

console.log(await command.stdout());

The result includes the exit code, stdout, stderr, and combined output.

For longer-running processes, you can start commands in detached mode and stream logs using an async iterator.

Vercel also supports named sandboxes through getOrCreate, so an agent can return to an existing environment instead of provisioning a new one every time.

Sandboxes are persistent by default and automatically snapshot their filesystem when stopped.

What it does

Executes commands through a TypeScript SDK
Returns exit codes, stdout, and stderr
Supports detached long-running commands
Streams logs from running processes
Supports named persistent sandboxes
Automatically snapshots filesystem state
Uses Firecracker microVMs
Provides a Python SDK alongside JavaScript
Fits naturally into Vercel preview and deployment workflows

💡 When I would pick it

I would use Vercel Sandbox for an app-building agent.

Think of an agent that receives a prompt, changes a Next.js project, installs something, runs the build, starts a development server, and then gives the user a preview.

That workflow maps really well to Vercel.

It also makes sense if your stack is already heavily TypeScript-first.

For agents that build web applications and need to show the result quickly, Vercel Sandbox is a pretty obvious one to check.

Resources

Vercel Sandbox
Vercel Sandbox SDK
Working with Sandbox

6. AWS Bedrock AgentCore Code Interpreter

ℹ️ Best when the rest of your agent stack already lives inside AWS.

AWS also has a managed code interpreter built specifically for agent workloads through Bedrock AgentCore.

Instead of setting up your own sandbox infrastructure, you start an interpreter session and send code to it through the AgentCore API.

It supports operations such as:

executeCode
executeCommand
readFiles
writeFiles
listFiles
removeFiles

For example, you can execute Python using invoke_code_interpreter:

response = client.invoke_code_interpreter(
    codeInterpreterIdentifier="aws.codeinterpreter.v1",
    sessionId=session_id,
    name="executeCode",
    arguments={
        "language": "python",
        "code": "print(2 + 2)"
    }
)

The result comes back as an event stream.

The interpreter supports Python, JavaScript, and TypeScript, and state can remain available during the session unless you explicitly clear the context.

You also get the things you would normally expect from AWS around IAM, networking, and integration with the rest of the Bedrock stack.

What it does

Executes Python, JavaScript, and TypeScript
Runs shell commands
Reads and writes files
Supports stateful interpreter sessions
Returns results through an event stream
Supports asynchronous command execution
Integrates directly with Bedrock AgentCore
Works with AWS security and networking controls
Has Boto3 and higher-level AgentCore SDK support

💡 When I would pick it

I would use AgentCore Code Interpreter when the rest of the agent already runs on AWS.

If you're already using Bedrock, IAM, VPCs, CloudWatch, and the rest of that ecosystem, adding another execution provider might just create more moving pieces.

AgentCore keeps the execution layer inside AWS.

It becomes less interesting if you want portable agent environments, hosted Git, snapshot forks, or infrastructure that isn't tied to AWS.

But for enterprise teams already building around Bedrock, it is probably one of the more straightforward choices.

Resources

AWS AgentCore Code Interpreter
Boto3 invoke_code_interpreter

7. Claude Code Execution

ℹ️ The easiest option when Claude just needs to run some code and you don't want another infrastructure layer.

The last one is slightly different from everything above.

With Claude's code execution tool, you aren't directly creating or controlling the execution environment.

You give Claude access to the server-side tool through the Messages API:

{
  "type": "code_execution_20250825",
  "name": "code_execution"
}

Claude decides when code execution is useful, runs the code inside Anthropic's infrastructure, and returns the result as part of the model response.

You don't need to manage a sandbox SDK, provision a VM, or deal with the underlying environment yourself.

Newer versions also support persistent REPL state, so Claude can keep interpreter context across executions inside the same container.

Generated files can then be pulled through Anthropic's Files API.

The main limitation is pretty much the same thing that makes it easy.

You don't control much of the environment.

There is no custom container image, the environment has no internet access, and the whole thing is tied directly to Claude's API workflow.

What it does

Lets Claude execute code server-side
Requires no separate sandbox infrastructure
Supports Bash and file operations
Returns execution results directly through Claude
Supports persistent REPL state in newer versions
Lets applications reuse a container
Supports generated-file retrieval
Runs without internet access

💡 When I would pick it

If Claude needs to analyze a dataset, run some calculations, manipulate files, or execute generated Python as part of a normal application flow, I would probably start here.

You avoid almost the entire infrastructure problem.

But I wouldn't use it as the execution layer for a full coding-agent platform.

You can't bring your own image, control the environment like a normal sandbox, or attach the same kind of persistent workspace infrastructure you get with Tensorlake, E2B, Daytona, or Vercel.

For lightweight execution inside Claude applications though, having almost nothing to manage is a pretty nice advantage.

Resources

Claude code execution tool

So, which code execution API should you use?

There isn't one answer here because these products are solving slightly different parts of the same problem.

If your agent needs a persistent workspace it can resume later, fork into parallel runs, and keep files and Git state around, Tensorlake is the one I would look at first.

That makes it especially useful for coding agents, RL rollouts, evals, and CI-style workflows where execution is only one part of a much longer process.

For more interpreter-style execution, E2B is a strong option. Daytona works well if you want shell execution, sessions, and interpreter APIs together, while Modal makes more sense for GPU-heavy workloads.

Vercel Sandbox fits web-building agents nicely, AgentCore is a natural choice if you're already deep in AWS, and Claude's built-in execution tool is handy when you just need lightweight code execution inside Claude.

The main thing is to pick based on what happens after the code runs, not just whether the API can execute it.

That's all for this one. I'll see you in the next one. ✌️

How to Moderate Live Chat in Real Time with Jev and Composio (Discord + Twitch)

Shrijal Acharya — Wed, 30 Sep 2026 13:05:00 +0000

Anyone who has moderated a live chat knows how fast things go wrong.

Someone says something bad and by the time a mod sees it, everyone in the chat already has. 🫩

Most AI moderation setups use an LLM for this. That works, but an LLM is usually doing way more than we need here and is way slower.

For moderation, I mostly want one thing: should I remove this message or not? So I tried Jev, a decision model from TypeSafe. I guess you're already familiar with the model, as its been over a week the model dropped.

It does not generate text. You give it a typed question, and it gives you a probability back.

Thats pretty much perfect for a moderation bot. 👀

I called Jev through Composio's Jev toolkit, so I didn't have to set up a separate HTTP client or keep the Jev API key inside the code.

Then I connected the same moderation logic to Discord and Twitch.

If you want to play around with the project, or build on top of it, you can find the source code here:

Source Code: shricodev/jev-youtube-discord-moderator

Let's get into it. ✌️

What's Covered

Here's what we'll go through:

What Jev is and how it is different from a normal LLM (in short)
Setting up Jev through Composio
Building a Discord bot that removes such messages
Building the same moderation flow for Twitch with EventSub
A quick YouTube version (I leave that upto you)😉

⚠️ NOTE: This is still an AI bot, so you might want to keep human mods around. But considering how cheap it is, at $0.042 per million input tokens with output basically free, it could be worth trying if you're a streamer or run a large live chat.

What's Jev, Anyway?

The easiest way I think about Jev is this:

An LLM generates an answer. Jev makes a decision.

You send Jev a state, which for us is the chat message, along with one or more questions. It can answer all of those questions in the same call.

Jev currently gives three question types:

Noul - Ask whether something is true and get a probability between 0 and 1.
Choice - Give it a few options and let it pick one, along with probabilities.
Score - Ask it to place something somewhere on an ordered scale of 2 to 10 options.

For moderation, Noul is the important one.

You can ask:

Is this a hostile comment at a person?

And instead of getting a paragraph back, you get something like:

0.92

Now I can simply decide what to do with that number.

If p ≥ 0.8, delete it.

If it sits between 0.5 and 0.8, send it for review. Everything else stays.

No parsing generated text. No asking the model to "only answer YES or NO." 🤌

It's also super cheap ($42 per billion input token, BILLION!!, output is free).

To give you an approximate, moderating 1,000 messages (roughly counting ~400 input token per msg) comes out to less than 2 cents, almost free.

What We're Building

We are building two bots, but both use the same moderation logic.

The flow looks like this:

new chat message
   │
   ▼
Jev via Composio (JEV_EVALUATE_STATE)
   ├── negative: noul   → "is this a hostile comment at a person?"
   └── category: choice → insult / harassment / spam / fine
   │
   ▼
p ≥ 0.8  → delete the message
p ≥ 0.5  → log it for review
else     → keep it

The nice part is that the Jev code stays exactly the same on both platforms.

Only the part that receives and deletes messages changes.

Discord: discord.js listens for messages and deletes them.
Twitch: EventSub sends us messages over WebSocket, and the Helix API deletes them.

💁 I've never used Twitch, so I'm not sure how good this works in a long term compared to Discord.

We also need a way to properly test the bots.

So I added a small flooder script that sends a bunch of normal/racist messages into the test chat.

We'll use that later.

💡 Why not use Composio for Discord too? Composio's Discord toolkit does not currently have a real-time new-message trigger. Polling would add extra delay, so I use the native SDK for incoming chat and Composio for Jev.

Tech Stack

The project is pretty small.

Jev (TypeSafe) - Decides whether the message is hostile and gives us a probability.
Composio - Handles the Jev connection and gives us one tools.execute() call.
discord.js - Listens for and deletes Discord messages.
Twitch EventSub + Helix - Receives Twitch chat messages and deletes them when needed.
Node.js 22 - Everything runs as plain .mjs files, so there is no framework or build step (perfect to test right away).

Node 22 already includes fetch and WebSocket, which also keeps the Twitch side pretty clean.

How to Set Up the Project

Getting the project running is pretty simple.

Clone the repo and install the dependencies:

git clone https://github.com/shricodev/jev-youtube-discord-moderator
cd jev-youtube-discord-moderator/jev-discord-demo
npm install

Configure Composio

First, you need a Composio API key.

Create an account on Composio, get your API key, and add it to the .env file in the root of the project.

Then add the Jev toolkit from the dashboard and connect it using your TypeSafe API key.

Also keep the user ID for that connection. We'll need it because every Composio tool call is scoped to a user.

Discord and Twitch Setup

This part can easily become its own guide, so I'll keep it short.

ℹ️ You can find proper information in the Discord and Twitch own documentation.

For Discord:

Create an application and bot in the Discord Developer Portal.
Copy the bot token.
Turn on Message Content Intent under the Bot tab.
Invite the bot with View Channel, Read Message History, and Manage Messages permissions (permissions=74752).
Create a webhook inside your test channel for the flooder.

Without Message Content Intent, Discord will send messages to the bot without their content.

For Twitch:

Register an app at dev.twitch.tv/console.
Set the client type to Public.
Copy the Client ID.
Turn off AutoMod, blocked terms, and the non-mod chat delay while testing.

Otherwise Twitch may remove the test messages before our bot even gets them.

Now add everything to .env:

// 👇 .env

COMPOSIO_API_KEY=
COMPOSIO_USER_ID=

DISCORD_TOKEN=
WATCH_CHANNEL_ID=
FLOOD_WEBHOOK_URL=

TWITCH_CLIENT_ID=

COMPOSIO_API_KEY comes from the Composio dashboard.

COMPOSIO_USER_ID is the user that owns your Jev connection.

Then we have the Discord bot token, channel ID, webhook URL, and Twitch Client ID. That's all we need for now. ✌️

The Jev Call

This is basically the main part of the project.

And the nice thing is that both bots use the same code.

Create jev-mod.mjs:

// 👇 jev-mod.mjs
import { Composio } from "@composio/core";

const composio = new Composio({ apiKey: process.env.COMPOSIO_API_KEY });

// Both questions run in parallel inside one Jev call.
const QUESTIONS = {
  negative: {
    type: "noul",
    instructions:
      "The message is a hostile comment aimed at a person: an insult, harassment, a personal attack, " +
      "or telling someone to leave. Casual swearing, self-deprecating jokes, complaining about a game " +
      'or product, and hype like "that was sick" do NOT count.',
  },
  category: {
    type: "choice",
    criteria: {
      insult: "Insults or belittles someone",
      harassment: "Targets, dismisses or tells someone to go away",
      spam: "Repetitive, promotional or meaningless flooding",
      fine: "Normal chat, including jokes, hype and mild frustration",
    },
  },
};

async function judge(text) {
  const t0 = performance.now();
  const res = await composio.tools.execute("JEV_EVALUATE_STATE", {
    userId: process.env.COMPOSIO_USER_ID,
    arguments: { state: text, questions: QUESTIONS },
    dangerouslySkipVersionCheck: true,
  });
  const ms = performance.now() - t0;
  if (!res.successful)
    throw new Error(res.error || "Bruhh! JEV_EVALUATE_STATE failed");

  const a = res.data.answers;
  return { ms, p: a.negative.noul, category: a.category.choice };
}

There are two things worth looking at here. First is the negative question.

Chat messages can be weird. Someone might say:

this thing is absolutely killing me

It has negative words in it, but isn't an attack on another person.

So I don't just tell Jev what counts as hostile. I also tell it what doesn't count.

Then we have category.

This one isn't actually used to decide whether a message gets deleted. I mainly added it because seeing insult, harassment, spam, or fine in the logs makes debugging much easier.

And since both questions run in the same Jev call, we get it without adding another request.

That's basically the entire AI part. 🔥

⚠️ NOTE: Direct tool execution in Composio needs a toolkit version. You can either pin one with version: "<toolkit-version>" or use dangerouslySkipVersionCheck: true to run the latest version. If you use neither, the SDK throws a ToolVersionRequiredError.

The Discord Bot

Now we can actually use it.

The Discord bot is a normal discord.js client.

💁 You'll find ton of guides on the internet that show you how to use discord.js. I also wrote one for freeCodeCamp, so if you're not sure how to get started, you might want to check it out: How to Build a LangGraph and Composio-Powered Discord Bot

It listens for a message, sends the text to judge(), and then decides what to do.

// 👇 jev-mod.mjs
import { Client, GatewayIntentBits, Events } from "discord.js";

const DELETE_AT = 0.8; // noul
const REVIEW_AT = 0.5;

const client = new Client({
  intents: [
    GatewayIntentBits.Guilds,
    GatewayIntentBits.GuildMessages,
    GatewayIntentBits.MessageContent,
  ],
});

client.on(Events.MessageCreate, async (msg) => {
  if (msg.author.id === client.user.id) return;
  if (msg.channelId !== process.env.WATCH_CHANNEL_ID) return;
  if (!msg.content) return;

  const v = await judge(msg.content);

  if (v.p >= DELETE_AT) {
    await msg.delete();
    console.log(
      `🗑️ DELETE  jev ${Math.round(v.ms)}ms  p=${v.p.toFixed(2)}  "${msg.content}"`,
    );
  } else if (v.p >= REVIEW_AT) {
    console.log(`🟡 REVIEW  p=${v.p.toFixed(2)}  "${msg.content}"`);
  }
});

// rest of the code…

await client.login(process.env.DISCORD_TOKEN);

If you've built a Discord bot before, most of this should look familiar.

The interesting part is the two thresholds. Why not just have one?

Because automatically deleting someone's message is a pretty strong action.

If Jev gives us a high probability, we remove it.

If the probability is somewhere in the middle, we simply log it and let a human take a look. 🧐

That gives us a nice middle ground instead of forcing every message into delete or keep.

The full repo adds two more things. There is a small limit on how many Jev requests can run at once, so a sudden flood does not fire hundreds of calls together.

There is also a delete queue for every channel.

When messages start piling up, it uses bulkDelete() instead of deleting every message separately.

Discord rate limits individual deletes pretty quickly, so this helps a lot during the flood test. ✅

The Twitch Bot

Twitch needs a little more work.

There isn't really a discord.js equivalent that we are using here, so I talk to Twitch directly.

New chat messages come through EventSub over WebSocket.

That means Twitch sends the message to us when it happens instead of making us keep checking for new messages.

When the WebSocket connects, Twitch sends a welcome message containing a session ID.

We then have 10 seconds to subscribe to the chat events using that ID.

// 👇 twitch-mod.mjs

const ws = new WebSocket("wss://eventsub.wss.twitch.tv/ws");

ws.onmessage = async ({ data }) => {
  const msg = JSON.parse(data);
  const type = msg.metadata.message_type;

  if (type === "session_welcome") {
    // Must subscribe within 10 s of the welcome.
    await helix("bot", "POST", "/eventsub/subscriptions", {
      type: "channel.chat.message",
      version: "1",
      condition: { broadcaster_user_id: broadcasterId, user_id: me.user_id },
      transport: { method: "websocket", session_id: msg.payload.session.id },
    });
  } else if (type === "notification") {
    onChatMessage(msg.payload.event);
  }
};

async function onChatMessage(ev) {
  const v = await judge(ev.message.text); // 👈 the exact same Jev call

  if (v.p >= DELETE_AT) {
    await helix(
      "bot",
      "DELETE",
      `/moderation/chat?broadcaster_id=${broadcasterId}&moderator_id=${me.user_id}&message_id=${ev.message_id}`,
    );
  }

  // rest of the code…
}

And notice this line:

const v = await judge(ev.message.text);

That's the exact same function we used for Discord. Nothing changes on the Jev side.

helix() is just a small fetch wrapper. It adds the token and Client ID headers and refreshes the token when needed.

For Twitch login, I use the Device Code flow.

The script prints a link and code, you approve it in your browser, and you're done.

So we don't need to set up a redirect server just for authentication.

⚠️ NOTE: Twitch does not let you delete messages from the broadcaster or other moderators. If you test this by sending bad messages from your own broadcaster account, the bot won't be able to remove them. Use a second regular account for testing.

The repo also handles session_reconnect and checks whether the EventSub connection is still alive.

Connections can drop, and having the mod bot silently stop receiving messages in the middle of a stream would be... not great. 🫠

Flooding the Chat

Now we need to actually test the thing.

Sending one bad message manually isn't very useful.

So the repo includes a flooder that sends a random mix of hostile and normal messages.

Run the bot in one terminal:

npm run bot

Then run the flooder in another:

npm run flood -- 60

The 60 means it will send 60 test messages.

For Twitch, use:

npm run twitch:bot
npm run twitch:flood -- 60

Bot in Action

Here's a quick demo of the Discord bot:

And for the Twitch one, simply run the command I gave you earlier. That should do it.

While the bot is running, you'll see logs that look something like this:

✅ keep    jev  389ms  p=0.06 fine ...
🟡 REVIEW  jev  909ms  p=0.73 insult ...
✅ keep    jev  372ms  p=0.03 fine ...

🗑️  DELETE  jev  341ms  delete  641ms  total  984ms  p=0.92 insult ...
🗑️  DELETE  jev  399ms  delete  418ms  total  979ms  p=0.93 insult ...

Here's what those numbers mean:

jev is the time Jev took to make the decision.
delete is how long Discord took to remove the message.
total is the full time from receiving the message to deleting it.

Once the connection set up properly, Jev was usually around 340 to 460 ms in my test.

Funny enough, Discord deleting the message was sometimes slower than Jev deciding whether it should be deleted. 😭

From my first 10-message Discord run:

Metric	Result
Jev latency (p50)	399 ms
Receive → deleted (p50)	979 ms
Bad messages caught	4 of 5
Normal messages deleted	0 of 5

One message did not cross the delete threshold:

If you want a stricter bot, you can simply lower DELETE_AT. That's also why I like using probabilities here.

What About YouTube Live?

I left YouTube Live as the one for you to finish. 🙂

The same idea works there, but testing it takes a little more setup.

You can poll liveChatMessages.list, and YouTube tells you how long to wait before making the next request.

There is also a newer streamList method that streams messages, which is probably what I would try for this project.

But the important bit is that the Jev code does not change.

Here's a rough version using googleapis:

// 👇 youtube-mod.mjs
import { google } from "googleapis";

// OAuth client for the channel owner or a moderator, with the youtube.force-ssl scope
const youtube = google.youtube({ version: "v3", auth: oauthClient });

const { data: b } = await youtube.liveBroadcasts.list({
  part: ["snippet"],
  broadcastStatus: "active",
});
const liveChatId = b.items[0].snippet.liveChatId;

let pageToken;
let first = true;
while (true) {
  const { data } = await youtube.liveChatMessages.list({
    liveChatId,
    part: ["snippet", "authorDetails"],
    pageToken,
  });

  // The first page is chat history, so skip it.
  if (!first) {
    for (const m of data.items) {
      const v = await judge(m.snippet.displayMessage); // same Jev call again
      if (v.p >= 0.8) await youtube.liveChatMessages.delete({ id: m.id });
    }
  }

  first = false;
  pageToken = data.nextPageToken;
  await new Promise((r) => setTimeout(r, data.pollingIntervalMillis));
}

Replace the polling part with streamList, add the same review threshold and request limit we used for Discord, and you basically have the third bot.

Try it and let me know what kind of latency you get. ✌️

Conclusion

And that's basically it. 🙌

We built the same real-time moderation flow for Discord and Twitch using one shared Jev function.

In my small Discord test, the full receive-to-delete flow came in at around a second. And the actual AI part is tiny.

It's just this:

const res = await composio.tools.execute("JEV_EVALUATE_STATE", {
  userId: process.env.COMPOSIO_USER_ID,
  arguments: { state: text, questions: QUESTIONS },
  dangerouslySkipVersionCheck: true,
});

Finally, you don't have to wait on an LLM sending response back 10s after you send the msg. This is exactly why I did this blog post as well. Right model with right pricing for exactly this tyoe of tasks.

If you find a bug, want to improve something, or manage to get the YouTube version working, feel free to open an issue or PR.

You can find the entire project here: shricodev/jev-youtube-discord-moderator

That's all for this one. I'll see you in the next one. ✌️

I Tested 7 Document Parsing APIs on Real Scanned PDFs (2026): Code and Results

Shrijal Acharya — Mon, 21 Sep 2026 12:56:36 +0000

Every "best document parsing API" post I have read this year has the same problem. Nobody actually calls the APIs.

They copy the pricing page, add a feature table, rank everything, and call it a day. Zero PDFs were harmed in the making of that content. 🥴

So I did something else. I took 11 real documents from public datasets. Invoices, a French bank statement, a photographed receipt, a handwritten cheque, two contracts, and a medical EOB.

Then I wrote one JSON schema for each document type and sent the exact same files through Extend, Reducto, LlamaExtract, AWS Textract, Mistral OCR, Claude, and GPT. 119 API calls in total.

I saved every response too, so you can check the work yourself. This one got interesting pretty fast.

Let's get into it.

ℹ️ NOTE Everything below comes from stored API responses. The harness is public too, so you do not have to take my word for any of it.

TLDR

Category	Winner	Runner up	Notes
Raw field accuracy	Claude direct 0.991	GPT direct and Reducto 0.982	11 docs, fixed scoring, no LLM judge
Never hallucinated a value	Extend 0 out of 11	Claude, GPT, Reducto also 0	Textract 2, Mistral 3, LlamaExtract 3
Table rows never missed	Extend row F1 1.00	Everyone else 0.99	Only perfect row F1
Cheapest per correct field	GPT direct $0.00047	Mistral OCR $0.00049	Textract was 14x GPT
Fastest	Mistral OCR 4.2s median	Claude direct 6.4s	Mistral also hallucinated 3 times
Developer experience	Claude and GPT about 10 min	Extend about 25 min	Textract took about 45 min plus another LLM step
Agent support	Extend llms.txt, agents.md, MCP, plugin	Reducto 3 of 4	AWS has nothing specific for Textract
Hardest to justify	AWS Textract		Highest cost in this test and some ugly misses

The short version is pretty simple. Claude had the best raw accuracy. GPT was the cheapest per correct field. Extend was the only document API with zero hallucinations and perfect row recall.

That last part matters a lot once the output is going into a real system. If a human is checking the JSON, raw accuracy can be enough. If software is going to act on the result, I care a lot more about made up values.

Textract had the roughest run here. I will show you why.

What I tested

There are six products in the test. Claude and GPT run directly, so the result table has seven columns.

#	Provider	Endpoint	Native schema	Mode
1	Extend	`POST /extract`	yes	default, API `2026-02-09`, `extend-ai` 1.19.0
2	Reducto	`POST /extract`	yes	standard, `deep_extract` off
3	LlamaExtract	`extract_stateless()`	yes	`BALANCED`
4	AWS Textract	`AnalyzeDocument` FORMS + TABLES	no	plus LLM normalizer, us-east-1
5	Mistral OCR	`/v1/ocr` + `document_annotation_format`	yes	`mistral-ocr-latest`
6a	Claude direct	`/v1/messages`, forced tool	yes	`claude-opus-5`
6b	GPT direct	`/v1/responses`, `json_schema`	yes	`gpt-5.5`, `strict: false`

A couple of things before we start.

LlamaParse structured output is deprecated. If you want schema extraction on LlamaCloud now, you use LlamaExtract.

The old llama-cloud-services package is also deprecated, so I used llama-cloud>=1.0.

AWS Textract does not return your schema as JSON. It gives you blocks like LINE, KEY_VALUE_SET, TABLE, and CELL.

So I had to turn those blocks into text and pass that text to claude-haiku-4-5 to get the final JSON. That extra model call counts toward Textract cost and latency.

💡 In simple words, Textract could read the page, but it needed another model to shape that reading into the same output every other tool was asked to return.

Keep that in mind for the numbers later.

The documents

I used 11 documents and 35 pages in total. All of them came from public datasets.

The mix was roughly 3 easy, 4 medium, and 4 hard.

#	Document	Source	Pages	Why it is here
01	invoice_clean	Voxel51 synthetic invoices	1	clean invoice, 7 rows, European decimals
02	invoice_complex	FATURA2	1	different layout, discount row, weird total
03	receipt_photo	SROIE 2019	1	low resolution and handwriting on top
04	contract_simple	CUAD	7	agreement date is blank
05	contract_complex	CUAD pages 1 to 8	8	governing law is outside the page limit and `$6,200,000` is not the contract value
06	bank_statement	Bankstatemently	2	French Canadian, 25 transactions, watermark, 2 digit years
07	noisy_form	FUNSD	1	scanned form with 41 label and value pairs
08	handwritten_note	handwritten cheque dataset	1	written amount and number box disagree
09	EOB	CMS sample	1	medical EOB with `XXXXXX` placeholders
10	mixed_packet	built from 01, 04, and 09	5	checks page boundaries
11	contract_scanned	scanned copy of 04	7	same contract, but rasterised at 200dpi

Every file is locked with a sha256 in manifest.lock.json. If one byte changes, the runner stops.

That sounds a bit extreme, but it saves a lot of headaches later.

💡 It means every provider saw the exact same file. No accidental OCR. No changed PDF. No weird hidden advantage.

I also made every image source into a pixel only PDF with img2pdf. Then the build script checks it with pdftotext.

If any text layer exists, the build fails. Why bother

Because if I OCR the file before the benchmark, I have basically handed every provider the answer key. And that kinda kills the whole test. 💁

The traps

This is probably the most useful part of the corpus.

A few fields are genuinely missing from the documents. The right answer for those is null.

If a provider gives me a value anyway, it made something up.

Doc 04 has a blank agreement date. The effective date exists, so it is very easy to copy the wrong date.
Doc 05 stops at page 8. The governing law is on page 26. It also contains $6,200,000, but that number is a financing cap, not the contract value.
Doc 01 has no due date and no discount.
Doc 08 disagrees with itself. The words say one amount and the number box says another. The correct output keeps both.

💡 These are not OCR tricks. I wanted to see what happens when the safest answer is simply "I do not know".

How I kept the test fair

The numbers are useless if every provider gets a different setup. So each one got the same thing.

The exact same PDF
The exact same JSON schema
The exact same instruction
The same 2 second poll time and 300 second timeout
Retries only for network or server errors
The normal default mode for each product

No custom prompt for one provider. No premium mode for another.

A wrong answer did not get a second try. The prompt was tiny too.

# bench/providers/base.py
SHARED_INSTRUCTIONS = (
    "Extract the fields defined by the schema from this document. "
    "Use exactly the values printed in the document. "
    "Return dates as ISO 8601 (YYYY-MM-DD) and amounts as plain numbers "
    "without currency symbols or thousands separators. "
    "If a field is not present in the document, return null for it "
    "rather than guessing or inferring a value."
)

POLL_INTERVAL_S = 2
TIMEOUT_S = 300

And a schema looks like this.

{
  "type": "object",
  "properties": {
    "vendor_name": { "type": ["string", "null"] },
    "due_date": { "type": ["string", "null"], "description": "ISO 8601" },
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "description": { "type": ["string", "null"] },
          "quantity": { "type": ["number", "null"] },
          "amount": {
            "type": ["number", "null"],
            "description": "line total EXCLUDING tax"
          }
        }
      }
    }
  }
}

The important bit is that fields can return null. That lets the provider say the value is missing instead of forcing it to guess.

💡 null is not a failure here. Sometimes it is exactly the right answer.

That "line total EXCLUDING tax" line exists because I got burned while building this. My first schema just said amount.

The invoice has Net price, Net worth, and Gross worth. Every provider picked something different and for about an hour I thought I had found a huge accuracy gap.

But nope.

My schema was just vague. 🫠

One adapter for every provider

Every company has a completely different way to say "send PDF, give JSON".

extend        POST /extract         upload -> file id   -> inline schema config
reducto       POST /extract         upload -> reducto:// -> instructions.schema
mistral       POST /v1/ocr          upload -> signed URL -> document_annotation_format
llamaextract  extract_stateless()   upload -> file id   -> ExtractConfig, maybe poll
claude        POST /v1/messages     base64 inline       -> forced tool, input_schema IS the schema
gpt           POST /v1/responses    base64 inline       -> text.format json_schema
textract      AnalyzeDocument       per-page blocks     -> LLM normalizer stage

Some upload first. Some want a signed URL. Some poll. Some return the JSON inside another object.

Mistral even returns document_annotation as a JSON string. Thanks Mistral. 😭

I hide all of that behind one function.

class Provider(Protocol):
    name: str
    def extract(self, pdf: Path, schema: dict, schema_id: str) -> ExtractResult: ...

That is what keeps the runner fair. The runner does not know which provider it is calling.

It just sends a PDF and expects the same result shape back.

The Claude adapter is tiny.

# bench/providers/direct_llm.py (trimmed)
def extract(pdf: Path, schema: dict, schema_id: str) -> ExtractResult:
    b64 = base64.b64encode(pdf.read_bytes()).decode()
    resp = client.messages.create(
        model="claude-opus-5",
        max_tokens=8192,
        tools=[{"name": "emit", "description": "Return the extracted fields.",
                "input_schema": schema}],
        tool_choice={"type": "tool", "name": "emit"},
        messages=[{"role": "user", "content": [
            {"type": "document",
             "source": {"type": "base64", "media_type": "application/pdf", "data": b64}},
            {"type": "text", "text": SHARED_INSTRUCTIONS},
        ]}],
    )
    block = next(b for b in resp.content if b.type == "tool_use")
    return ExtractResult(
        data=block.input,
        usage=Usage.from_anthropic(resp.usage),
        raw=resp.model_dump()
    )

No upload. No job ID. No polling.

The schema itself becomes the tool input.

Textract is the opposite. It needs about 60 lines just to rebuild rows from CELL blocks, then another model call to turn that into the target JSON.

# bench/providers/textract.py (trimmed)
def _table_rows(blocks_by_id, table):
    cells = [
        blocks_by_id[i]
        for r in table.get("Relationships", [])
        if r["Type"] == "CHILD"
        for i in r["Ids"]
    ]

    grid = defaultdict(dict)

    for c in cells:
        grid[c["RowIndex"]][c["ColumnIndex"]] = _cell_text(blocks_by_id, c)

    return [
        [grid[r].get(col, "") for col in sorted(grid[r])]
        for r in sorted(grid)
    ]

normalized = normalizer.run(text=rendered, schema=schema)

💡 Same task on paper. Very different amount of work in code.

Scoring without an LLM judge

I did not use another LLM to decide who won. That just creates a second argument.

The scorer is fixed.

Normalise first Money becomes Decimal, dates become ISO, and strings get cleaned so tiny formatting changes do not count as errors.
Scalars Exact after normalisation. Longer text uses rapidfuzz token_set_ratio. A score of 90 or more counts as correct.
Verdicts A field can be correct, near, wrong, missing, or hallucinated.
Arrays Rows are matched using text fields only.
Missed rows count If a provider returns 1 perfect row out of 7, it does not magically get 100 percent. 🥴

Before scoring the providers, the scorer checks itself. First, all 11 ground truth files are scored against themselves.

Every one must return exactly 1.000.

Then I deliberately break the data. Wrong total. Fake date. Missing vendor. Dropped rows.

Every one of those changes has to lower the score. Dropping 4 of 7 rows gives 0.579.

Not some fake 1.0.

$ uv run bench score
self-test  identity     11/11  =1.000  OK
self-test  corruption   6/6    reduce  OK
scoring 77 runs ...

run and score are separate commands. run saves the raw API response to disk. score only reads those saved files.

That meant I could rewrite the scorer four times without paying for the APIs again. Pretty useful when you find a bug at 1 AM. 👍

1. Accuracy

There were 77 main runs. Zero API errors. Zero invalid JSON responses.

Every provider returned something usable on every document.

Provider	Field accuracy	Row F1	Hallucinated	Missing	Median latency
Claude direct	0.991	0.99	0	2	6.4s
GPT direct	0.982	0.99	0	2	10.1s
Reducto	0.982	0.99	0	4	10.1s
Extend	0.962	1.00	0	2	22.1s
AWS Textract	0.936	0.99	2	1	14.4s
LlamaExtract	0.903	0.99	3	3	22.5s
Mistral OCR	0.884	0.99	3	3	4.2s

I kept field accuracy and row F1 separate. I did not turn everything into some made up score out of 10.

Because then half the benchmark becomes me deciding how much each number should matter.

Claude came first on raw field accuracy. GPT and Reducto tied next.

Extend landed at 0.962, but it had the only perfect row F1 in the whole test. It did not miss a single table row across 25 bank transactions, 7 invoice lines, 41 form pairs, and all the EOB rows.

That is a pretty important detail. 0.991 and 0.962 look different on a leaderboard.

Across these 11 documents, the gap was under 3 points.

Per document

Document	Claude	GPT	Reducto	Extend	Textract	LlamaExtract	Mistral
01_invoice_clean	1.00	1.00	1.00	1.00	1.00	1.00	1.00
02_invoice_complex	1.00	1.00	1.00	1.00	0.97	1.00	1.00
03_receipt_photo	1.00	0.92	1.00	1.00	1.00	1.00	1.00
04_contract_simple	1.00	1.00	1.00	0.93	0.86	1.00	0.86
05_contract_complex	0.93	0.93	0.86	0.93	0.79	0.86	0.64
06_bank_statement	1.00	1.00	0.99	1.00	0.98	1.00	0.99
07_noisy_form	0.98	0.95	0.95	0.95	0.86	0.74	0.93
08_handwritten_note	1.00	1.00	1.00	1.00	1.00	0.67	0.83
09_EOB	1.00	1.00	1.00	1.00	1.00	0.75	0.95
10_mixed_packet	1.00	1.00	1.00	0.92	0.92	0.92	0.67
11_contract_scanned	1.00	1.00	1.00	0.86	0.93	1.00	0.86

A few things stand out.

The clean invoice is a 7 way tie at 1.00. If your documents all look like that, honestly, pick the cheapest one and move on. 🤷‍♂️

The complex contract is where Mistral drops hard to 0.64. LlamaExtract had the roughest time on the noisy form, handwriting, and EOB.

Now the fun part.

The failures.

2. Where things broke

This is why I saved every raw response. A final accuracy number hides way too much.

The hallucination traps

The first trap was the blank agreement date in Doc 04. The correct answer is null.

Mistral returned 2012-02-17. Textract returned 2012-02-17.

Claude, GPT, Reducto, Extend, and LlamaExtract returned null.

The second trap was the $6,200,000 financing cap in Doc 05. Again, the correct contract value is null.

Mistral, LlamaExtract, and Textract returned 6200000. Claude, GPT, Reducto, and Extend returned null.

Three products turned a financing limit into a contract value. That is not the kind of error I want quietly flowing into an AP system.

💡 The scary errors are not always random garbage. Sometimes the wrong answer looks completely reasonable.

Textract dropped digits from money

On the bank statement, I found these two.

1 200,45 $ became 200.45

6 675,00 $ became 675.00

These are not formatting misses. The first digit is just gone.

And this is money. 🤧

I checked the summary on the same page too. The credits, debits, and closing balance all match the ground truth.

Then I checked Textract raw blocks. The digit was already missing before Haiku saw anything.

So the normalizer did not cause this one.

LlamaExtract shifted a whole column

The EOB result was sneaky. LlamaExtract found the right rows, but some values moved one column to the left.

Field	Expected	LlamaExtract
`services/0/plan_paid`	2.15	0.0
`services/0/patient_responsibility`	0.0	2.15
`services/1/billed_amount`	375.00	118.12
`services/1/allowed_amount`	118.12	35.00

Every number exists in the row. It is just under the wrong field.

That is exactly the kind of JSON you can skim and think looks fine.

Mistral found pages that do not exist

The mixed packet has five pages. Mistral said the contract was on pages 2 to 10 and the EOB was on 11 to 13.

Every other provider got 1-1, 2-4, and 5-5.

I still do not know where page 13 came from. 🫩

Handwriting fooled GPT

The receipt has tan chay yee written by hand at the top. GPT used that as the merchant_name.

The printed merchant is OJC MARKETING SDN BHD. Every other provider ignored the handwriting.

So yeah, best overall does not mean best on every file.

Mistral answered a slightly different question

Doc 05 calls the two parties Sekisui and Qualigen. Mistral returned Distributor and Manufacturer.

Those words make sense in the contract, but they are not the defined names I asked for. Part of this is schema sensitivity.

Still, it cost two fields.

Apparently the pipe symbol was hard

One invoice row says this.

Dell Core 2 Duo Desktop Computer | Windows XP Pro | 4GB | 500GB

Textract read | as I. Mistral removed the pipes.

Everyone else got it exactly right.

Tiny character. Same invoice. Two different OCR mistakes.

Lol.

Extend never made up a value

Extend had a very different failure pattern.

Zero hallucinations across all 11 documents
Passed both traps
Only perfect row F1 at 1.00
8 of 11 documents scored 1.000
0.962 field accuracy compared with 0.991 for the leader

Most of the lost points came from the contracts. A couple of fields were present, but Extend returned null.

That is still wrong for the benchmark. But it is a much easier failure to deal with in a production flow.

A missing value can go to review. A confident wrong number can move straight into the next system.

💡 Extend was more likely to leave a field empty than guess. Depending on your workflow, that can be a useful trade.

3. Pricing

Here is what the 35 page test cost.

Provider	Total	Cost per document	Cost per 1000 pages	Cost per correct field
GPT direct	$0.1736	$0.0158	$4.96	$0.00047
Mistral OCR	$0.1750	$0.0159	$5.00	$0.00049
Claude direct	$0.3734	$0.0339	$10.67	$0.00100
LlamaExtract*	$0.4375	$0.0398	$12.50	$0.00126
Reducto*	$0.5250	$0.0477	$15.00	$0.00142
Extend	$2.1875	$0.1989	$62.50	$0.00593
AWS Textract	$2.3553	$0.2141	$67.30	$0.00663

Pricing for Reducto and LlamaExtract could not be fully verified from usage data.

GPT was easily the cheapest per correct field. Textract was the most expensive in this test.

And because it also needed the extra Haiku step, its real setup is not just the Textract call.

What you get with Extend pricing

Extend reports both extraction credits and total credits.

"usage": { "credits": 3.0, "totalCredits": 5.0 }

Across my runs, that came to 5 credits per page. On pay as you go pricing, that is about $62.50 per 1,000 pages.

But you are also getting more than a JSON response. Extend includes parsing, structured extraction, field citations, confidence scores, and a review flow in the dashboard.

Scale pricing drops to roughly $50 per 1,000 pages. Enterprise pricing is custom.

💁 So I would not compare this only as price per page. The extra cost is paying for the workflow around extraction too.

Textract pricing gets messy fast

Textract prices features separately. FORMS is $0.05 per page. TABLES is $0.015 per page.

Using both comes to $0.065 per page. For 35 pages, that was $2.275.

Then I had about $0.08 of Haiku cost for the JSON normalizer. That brings the matrix to around $2.36.

The billing screenshot can show more than that because it also includes my reruns and integration tests.

There is also DetectDocumentText. That costs $0.0015 per page and has a bigger free tier.

I did not test that setup here. But plain OCR plus a cheap model would be a very interesting follow up.

Reducto and LlamaExtract pricing

I could not fully verify Reducto extraction cost from the usage dashboard. It was still showing the usage page as under construction when I checked.

So the $15 per 1,000 pages number uses the published parse rate. Extraction can add more credits.

LlamaExtract uses its published rate in the table with the same caveat.

4. Developer experience

I also tracked how annoying each integration was to build. Very scientific metric, I know. 😆

Rank	Provider	Time	What happened
1	Claude	about 10 min	worked
1	GPT	about 10 min	worked
3	Extend	about 25 min	documented path worked first try
4	Reducto	about 20 min	runtime typing issue and serializer crash
5	Mistral	about 15 min	documented import failed
6	LlamaCloud	about 30 min	old package docs and mixed pydantic versions
7	AWS Textract	about 45 min	no schema output, table rebuild code, extra LLM step

Claude and GPT were simple because the PDF can go straight into the model call. Extend was the easiest document API.

Its documented path worked on the first try. That should be normal.

It was not. ✌️

Mistral import issue

The docs showed this.

from mistralai import Mistral

On v2.9.4 I got this.

ImportError: cannot import name 'Mistral' from 'mistralai' (unknown location)

The working import was this.

from mistralai.client import Mistral

15 minutes of my life. 😴

LlamaCloud setup

LlamaCloud sent me through two old paths before I even started. LlamaParse structured output is deprecated.

llama-cloud-services is also deprecated.

Then the client lives at llama_cloud.client.LlamaCloud, and ExtractConfig still uses pydantic v1 style while the rest of the package uses v2.

It works. It just took more digging than I expected.

Extend setup

Extend was boring in the best way. The documented SDK flow worked.

No weird import hunt. No second package. No guess and check loop.

That is also where its agent files helped.

5. Agent experience

This part is becoming more important now. Can Claude Code, Codex, or another coding agent actually understand the SDK without me spoon feeding it everything

I checked what each vendor ships for agents.

What vendors publish

Checked on September 8 2026.

Provider	llms.txt	agents.md	MCP server	Official agent plugin
Extend	yes	yes	yes	yes `extend-hq/extend-agent-plugin`
Reducto	yes	yes	yes	no
Mistral	yes	no	no	no
LlamaCloud	yes	no	no	no
Anthropic	yes
OpenAI	yes
AWS Textract	org wide only	no	no	no

llms.txt alone is not that special anymore. Six of the seven have one.

Extend agents.md had all four. Reducto had three.

AWS has a general AWS llms.txt, but nothing specific for Textract.

What actually helped

Extend agents.md had one small line that saved real time. It says that REST paths and SDK methods can have different names.

For example

POST /extract_runs

maps to

client.extract_runs.create()

That sounds tiny. But it stopped the exact guess and check loop I hit with another SDK.

💡 Agent docs are useful when they answer the weird little questions a coding agent would otherwise guess.

Things I got wrong

This benchmark also caught one annoying problem with the datasets themselves. Published ground truth can be wrong too.

The SROIE answer key says the receipt postcode is B1750. The receipt clearly says 81750.

FUNSD has Macket where the form says Market.

So if I blindly used the dataset labels, I would punish a provider for reading the document correctly.

Nice.

This is another reason I kept the raw files and responses. Benchmarks can be wrong in boring ways.

Which one should you pick

There is no one winner for every workflow.

Pick Claude directly when

You want the highest raw field accuracy from this test
A human will review the result
You do not need field citations or confidence scores
You already use the Anthropic API

Pick GPT directly when

Cost matters a lot
You still want very strong raw accuracy
A human can check the weird cases

Pick Extend when

A made up value is worse than a missing one
You need field citations
You need confidence scores
You want a review UI
The extracted data goes into a real production workflow
Your team uses coding agents and good agent docs matter

Pick Reducto when

You want document specific tooling with strong raw accuracy
You want zero hallucinations in this test
You have checked the extraction pricing for your own workload

Pick Mistral when

Speed is the main thing you care about
A human checks the result before anything acts on it

Textract

For this exact setup, Textract was hard to justify. It cost the most, needed an extra model step, hallucinated twice, and dropped leading digits from money.

Your workload may be different. But based on these 11 documents, I would test the other options first.

Reproduce it

Everything is in the repo: shricodev/document-parsing-api-benchmark

uv sync
uv run bench doctor
uv run bench run --all
uv run bench score
uv run bench report

bench doctor checks credentials, corpus hashes, and spend limits. bench run will not call an API again if a successful result is already saved.

So if the run stops halfway through, you can resume without paying for the same documents twice.

$ uv run bench run --provider extend --doc 06_bank_statement
extend / 06_bank_statement  cached (runs/extend/06_bank_statement/default.json)  skipping

Every raw response lives in runs/. That means every result in this post can be scored again without an API key.

The source PDFs are not committed because of licensing. corpus/build/fetch.py downloads and freezes them instead.

Conclusion

The result was closer than I expected.

Claude had the best raw field accuracy. GPT was extremely cheap for how well it scored. Reducto was right behind them.

And Extend had the cleanest safety result in the group. Zero hallucinations. Zero missed table rows. Its SDK path worked on the first try too.

That is why I would not pick one of these from the accuracy column alone.

If a human is reading the output, calling Claude or GPT directly can make a lot of sense. If the JSON is going straight into AP, claims, finance, or another production system, I care much more about citations, confidence, review, and what happens when the model is not sure.

That is where Extend made the strongest case in this test.

And yeah, run your own benchmark. Freeze the files, save the responses, and add a few nasty cases.

This one cost me $9.67. It changed what I would actually use in production. ✅

DeepSeek Harness (DSH) vs Pi Agent: Everything you need to know

Shrijal Acharya — Wed, 09 Sep 2026 12:45:00 +0000

DeepSeek released its own agent harness this August.

It crossed 66k stars in about a day (yes, sir!) and went over 200k by early September. That is already pretty crazy for something that is still a developer preview.

But while researching this post, I found something much more interesting. DeepSeek Harness actually uses Pi's model layer to connect to models outside DeepSeek. Yep.

The new agent runtime from DeepSeek uses Pi under the hood for part of its model support. And Tianyi Cui, who leads the Harness project, has publicly said Pi is a favourite daily driver for many people at DeepSeek. Lol.

That makes this comparison way more fun because they agree on quite a lot. Both think the harness matters, both are open source, both let you change a lot of the runtime, and both try to avoid locking you into one model.

But they take the idea in very different directions. Pi gives you a tiny coding agent with four tools and lets you build from there. DeepSeek Harness gives you a full agent runtime where even the agent loop itself can be swapped.

This is going to be an interesting one. Stick to it!

TL;DR

Category	Pi	DeepSeek Harness	Winner
Real tool use in our eval	21/30 passed	20/30 passed	Pi
Cost per shared success	$0.031	$0.028	DeepSeek Harness
Median time per task	362.9s	252.1s	DeepSeek Harness
Avg runtime tokens in our run	924,990	88,562, with a catch	DeepSeek Harness
Simplicity	Four main tools and a tiny prompt	Much larger runtime with 53 built in tools	Pi
Deep runtime control	Powerful TypeScript extensions around a small core	Even the agent loop can be replaced	DeepSeek Harness
Model support	25+ providers and very easy switching	Broad support, partly through Pi's model layer	Pi
Local models	Ollama, vLLM, and custom providers are well supported	Possible, but needs more setup	Pi
Sandbox	None in the core	Three built in sandbox modes	DeepSeek Harness
Session inspection	Very readable session tree	Full append only trajectory and replay	DeepSeek Harness
Daily use	Very small and easy to run	Web UI and more setup	Pi
Maturity	Mature enough for normal daily work	Still a developer preview	Pi

In summary

Scorecard: Pi 6, DeepSeek Harness 6. Dead tie.

And the benchmark is basically a tie too. Pi passed 21 of 30 tasks and DeepSeek Harness passed 20. DeepSeek Harness was cheaper on the shared successful tasks and also finished tasks faster.

But there is an important catch. DeepSeek Harness ran through DeepSeek's own endpoint. Pi ran through OpenRouter. So I would trust the pass rate and cost comparison more than I would trust a clean 10x token claim.

Still, the overall result is pretty clear. Pi is the better choice if you want a coding agent you can install, understand, modify, and use every day.

What is DeepSeek Harness?

ℹ️ DeepSeek's open source agent runtime where almost every part of the agent can be replaced as a plugin.

DeepSeek Harness, or dsh, is DeepSeek's official agent harness. DeepSeek released it on August 13, 2026 alongside DeepSeek V4 Pro.

And the basic idea is simple:

Agent = Model + Harness

The model does the thinking. The harness handles everything around it. It gives the model tools, builds the prompt, runs shell commands, manages files, stores sessions, handles context, runs the loop, and decides how all these pieces connect.

That part is normal. Claude Code, Codex, Pi, OpenCode, and basically every coding agent already has some version of this.

DeepSeek did something different with how it built the runtime. It uses a plugin system called Cordis, and almost everything runs as a plugin. The model adapter is a plugin, the tools are plugins, the sandbox is a plugin, the session system uses plugins, and even the main agent loop can be replaced.

Cordis also gives plugins a proper lifecycle. You can mount them from config, reload them, and clean them up when they are removed. So this is much closer to building an agent runtime from replaceable parts than adding a few extensions to a coding CLI.

💁 DeepSeek's own docs basically say there is no special core that you need to patch.

It also ships several different modes. Standard gives you the full agent setup. Code mode, also called PTC, gives the model a generated TypeScript SDK for its tools. So instead of making lots of separate tool calls, the model can write one small program that calls those tools and handles the intermediate data itself.

Minimal gives it basically shell plus a file editor. Creator lets the agent inspect and modify parts of its own runtime. 🤯

Yes, that last one is as weird as it sounds. It is very different from most coding agents.

It also comes with 53 built in tools, MCP support, subagents, LSP, web search, scheduling, sandboxing, and a local browser UI. So despite the plugin first design, this is not a tiny framework with nothing included.

What is Pi?

ℹ️ A minimal terminal coding agent that gives the model a few tools and expects you to build whatever else you want.

You probably know Pi by now if you have read the other posts in this series. Mario Zechner built it after getting tired of how much stuff Claude Code kept adding.

Pi goes almost completely in the other direction. The model gets read, write, edit, and bash. The system prompt stays under roughly 1,000 tokens. And that is most of the product.

Need grep? Use bash. Need Git? Use bash. Need another tool? Write a TypeScript extension. Need a whole new workflow? Ask Pi to build the extension for you.

The project now lives under Earendil, has around 100k GitHub stars, and is still open source and MIT licensed.

The interesting part is that Pi is not small because nobody built more features yet. It is small because that is the actual design. Every extra tool schema and every extra instruction costs tokens on future model calls, so Pi tries to keep the default harness small.

DeepSeek Harness "kinda" agrees with some of this, but there's a whole lot to it.

1. Philosophy: Pi vs DeepSeek Harness

This is probably the most interesting part of the whole comparison because at first they look very similar. Both are open source, both are written mostly in TypeScript, both have the concept of extensions, and both want the model layer to stay flexible.

And DeepSeek Harness literally uses Pi's model layer for its broad provider support.

But they disagree on what the starting point should look like.

Pi starts tiny

Pi has a small core agent loop. That loop stays pretty simple and extensions can change almost everything around it.

The idea is that the model already knows how to code. So give it file access, shell access, editing, and enough freedom to work. Then stop adding stuff unless the user actually needs it.

DeepSeek Harness starts from replaceable parts

It splits the runtime into parts and lets you swap those parts through Cordis. Pi gives you a small core and says, build around this. DeepSeek Harness gives you a way to replace its parts too.

Even the actual loop works differently from an extension point of view. DeepSeek Harness fires events through each turn, such as turn/start, system-prompt/assemble, tools/pre-execute, and turn/end. A plugin can hook into these points, change messages, add behaviour, or stop a tool before it runs.

That is a lot deeper than adding another command or tool.

That sounds like a small difference. It is actually the whole comparison.

💁 Pi is easier to understand. DeepSeek Harness gives you deeper control (DeepSeek gives you deeper control, what a nice rhyme to the name itself 🫡).

There is another interesting part here. DeepSeek used its own Minimal mode for its public V4 coding benchmarks. Minimal mode is basically shell, a file editor, and a tiny system prompt.

So even DeepSeek's own benchmark setup gets pretty close to the same minimal idea Pi has been pushing from the start.

2. Architecture: four tools vs basically the whole runtime

Underneath, both still do the normal agent thing. You send a task, the model responds, it calls tools, the harness runs them, the results go back to the model, and this repeats until the model finishes.

The difference is everything around that loop.

Pi Agent

Pi runs as one small process. It has the model layer, the agent loop, the terminal UI, session handling, and the extension system.

Its session format is also pretty neat. Messages form a tree (/tree slash command), so you can jump back to an old point and continue in another direction without throwing away the old path. And because the core is small, you can actually open the source and follow what is happening.

It is one of Pi's best features, at least for me.

Pi extensions are also just TypeScript files. You can put them inside ~/.pi/agent/extensions/ and use them to add tools, block calls, change context, replace compaction behaviour, or add your own TUI stuff.

That is a very simple way to extend a coding agent because you do not need to understand a big plugin runtime first.

DeepSeek Harness

DeepSeek Harness is much bigger, and it does much more. It has a full plugin graph, sandboxing, subagents, MCP, scheduling, compaction, a browser UI, an append only session log, replay, and much more.

The session system is probably my favourite part. Every model visible event gets logged. So if the model saw something, there should be a record of it. You can inspect the trajectory later and see what happened, or replay and fork sessions from the stored events.

There is a technical reason for doing this too. DeepSeek Harness does not go back and rewrite old session history. It keeps appending new events instead.

That makes it easier to keep the beginning of the model context stable, which is useful for DeepSeek's prefix caching.

That is very useful if you are building agent infrastructure.

I would take Pi for understanding my own coding session. I would take DeepSeek Harness if I were building a system that runs thousands of agent sessions and I needed to inspect exactly what happened.

3. The benchmark: same model, different harness

This is the part I was most curious about. We ran 30 hard agent tool use tasks across the harnesses.

The agent has to work with real apps and real data through tools, and then our verifier checks whether it actually completed the job. Each task had a 900 second cap.

For this run, we used DeepSeek V4 Pro.

Here is what happened:

Harness	Tasks passed	Median time	Avg runtime tokens	Cost per shared success
Pi	21/30	362.9s	924,990	$0.031
DeepSeek Harness	20/30	252.1s	88,562	$0.028

Pi won by one task. That is basically a tie on quality. Pi passed 21/30 tasks, while DeepSeek Harness passed 20/30.

And that matters because DeepSeek built Harness around its own model. You might expect the first party harness to have some huge advantage. It did not. Pi actually finished one more task.

But pass rate is only one part of the story.

DeepSeek Harness also came out slightly cheaper. It cost around $0.028 per shared success, compared to $0.031 for Pi. OpenCode was very close at $0.032, while Claude Code came out much higher at $0.074.

The difference gets more interesting when you look at speed.

DeepSeek Harness had a median time of 252.1 seconds per task, while Pi took 362.9 seconds. Claude Code was the fastest at 181.8 seconds, but it also cost much more per successful task.

So DSH did not beat Pi on task success, but it was faster and slightly cheaper in this run.

Then look at the runtime token column. That number is wild.

DeepSeek Harness recorded about 88k runtime tokens per task. Pi recorded about 925k.

Again, there is a provider routing catch here, and I will get into it in a second. But DSH clearly ran DeepSeek V4 Pro very efficiently on its native route.

The task split is also interesting

Most of the tasks ended the same way for Pi and DSH. Pi and DeepSeek Harness agreed on 27 of the 30 outcomes. The entire 21 vs 20 difference came down to three tasks.

Pi passed the CRM migration archive task while DeepSeek Harness failed it. Pi also passed the handover audit while DeepSeek Harness failed that one. DeepSeek Harness got the Sheets to Airtable tracker right while Pi failed it.

Task	Pi	DeepSeek Harness
CRM migration archive	✅	❌
Handover audit	✅	❌
Sheets to Airtable tracker	❌	✅

The bigger chart is useful because it shows that this did not only happen between Pi and DSH. The same DeepSeek model changed behaviour when we moved it between Claude Code, OpenCode, Hermes, Pi, and DeepSeek Harness.

For example, Claude Code cleared every check on the renewal screen task while Pi and DSH only cleared 5 of 11. On Slack action items, Pi, DSH, Claude Code, and Hermes all passed, while OpenCode only cleared 2 of 5 checks.

So we did not see one harness destroying the others across every kind of work. The same model behaved differently depending on the harness around it.

And I think that is the important result. The harness changed efficiency a lot. It changed speed. It changed cost. And on some tasks, it changed whether the model actually finished the job.

💁 Same model. Almost the same pass rate. Very different runtime behaviour.

So did DeepSeek Harness really use 10x fewer tokens?

The raw table makes it look like Pi is eating up tokens, but that is not really the full story.

DeepSeek Harness used DeepSeek's official endpoint, while Pi used OpenRouter. That changes how caching and token usage get reported.

Pi also has a much smaller base prompt. One separate comparison measured around 4.5k uncached input for Pi against 47.6k for DSH on the first turn. Pi keeps the prefix tiny, while DSH keeps a much larger prefix stable and gets strong cache reuse from it.

💁 They are solving the same token problem in different ways.

And once you look at actual cost per success, the gap is tiny: $0.028 for DSH vs $0.031 for Pi.

So DSH was very efficient in our run, but I would not treat the 88k vs 925k token number as a clean 10x win.

4. DeepSeek Harness uses Pi

This still makes me laugh. 🥴

For DeepSeek models, DSH has its own model adapter. Makes sense. That adapter understands things like DeepSeek reasoning levels and cache usage.

But when it needs other model support, it can use an adapter built on Pi's pi-ai layer.

So DSH can connect to Anthropic, OpenAI, Gemini, Bedrock, Vertex, Azure, Codex, and other providers through the model work Pi already did.

That tells you quite a lot about Pi. It looks like a tiny terminal coding agent from the outside, but its model layer has become super useful infrastructure on its own.

DeepSeek could have written another provider abstraction. It reused Pi's.

💁 I would probably do the same thing. Why not?

Pi still wins on model flexibility

Pi has had much more time to build this part out. It works with 25 plus providers, local models, OpenRouter, Ollama, vLLM, cloud providers, and custom compatible endpoints. You can also switch models during a session.

DeepSeek Harness can work with many of the same providers, but the whole product still feels very "DeepSeek first." And that is fine.

If you mainly use V4 Pro or V4 Flash, DSH has a real advantage because its native DeepSeek adapter understands the provider directly.

If you change models often, run local models, or use different cloud providers, Pi is easier.

5. Extensibility

Pi is one of the most hackable coding agents I have used. Its extensions run in TypeScript inside the same process. They can change tools, add UI, inject context, block tool calls, store custom state, and so much more...

And because the codebase is small, Pi can often write its own extension when you ask it to. That is still very cool.

A basic Pi extension can just live in ~/.pi/agent/extensions/. You do not need to learn a separate framework before adding one tool or one hook.

But DeepSeek Harness goes further. With DSH, the loop itself can be a plugin. So can the sandbox, model handling, context management, and a lot of the runtime around them.

A Cordis plugin can join the runtime's dependency graph, expose a service, listen to agent events, and clean itself up when it gets removed. That is more work for a simple extension, but it gives you much deeper control.

The agent can even inspect its own plugin tree and, in Creator mode, experiment with runtime changes. That is more control. It is also more stuff to understand and tbh, most of it is something I don't know at all.

If you are researching agent loops, building your own agent product, testing new context systems, or replacing parts of the runtime itself, DSH gives you way more room.

💁 Pi is a lot easier to extend. DSH is the deeper thing to change.

6. Safety and Security

Pi is chill about permissions. The agent gets your user permissions and can run commands. There is no big built in approval system protecting you from every shell command.

You can add safety through extensions or run Pi inside a container or VM, but the core itself does not try to do much here.

DeepSeek Harness comes with actual sandbox modes. It can run read only, it can allow writes inside the workspace, or you can give it full access.

And this is not just a UI permission setting. On Linux, DSH can use bwrap and Landlock. On macOS, it uses Seatbelt. Windows uses a restricted token model.

The sandbox also tries to fail closed. If DSH expects a sandbox and cannot create one, it should not silently continue with full access.

That is a real DSH win.

There are two catches though.

The sandbox controls tool execution, but plugins themselves run inside the harness process. So if you install a random third party plugin, you are still trusting that code with serious access to your machine.

And DeepSeek itself says the sandbox has not been security audited yet.

So I would still be careful with unattended runs, but DSH gives you much stronger safety defaults than Pi.

7. The biggest DSH problem right now is maturity

This is where the comparison stops being close for me.

Pi has been around for more than a year. People use it every day. OpenClaw runs on it. And the core is stable enough that people build real workflows around it.

DeepSeek Harness launched in August. It is still a developer preview and warns that compatibility breaking changes will happen.

And those changes are actually happening. The project pushed several alpha versions within days, and community reports have already found plugin breaks and many more issues.

There is also one governance thing worth knowing. GitHub Issues are disabled right now, and DeepSeek is not accepting outside pull requests yet. Feedback mainly goes through Discussions and Discord.

That does not make the project bad, especially this early. But if I am deciding which open source agent I want to build on today, that matters.

This is normal for a project this new, but it changes who I would recommend it to.

I would happily use DSH to experiment with agent architecture today. I would not build an important production workflow around a runtime that is still telling me its interfaces can break.

8. Daily use

This one is pretty simple.

Pi is a pretty regular agent that opens in the terminal. You pick a model and start working. The whole thing feels small because it is small.

DeepSeek Harness starts a local web app by default. You run it and open the UI in your browser. The official command is basically one npx call, so installation itself is not hard.

But once you start changing profiles, plugins, providers, Cordis config, sandbox settings, and runtime parts, there is much more to learn.

Pi can get you to a useful coding task in a few minutes. DSH needs more setup and its docs make much more sense once you already understand agent infrastructure.

That is also why I do not really see DSH replacing Pi for normal users right now. It solves a bigger problem, and bigger does not automatically mean better. 🤷‍♂️

Which one should you pick?

Pick Pi if:

You mainly want a coding agent
You want something small enough to understand
You use many different model providers
You run local models
You care about keeping the fixed prompt tiny
You want a clean TypeScript extension system
You want something mature enough to use every day
You would rather add the few features you need than start with 50 tools

Pick DeepSeek Harness if:

You are building agent infrastructure
You want to replace the agent loop itself
You need strong session traces and replay
You want a built in sandbox
You want MCP, subagents, LSP, and more tooling out of the box
You mainly use DeepSeek models and want the native integration

And you can obviously run both. They are free and open source. They even share some code.

Conclusion

The scorecard says 6 to 6. Our benchmark says Pi 21, DeepSeek Harness 20. That is as close as it gets.

But I do not think these two are actually fighting for the same user yet.

Pi feels like a coding agent made by someone who wanted the tool to stay out of his way. DeepSeek Harness feels like something built by people who want to inspect and change every part of how an agent works. Both are right. It depends on your preference. 💁

For normal coding, I would pick Pi today. It is smaller, easier to understand, and works with basically anything. It actually passed one more task in our run.

For agent infrastructure work, DSH is way more interesting. Being able to swap the loop, sandbox, model layer, and other runtime parts without rebuilding the whole agent is genuinely useful.

And DSH is still very new. It shipped all of this in a developer preview. If DeepSeek can make the runtime stable without losing the parts that make it interesting, I think it is going to be worth watching.

For now, I just would not treat it like a daily driver yet. DeepSeek itself tells you it is a developer preview.

So yeah. If I just want to code, I am still picking Pi. ✌️

Shrijal AcharyaFollow

SDE • GOLD @Microsoft Student Ambassador • Prev Lead Collab and Dev-Team Lead @oppiaorg • Mail for collaboration

Pi Agent vs OpenCode after 100+ Hours of Real Use ✌️

Shrijal Acharya — Fri, 21 Aug 2026 13:31:28 +0000

Open-source coding agents had a weird start to 2026.

In January, Anthropic suddenly blocked third-party tools from using Claude subscriptions. Overnight, with no warning.

OpenCode got hit the hardest. The team had to remove Claude login support after legal pressure. The commit message was literally just: “anthropic legal requests.”

PI never really cared. It was never built around one provider anyway.

And that kinda brings us to the obvious question. If you’re using your own API keys anyway, which open-source agent do you actually pick?

OpenCode is sitting at 199K stars, probably past 200K by the time you read this, with around 8 million monthly devs.

Then there’s Pi. Bare minimum by design. Four tools. A system prompt under 1,000 tokens. And somehow, top of our benchmark. Again.

ℹ️ We recently covered Pi Agent vs. Claude Code. Check that out if you're not sure whether Anthropic or Pi suits you better

The easiest way I’d frame it is this:

💁 OpenCode is VS Code (bit bloaty). Pi is NeoVim (bare minimum).

So which one do you go for? Let's get into the difference.

TL;DR

Category	Pi	OpenCode	Winner
Real tool use (our eval)	21/30 tasks passed, $0.078 per success	19/30 tasks passed, $0.119 per success	Pi
Total eval spend	$1.64 for all 30 tasks	$2.25 for all 30 tasks	Pi
Speed	362.9s median per task, slowest of the bunch	280.6s median per task	OpenCode
Context overhead	Under 1,000 tokens, prompt + tool defs	~6,900 tokens per request	Pi
Out-of-the-box features	Four tools	Plan mode, MCP, LSP diagnostics, undo/redo, permissions, subagents, and a lot more	OpenCode
Model support	20+ providers, 300+ models, mid-session switching	75+ providers via AI SDK and `models.dev`	OpenCode
Cheap model access	BYOK or local, that's it	Zen gateway at cost, plus Go at $10/month	OpenCode
Extensibility	TypeScript extensions running inside the agent process	Plugins, custom agents, skills, mostly config-driven	Pi
Sessions	Branchable session trees with fork and rewind	Linear log with undo/redo, but survives disconnects	Pi
Surfaces	Terminal	TUI, desktop app, web UI, IDEs via ACP, GitHub and GitLab CI	OpenCode
Permissions	Nothing. Full system access from prompt one, which isn't really my preferred style	allow/ask/deny per tool with glob patterns	OpenCode
Resource footprint	Lightweight, single process	1GB+ RAM for a TUI is kinda crazy	Pi

In summary

Scorecard: Pi 6, OpenCode 6. Dead tie.

And honestly, that feels right. These two are built for pretty different kinds of people anyway.

Remember the VS Code vs. Neovim thing I said earlier?

Pi is more like a platform you program. You get a tiny core and build your own harness around it in TypeScript.

OpenCode is more like a product you configure. You get something much closer to a full Claude Code replacement on day one, then tweak it through config.

Pi wins on efficiency, hackability, and, at least in our eval, raw results.

OpenCode wins on features, provider support, surfaces, and the simple fact that a normal person can pick it up and start using it without building half the setup themselves.

So yeah, the real tiebreaker is basically what kind of person you are. Keep reading.

What is Pi?

ℹ️ A minimal terminal coding agent that ships four tools and expects you to build the rest yourself.

Quick recap if you missed our Pi vs Claude Code post.

Pi is Mario Zechner's answer to harness bloat. He got tired of Claude Code changing under him, so he built an agent where the model gets read, write, edit, and bash, a system prompt of a few hundred tokens, and nothing else. No MCP. No permissions. No plan mode. No subagents. All on purpose.

It now lives under Earendil after his April blog post "I've sold out," and the core stays open-source.

The real product is the extension system. TypeScript files run inside the same process as the agent loop, and Pi can read its own source code. So when you need a missing feature, you literally ask Pi to build it.

That design is also why Peter Steinberger picked Pi as the engine for OpenClaw.

💁 Fun Fact: Pi has "only" 94k stars, but the npm package pulls around 1.3 million downloads a week. Growing crazy fast among developers.

Not everyone is fond of VS Code. 🤷‍♂️

What is OpenCode?

ℹ️ The most starred open source coding agent in the world, and the closest thing to an open, model agnostic Claude Code.

OpenCode is basically an open-source, model-agnostic coding agent that tries to give you the full Claude Code experience without locking you to Anthropic

It is built by the team behind SST, which rebranded to Anomaly in 2026 and moved the repo to anomalyco/opencode.

The numbers are crazy for a free tool. Around 199k stars. Roughly 8 million monthly active developers. Cloudflare publishes official OpenCode setup docs for its own engineers.

Can you believe this free thing is doing over $25M ARR? It's just the tool that's free, there's subscription plans they've introduced from where they make tons of money.

The origin story is crazier than Pi's. The original agent, called TermAI, was created by Kujtim Hoxha back in 2024. Dax Raad and Adam Elmore did the OpenCode rebrand and made it popular. Then Charm hired Hoxha and pulled the repo into its own org. That causes rewritten git history, banned contributors, the whole drama. After the backlash, Charm renamed its fork Crush, and the Dax and Adam project kept the OpenCode name as a from scratch rewrite.

Not anything interesting if you don't care about "history". Lol.

So when anyone says OpenCode in 2026, they mean the Anomaly one.

💁 Fun fact: For the 1.0 release, the team threw out their entire Go based TUI and wrote a new framework called OpenTUI in Zig and SolidJS, because the old one "had performance and capability issues." Can't believe Golang had performance issues at that scale.

A coding agent team writing their own Zig rendering engine for the terminal is the most 2026 thing I can think of. Crazy stuff.

1. Philosophy: Pi vs OpenCode

Our earlier comparisons were open vs closed. This one is open vs open.

Both teams agree on the big stuff. The harness should be MIT licensed. You should bring your own model. Nobody should be vendor-locked in.

They completely disagree on how much harness should exist.

OpenCode's philosophy is that the harness is a real product. Ship plan mode, permissions, LSP integration, MCP, a desktop app, a web UI, IDE support, GitHub actions. Give people everything Claude Code has, minus the lock in.

Pi's bet is that most of that is overhead (not a feature, what OpenCode believes). Frontier models have been RL trained so hard on coding that they already know what a coding agent is. Every tool schema and every injected instruction is context you pay for on every request. So the harness should be close to zero, and you add back only what you personally need.

The disler comparison has the cleanest one liner on this.

"Pi gives you control at the runtime level. OpenCode gives you control at the configuration level."

And yes, there is real beef here. Zechner has publicly called out OpenCode's compaction for pruning tool results and breaking prompt caching.

He is not a fan of their Vercel AI SDK dependency either. More on that below.

2. Agent Architecture: Pi vs OpenCode

Underneath, both use the same basic loop: read the task, call the model, run tools, feed results back, repeat.

The architecture around that loop is where they differ.

OpenCode: client + server

OpenCode splits into a persistent local server and separate clients.

The server handles:

agent logic
LLM calls
tool execution
SQLite session storage
MCP servers

The TUI, desktop app, web UI, and IDE integrations all talk to that same server.

This makes sessions more safe to terminal crashes or SSH drops, and the same server can also run headlessly in CI. The downside is a heavier footprint, with people reporting 1GB+ RAM usage for the TUI setup.

You read it right. Read more about it here: OpenCode is pretty resource inefficient

Pi: single process

Pi keeps everything in one process:

provider layer
agent loop
CLI

It is all TypeScript and small enough to understand without going through a huge codebase.

Pi supports four run modes:

interactive TUI
print mode for scripts
RPC over stdin/stdout
SDK mode for embedding

That SDK is also how OpenClaw embeds Pi.

OpenCode's way of LSP

I'd call it a star feature of OpenCode, the way it handles LSP (Language Server Protocol).

OpenCode can run real language servers such as gopls and typescript-language-server and feed diagnostics back into the agent after edits.

So the flow looks like this:

model edits code → LSP reports an error → model sees it → model fixes it

That gives OpenCode fast compiler and type feedback without needing to run the full test every time.

Pi does not have this out of the box.

Pi's Session trees

Pi way of handling sessions is kinda interesting.

Pi stores sessions as JSONL, with every entry pointing to a parent. That means the session history is a tree instead of one linear log.

A → B → C
     ↘ D → E
     ↘ F → G

You can:

use /tree to jump back to an earlier point
use /fork to branch from any past message
keep multiple debugging paths from the same point

OpenCode uses a linear history with /undo and /redo, which is simpler but less flexible.

Context compaction

OpenCode uses a hidden system agent to summarize the conversation when context gets tight, then hides older messages from the active context.

The criticism is that this can remove useful tool outputs and hurt prompt caching.

Pi also supports compaction, but the implementation is replaceable through extensions, which fits its more hackable architecture.

3. The benchmark. Same model, different harness

Now for the part I was most curious about.

We ran 30 hard agentic tool-use tasks against real apps. Every harness used the same model, DeepSeek V4 Pro (0813) at max reasoning, through our hosted MCP router.

Same model. Same tasks. Same tools. Same verifier checks.

So whatever gap shows up here is mostly the harness.

Here is the full result.

Harness (DeepSeek V4 Pro, hard 30 set)	Tasks passed	Cost per success	Cost per shared success	Avg tokens per task	Avg turns
Pi Agent	21/30 (70%)	$0.078	$0.031	924,990	16.3
Codex	20/30 (66.7%)	n/a*	$0.031	383,722	n/a
DeepSeek Harness	20/30 (66.7%)	$0.076	$0.028	88,562	0.9
OpenCode	19/30 (63.3%)	$0.119	$0.032	710,140	13.1
Claude Code	19/30 (63.3%)	n/a*	$0.074	649,900	12.1
Hermes Agent	18/30 (60%)	n/a*	$0.037	113,894	6.5

ℹ️ *Not comparable because cost measure was incomplete for some cells. Shared success only counts tasks every harness passed.

Pi actually topped the board

Pi finished first with 21/30 tasks passed, while OpenCode landed at 19/30.

It was also cheaper across the full run.

Pi spent $1.64
OpenCode spent $2.25
Pi cost per success was $0.078
OpenCode cost per success was $0.119

That last number looks rough for OpenCode, but there is a catch.

On tasks both harnesses actually passed, they were basically tied. Pi came in at $0.031 per shared success, while OpenCode was $0.032.

So OpenCode was not really more expensive when it worked. It just failed more tasks, and failed runs still burn tokens.

Pi was also painfully slow

Pi had the best pass rate, but it was also the slowest harness in the entire run.

Pi median time was 362.9 seconds
OpenCode median time was 280.6 seconds
Claude Code median time was 181.8 seconds

Pi also averaged 16.3 turns per task, compared with OpenCode's 13.1, and pushed more raw tokens than anyone else.

The difference is that its fixed overhead stays tiny.

The system prompt plus tool definitions stay under 1,000 tokens, and the prefix barely changes between turns. That makes a lot of that repeated context land as cache hits.

The task-by-task split is more interesting

Task	Pi	OpenCode
CRM migration archive	✅ 10/10	❌ 3/10
Slack action items	✅ 5/5	❌ 2/5
Vendor directory	✅ 11/13 checks	❌ 7/12
Handover audit	✅ 11/11	❌ 10/11
PagerDuty timeline audit	✅ 10/10	❌ 9/10
Renewal screen	❌ 5/11	✅ 11/11
Sponsorship screen	❌ 4/11	✅ 11/12
Sheets to Airtable tracker	❌ 8/10	✅ 10/10

Pi did better on the audit and census-style tasks, where patiently reading everything and checking every detail matters.

OpenCode did better on the longer screening and multi-step workflows.

Which is kinda perfect.

Pi wins by being small. OpenCode wins when the workflow benefits from more stuff around the model.

After all this non-sense, the eval ended up being spot on for both of them. 🤧

We did a complete breakdown in this tweet. You might want to check it out.

// Detect dark theme var iframe = document.getElementById('tweet-2090069397050097864-193'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2090069397050097864&theme=dark" }

4. Token overhead

This is basically the argument both tools were built around, so let's put some numbers next to it.

The most cited comparison here is the Systima study from July, which also ended up doing 700+ points on Hacker News.

They put a logging proxy between the harness and the model, then measured how much context gets sent before the model even sees your prompt.

Harness	Fixed overhead per request
Claude Code (measured by Systima)	~32,800 tokens
OpenCode (measured by Systima)	~6,900 tokens
Pi	Under 1,000 tokens

OpenCode already looks pretty good here. Around 4.7x less overhead than Claude Code.

Then Pi shows up with under 1,000 tokens, including the tool definitions.

That makes it roughly 7x leaner than OpenCode.

💁 Everyone is somebody's bloatware. 🗿

Matei Zaharia said on X the reason was pretty simple. Pi had the same success rate with 2x less cost.

Our eval points in the same direction too. Pi had the best pass rate and the lowest total spend.

But there is an important catch here.

OpenCode has had some pretty ugly token cases too. There is a GitHub issue where asking something as basic as "what's the tech stack in this project" burned 13,088 tokens on a fresh project.

And Pi is not crazy cheap in every sense either. In our eval, it actually pushed the highest raw token count of all six harnesses, mostly because it took more turns to finish tasks.

So yeah, tiny fixed overhead helps a lot. But lean overhead does not always mean lean sessions.

Worth keeping that distinction in mind.

5. Pricing and the ban

Both tools are free and MIT licensed. Fork them, modify them, ship them.

The difference starts after install.

Pi

BYOK for pretty much any provider
local models through Ollama, vLLM, or llama.cpp
OAuth login where providers still allow it
no paid CLI tier at all

💁 Pi is basically just the harness. You bring the model and pay whoever is serving it.

OpenCode

OpenCode does the same BYOK thing, but it also has its own model options.

BYOK where the provider bills you directly
Zen with curated models sold on a pay-as-you-go basis
Go with open coding models bundled into a cheap monthly plan

Go starts at $5 for the first month, then $10/month, with models like GLM, Kimi K3, Qwen, MiniMax, and DeepSeek.

⚠️ Double check the pricing. OpenCode changes this stuff pretty often.

And honestly, that Go plan is hard to ignore.

For $10/month, it is probably one of the cheapest legit ways to run capable open models through a coding agent right now.

Pi does not really have an equivalent. Its version of "cheap" is running local models, which is useful, but it is not the same thing.

Then Anthropic happened

In January 2026, Anthropic started rejecting Claude Pro and Max OAuth tokens when they were used outside Claude Code.

The error was pretty clear. The credential was "only authorized for use with Claude Code."

OpenCode had been using those subscription tokens by mimicking the Claude Code client, so this hit them directly.

Then came the legal requests, the Claude login code got removed, and we got the now-famous commit message from the intro.

George Hotz called the move "a huge mistake."

OpenAI reportedly went in the other direction and opened Codex subscription access to third-party harnesses, including OpenCode and Pi.

Lol.

The important part for this comparison is pretty simple now.

If you want Claude in either Pi or OpenCode, you are basically paying API rates. OpenCode also gives you Zen as another route.

6. Extensibility

Both are highly customizable, but the extension model is very different.

OpenCode

OpenCode is mostly config driven through opencode.json, plugins, agents, skills, and MCP.

You can add:

custom agents with their own model, prompt, permissions, and step limits
JS or TS plugins with lifecycle hooks like tool.execute.before
custom tools
local and remote MCP servers
skills from .opencode/skills and .claude/skills

{
  "permission": {
    "bash": {
      "git *": "allow",
      "rm *": "deny",
      "*": "ask"
    }
  }
}

Pi

Pi extensions are TypeScript modules loaded directly into the single agent process.

They can:

rewrite or block tool calls
replace built-in tools
filter history
replace compaction
persist custom session state
add TUI components
probably many more...

That gives Pi much deeper runtime control than OpenCode.

ℹ️ OpenCode plugins extend the harness. Pi extensions can modify the harness itself.

The tradeoff is security. Pi extensions run in process with full system access, so they need more trust than OpenCode's plugin model.

7. Permissions and safety

This is probably the biggest gap between the two.

OpenCode has a proper permission system. Every tool can be set to allow, ask, or deny, with glob patterns and per-agent overrides. The defaults are decent too. .env reads are denied by default, and the Plan agent asks before edits or bash commands.

What it does not have is real sandboxing. There is no container or OS-level isolation built in.

Pi goes the other way and basically ships no permission layer at all. From the first prompt, the agent runs with your full user privileges. 🥶

That is intentional. Zechner's say is that once an agent can write files and execute code, permission prompts only get you so far. Pi instead expects you to isolate the whole process using Docker, a microVM like Gondolin, or a policy sandbox like OpenShell.

💁 There is also an optional damage-control.ts extension that checks bash commands against YAML rules. Helps a bit.

So the actual difference is pretty simple.

ℹ️ OpenCode gives you tool-level guardrails. Pi expects you to handle isolation outside the agent.

Which one should you pick?

Pick OpenCode if

You want something close to a full Claude Code replacement without building half the setup yourself
You want plan mode, permissions, MCP, undo and redo, and LSP support out of the box
You want the $10 Go plan for cheap access to capable open models
You work in typed languages and want LSP diagnostics fed straight back into the agent
You are setting this up for a team and actually need proper allow, ask, and deny controls

Pick Pi if

You care about token cost at scale. In our eval, the same model passed more tasks and cost less with Pi
You want a harness small enough that you can actually read and understand the whole thing
You like branching while debugging. Pi's session trees are really good for that
You run local or smaller models where the difference between ~1,000 and ~6,900 tokens of fixed overhead actually matters
You would rather change the harness in TypeScript than spend time configuring someone else's abstraction

The benchmark kinda gives away the answer anyway.

Pi did better on the careful audit-style tasks. OpenCode did better on longer, more structured workflows.

So there probably isn't one winner for everything.

Both are free. Try both and keep the one that fits how you work. ✌️

Conclusion

After all that, I still don’t think there’s one obvious winner here. The scorecard ends 6-6.

OpenCode wins on adoption, features, and ease of use. Pi won our benchmark with 21/30 tasks vs 19/30, lower total spend, and way less fixed overhead.

The bigger difference is still the same one we started with:

OpenCode gives you configuration control. Pi gives you runtime control.

If you just want to install something and start working, pick OpenCode.

If you want to understand and modify the harness itself, pick Pi.

And if cost per successful task starts mattering at scale, keep an eye on Pi. 🤷‍♂️

Pi Agent vs Claude Code After 100 Hours of Real Use 🔥

Shrijal Acharya — Tue, 11 Aug 2026 13:36:48 +0000

While researching for this post, I found out something funny. Pi, the most interesting coding agent of 2026, was built by a guy who got fed up with Claude Code. 🥶

Mario Zechner, the creator of libGDX, was a hardcore Claude Code user. He built cchistory just to track its system prompt changes, and even patched the binary to add features Anthropic had not shipped.

Then he quit and wrote his own agent. Four tools, a tiny system prompt, no MCP, no permissions. He named it Pi so nobody could Google it. What a Chad! 🗿

That small project now has 85k+ GitHub stars, powers OpenClaw, and outscored Claude Code on our own tool use benchmark at a fifth of the cost.

So I put the two side by side. The default everyone uses, against the minimal harness built as a rejection of it.

Let's go!

TL;DR

Category	Claude Code	Pi	Winner
Tool price	Needs a paid plan or API key	Free, MIT licensed	Pi
Real tool use (our eval)	16/30 tasks passed, $0.195 per success	20/30 tasks passed, $0.028 per success	Pi
Predictable billing	$20/month flat on Pro	Pay per token, bills can spike	Claude Code
Model support	Claude models only but it’s built for it	20+ providers, 300+ models, mid session switching	Pi
Out of the box features	Sub agents, teams, plan mode, MCP, skills, checkpoints	Four tools and a text box	Claude Code
Context discipline	Auto compaction, silent context management	You see and control every token	Pi
Extensibility depth	Shell hooks, MCP, skills, plugins	TypeScript extensions running inside the agent process	Pi
Multi agent workflows	Native sub agents and Agent Teams	Spawn pi via bash, or install a community extension	Claude Code
Permissions and guardrails	Deny by default, five modes, sandboxing	Full system access from the first prompt	Claude Code
Surfaces	Terminal, VS Code, JetBrains, desktop, web, mobile, Slack	Terminal	Claude Code
Session model	Linear conversation log	Branchable session trees with fork and rewind	Pi

In Summary:

Scorecard: Pi 6, Claude Code 5.

Pi is the cheaper, more flexible, and more transparent harness, especially if you pay per token, switch models, or want full control. Claude Code remains the better daily driver for most people because it ships with stronger guardrails, predictable subscription pricing, and a polished feature set out of the box. The practical answer is to use Claude Code for everyday work and Pi for custom workflows, local models, and cost-sensitive tasks.

And here is the twist. The scorecard says Pi, but almost everyone who reviews both, including the biggest Pi fans, still runs Claude Code as their daily driver. I will explain why is that in a minute.

What is Pi?

ℹ️ A minimal, open source terminal coding agent that ships four tools and expects you to build the rest yourself.

Pi is the coding agent Mario Zechner released in late 2025, now maintained under Earendil. Zechner joined Earendil in April 2026 in a blog post he titled "I've sold out," and the core stayed MIT licensed.

The whole product fits in literally one sentence. The model gets read, write, edit, and bash, a system prompt of a few hundred tokens, and nothing else.

His reasoning: frontier models have been RL trained so hard on coding agent behavior that they already know what a coding agent is. You do not need 10,000 tokens of instructions to remind them. Need ripgrep? The model runs rg through bash. Need GitHub? It runs gh. Need a browser? Ask Pi to write itself a browser tool.

💁 If you're on the same philosophy, this might be worth reading: Top 10 CLI Tools to Level-Up Claude Code. No MCPs, nothing. Plain raw Bash tools gets most of the work done.

That last part is the actual product. Pi extensions are TypeScript files that run inside the same process as the agent loop, and Pi can read its own source and docs, so the standard workflow when you need a missing feature is to ask Pi to build it.

Everything Zechner refused to ship, the community shipped as extensions within weeks. Todo lists, plan mode, sub agents, permission gates, even an MCP adapter.

What is Claude Code

ℹ️ Anthropic's batteries included coding agent, and the default the entire agent category gets measured against

You know this one. Claude Code started as Boris Cherny's internal AppleScript toy at Anthropic in September 2024, hit research preview in February 2025, and became the fastest scaling developer product anyone has ever measured.

It ships everything: 10+ built in tools, sub agents, Agent Teams, plan mode, MCP as both client and server, Agent Skills, plugins, hooks, checkpoints with rewind, a permission system with five modes, and it runs in the terminal, VS Code, JetBrains, a desktop app, the browser, and your phone.

More than 80% of Anthropic's own engineers use it daily, and around "90% of Claude Code's code is now written by Claude Code". 🥴

1. Philosophy: Pi vs Claude Code

This comparison only makes sense once you understand that Pi exists because of Claude Code.

Zechner was an early Claude Code user. His complaint was never that it was bad. His complaint was that it kept changing underneath him. The system prompt and tool definitions shifted on every release, which broke his workflows and changed model behavior.

So the two tools represent opposite answers to the same question: how much harness does a frontier model actually need?

Anthropic's answer is interesting because it is drifting toward Pi's. Boris Cherny's team deletes harness code every time a new model ships, and in July 2026 Anthropic removed over 80% of Claude Code's system prompt for the Claude 5 generation models with no measurable loss on their coding evals. The bet behind Claude Code is that Anthropic trains the model and the harness together, so scaffolding today can become model behavior tomorrow.

Pi's answer is that the scaffolding was never load bearing in the first place.

💁 Anthropic quietly deleting most of its own system prompt is about the strongest validation of Pi's thesis you could ask for. Zechner just got there a year early.

2. Agent Architecture: Claude Code vs Pi

Both run the same loop underneath. Read the task, call the model, execute tool calls, feed results back, repeat until the model stops asking for tools.

The difference is everything wrapped around that loop.

Claude Code gives the model Read, Write, Edit, Bash, Glob, Grep, WebSearch, WebFetch, NotebookEdit, TodoWrite, there's a lot.... and the Task tool that spawns typed sub agents with their own isolated context windows. Plan mode forces a read only explore and plan phase before any file gets touched.
Pi gives the model four tools, plus optional read only grep, find, and ls behind a -tools flag. There is no plan mode, but pi --tools read,grep,find,ls gets you a read only session in one line. There are no sub agents, and they say that if you want one, tell Pi to run itself through bash, ideally in tmux so you can actually watch it work.

The system prompt gap is the main highlight:

	System prompt	Built in tools
Claude Code (pre Claude 5 era)	~10,000 to 14,000 tokens by community measurement	10+
Claude Code (current)	Cut by over 80% for Claude 5 gen models	10+
Pi	Under 1,000 tokens including tool definitions	4

One less obvious architectural difference: Pi stores sessions as trees, not logs. Every message carries a parent id, so under the hood this is basically a linked list that learned to branch.

A real use case for linked lists, in the wild. All that leetcode grind had something useful. 🤡

So /tree lets you jump back to any earlier point in the conversation and branch from there, and /fork spins a new session off any past message. Claude Code got checkpoints and rewind in 2.0, which covers the common case, but Pi's tree model is the better design.

3. The benchmark: same model, different harness

We ran 30 hard agentic tool use tasks against real apps, eight harnesses, every single one wired to the same model, DeepSeek V4 Flash, through our hosted MCP router, with a 900 second ceiling per task.

Same model, same tasks, same tools. The only thing that changes is the harness, so whatever gap shows up is the wrapper, not the model.

Here is how our two contenders did:

Harness (DeepSeek V4 Flash, hard 30 set)	Tasks passed	Median time	Avg tokens per task	Total cost	Cost per success
Claude Code	16/30 (53.3%)	122.7s	741,659	$3.12	$0.195
Pi	20/30 (66.7%)	132.2s	558,885	$0.56	$0.028

Pi passed more tasks than any of the eight harnesses we tested, and did it at the lowest cost of the whole bunch. Cost per successful task is the brutal column: 3 cents against 19.

Claude Code posted the fastest median time. It just burned 741,659 tokens per task getting there, against Pi's 558,885. The overhead is the story, again.

And yes, that is Claude Code running DeepSeek. That is the whole point of the setup. Only the harness changes.

For the curious, Codex ran the same and landed at 16/30 for $1.29 total. Right between the two.

⚠️ NOTE: Pi ran on high reasoning instead of max, and 24 of its 30 trials went through the official DeepSeek API instead of OpenRouter. Make sure to read the exact numbers with that in mind.

4. Pricing: Claude Code vs Pi

Claude Code

Pro, $20/month or $17/month annually, includes Claude Code
Max 5x at $100/month, Max 20x at $200/month

⚠️ Guys, make sure to double check the pricing. It changes frequently.

Pi

The tool costs nothing. MIT license, fork it, ship it
You bring your own API keys for any provider, or run local models for free through Ollama, vLLM, or llama.cpp
OAuth login for subscriptions where providers allow it

On paper Pi wins on price and Claude Code wins on predictability. A $20 flat rate is genuinely hard to beat for a solo dev using Claude models all day, and per token bills on heavy Pi usage can hit hundreds of dollars a month. I usually prefer the subscription system.

Anthropic locked its subscription OAuth tokens to Claude Code and Claude.ai only. Third party harnesses got cut off, starting with a silent server side block in January, formalized in the docs by February, fully enforced by April. OpenClaw, OpenCode, Cline, and yes, Pi, all lost the ability to run on your Claude Pro or Max subscription.

So the practical situation today: running Claude models through Pi means API rates. Running them through Claude Code means your cheap subscription.

DHH called the move "very customer hostile," Lol.

OpenAI's Codex team publicly went the opposite direction and encouraged third party harness usage.

💁 If your workflow is Claude models all day every day, the subscription math alone mostly settles this comparison. Pi's cost advantage is real on API keys and irrelevant on a Max plan you already pay for.

5. Models: one provider vs all of them

Claude Code runs Claude. Opus, Sonnet, and Haiku tiers, switchable with /model, with thinking effort levels, tuned end to end because Anthropic controls the whole stack.

That vertical integration is why the model feels so at home in the harness. It was trained in it.

Pi runs everything. The pi-ai layer normalizes four API shapes across 20+ providers, from Anthropic and OpenAI to Groq, Cerebras, Mistral, OpenRouter, and local models. Over 300 models total. You can switch models mid session with Ctrl+P, and Pi converts thinking traces between providers.

Claude Code's giant prompt overhead makes it a poor fit for small local models, which is exactly why Simon Willison flagged Pi as the lightweight option for that use case.

6. Extensibility

Both tools are deeply customizable.

Claude Code's model is external and config driven. Hooks are shell scripts that fire on 14 lifecycle events and talk JSON over stdin. MCP servers are separate processes speaking a protocol. Skills are markdown instruction packs with lazy loading. Plugins bundle all of it into installable packages with official and community marketplaces behind a single /plugin command.

{
  "permissions": {
    "allow": ["Bash(npm run test *)", "Bash(git status *)"],
    "deny": ["Read(./.env)", "Bash(curl *)"]
  }
}

Pi's model is internal. Extensions are TypeScript modules loaded into the same runtime as the agent loop, hooking 25 event types. They can block or rewrite tool calls before they run, override the built in tools entirely, inject context, replace the compaction logic, persist custom state into the session file, and draw their own TUI widgets. Claude Code's hooks can approve or reject what the agent does. Pi's extensions can change what the agent is.

The tradeoff is trust. Claude Code's marketplace plugins go through some checks and run outside the core process. Pi packages run with full system access, and its own docs tell you to review them before installing.

The sharpest criticism of Pi's ecosystem, from thevinter's "Bad Vibes From Pi", lands exactly here: a lot of those community extensions are themselves vibecoded, so that's something to watch-out for.

7. Permissions and safety

This is the widest gap between the two tools, and neither side considers it a flaw.

Claude Code is deny by default. Every risky action asks first, and you can loosen from there through five permission modes, allow and deny rules, and OS level sandboxing for bash.

When Claude Code has security incidents, and it has had real ones with CVEs attached, they get found, patched, and written up because millions of people are using it.

Pi has no permission system. First prompt, full user privileges, go.

Zechner's argument is that agent security is mostly "theater" 🫩, because once a tool can write code and run code the game is already over. Pi's answer to safety is isolation instead of prompts: run it in Docker, in a micro VM, or behind a sandbox, and there is an opt in extension that checks bash commands against YAML rules if you want a middle ground.

One of them requires you to have read the docs, and the other protects the "intern" who did not.

There isn't much to discuss here. You probably already know how things work by now...

What the Internet thinks

Maybe worth seeing what the internet thinks of Pi and Claude Code by now?

It's more of what we are discussing so far. The famous endorsements is around Pi, the daily usage clusters around Claude Code, and both facts are true.

Let me show you the Pi love first, because it is a lot.

Peter Steinberger built OpenClaw on top of Pi and quote tweeted Zechner with "Mario is the real one we don't deserve." Armin Ronacher wrote that Pi is the coding agent he uses "almost exclusively".

Theo from t3.gg did a whole episode about getting Pi pilled. His take was simple: "The more minimal, the better." Wes Bos and Scott Tolinski spent a Syntax episode arguing Claude Code is overkill for most work.

My favorite line comes from IndyDevDan, who calls Pi the only real Claude Code competitor: "Claude Code is the starter pack. Pi is the endgame."

About 80% of his work still goes through Claude Code. Only 20% goes through Pi. The endgame guy is still playing the starter pack. The internet is wild!

Claude Code does not need loud fans anyway. It has $2.5B in revenue (might increase, researched through Claude itself), 141k GitHub stars, most of the enterprise market, and Microsoft reportedly using it inside their own teams. The numbers are just enough, lol. 🤧

Which one should you pick?

Pick Claude Code if:

You want something that works at full power
You live on Claude models, because the $20 to $200 subscriptions are the cheapest way to run them and Pi is locked out of that pricing
You are deploying to a team and need permissions, SSO, spend caps, and audit trails
You want sub agents, plan mode, and multi surface access without building anything

Pick Pi if:

You pay per token at scale, since less context per task is real money, a fifth of the cost in our eval
You want to mix providers, run open models, or go fully local
You want to read, own, and modify every part of your harness, MIT license and all
You are the kind of engineer who would rather build the exact workflow you want than configure someone else's

Do what most reviewers actually do: run both. Claude Code as the primary builder on your subscription, Pi as the cheap second opinion, the local model runner, and the place where your custom workflows live. The two tools are converging anyway.

Claude Code keeps deleting scaffolding, and the Pi community keeps rebuilding Claude Code's features as extensions. 🤡

Conclusion

The scorecard reads Pi 6, Claude Code 5. And honestly, Pi deserves those wins. It is cheaper per task, open source, works with any model. What more do you want?

But here is the thing. When I sit down with a bug and just want it fixed, I open Claude Code. It works right away, the guardrails just works, the $20 plan keeps the bill consistent, and the model just feels at home in it.

No thinking about the tool itself.

That is the whole story really. Even the biggest Pi fans ship most of their work through Claude Code. It's a secret. 🤫

Zechner was right that these agents need way less scaffolding than we thought. Anthropic deleting 80% of its own system prompt basically admitted it. But being right and being the tool everyone opens every day are two different things, and Claude Code still owns the second one for me.

So use Claude Code for the day job. Keep Pi around for when you want to tinker, go local, or save some money. ✌️

Shrijal AcharyaFollow

SDE • GOLD @Microsoft Student Ambassador • Prev Lead Collab and Dev-Team Lead @oppiaorg • Mail for collaboration

How to build and deploy an MCP server to production (2026-07-28 spec)

Shrijal Acharya — Wed, 05 Aug 2026 12:13:02 +0000

MCP just had its biggest release since launch.

On July 28, the maintainers shipped the 2026-07-28 spec, and it changes how MCP servers work at a pretty fundamental level. The handshake is gone. Sessions are gone. Three long-standing features are deprecated.

The maintainers themselves called it the most substantial change since authorization was added. Their words, not mine.

Sounds scary. But it actually makes MCP servers much easier to deploy. And what am I here for? I'm here to help you build and deploy one.

Your MCP server is now just a regular stateless HTTP service. Round-robin load balancing, autoscaling, and caching all work. No sticky sessions or shared session state.

In this guide, we'll build a small MCP server on the new spec, connect a client to it, see every headline feature actually running, and then deploy it to Cloudflare Workers. For free.

ℹ️ All the code here uses the new TypeScript SDK v2, released alongside the spec. If you're on the old @modelcontextprotocol/sdk package, that's v1 now.

What's Covered

What actually changed in the 2026-07-28 spec in short
Building an MCP server with the new SDK v2
Stateless core in action
MRTR: how a tool requests user confirmation without holding a stream open
A graceful fallback for clients that don't speak MRTR yet (there are many)
Cacheable tool lists with ttlMs
Testing it with a client and raw curl
Deploying it to Cloudflare Workers on the free plan

Changes in the new MCP Spec (2026-07-28)

Quick rundown of what's new. If you want the full changelog, it's on the official spec site.

The handshake is gone

The initialize / initialized exchange and the Mcp-Session-Id header are officially retired.

Every request is now self-describing. It carries its own protocol version, client identity, and capabilities in _meta. Any request can land on any server instance behind a plain load balancer. Such a relief!!

There's an optional server/discover RPC if a client wants capabilities up front. But it's optional. One bare POST is a complete conversation now.

Multi Round-Trip Requests (MRTR)

This one is my favorite.

Before, if a tool needed something from the user mid-call, such as confirmation or a missing parameter, the server had to push an elicitation/create request back over a held-open stream. That meant you needed a held-open stream, which was bad for stateless deployments.

MRTR flips it. The server returns resultType: "input_required" with the questions it needs answered, and closes the connection. The client collects the answers and retries the original call with them attached, plus an opaque requestState token so the server knows where it left off.

No open streams. No sessions. Interactive tools on fully stateless infra.

Header-based routing

Requests now carry Mcp-Method and Mcp-Name HTTP headers. Your gateway, rate limiter, or WAF can route and meter on headers without parsing JSON bodies.

Cacheable list results

tools/list, prompts/list, resources/list, and resources/read responses now carry ttlMs and cacheScope fields, modeled on HTTP's Cache-Control. Clients cache your tool catalog instead of re-fetching it every time they connect.

Extensions framework + deprecations

Tasks moved out of the experimental core into an official extension (io.modelcontextprotocol/tasks). MCP Apps and Enterprise Managed Authorization live there too. You can build your own extensions as well.

And the deprecations:

Roots, Sampling, and Logging are deprecated. They keep working for at least 12 months, but new implementations shouldn't use them.
The legacy HTTP+SSE transport is deprecated with a year-long offramp.
Dynamic Client Registration is deprecated in favor of Client ID Metadata Documents (CIMD).

There's also a formal deprecation policy now: a 12-month minimum window for anything marked deprecated. So you get to plan upgrades, which is noicee!

The SDK Split

One more thing before we build: the TypeScript SDK is no longer one package.

v2 splits it into @modelcontextprotocol/server, @modelcontextprotocol/client, and thin framework adapters (@modelcontextprotocol/hono, express, fastify, node).

Building an MCP Server

Finally, we're onto the build. We will build a quick tiny deploy bot over MCP.

It has three tools:

deploy asks the user for confirmation before deploying (MRTR in action).
list_deployments reads back the deployment history
server_stats proves a fresh server instance handled every request

Here's the trick that pays off at deploy time: all the MCP logic lives in one platform-neutral file (bot.ts), and each platform gets a tiny entry file. Node gets server.ts. Cloudflare gets worker.ts. Both are about ten lines. An MCP server on the new spec is just a fetch handler; the platform is a serving shim.

You'll understand everything along the way.

Step 1: Install the SDK v2

Run the following command:

mkdir updated-mcp-spec-bot && cd updated-mcp-spec-bot
npm init -y && npm pkg set type=module
npm install @modelcontextprotocol/server @modelcontextprotocol/client \
  @modelcontextprotocol/hono @hono/node-server hono zod tsx

ℹ️ On TypeScript 6+, add "types": ["node"] to your tsconfig compilerOptions after installing @types/node. TS 6 no longer auto-includes @types/*, and you'll get Cannot find name 'process' errors without it. Ask me how I know. 😴

Step 2: The server logic

Create bot.ts. This is the whole MCP server, with zero platform code in it:

// 👇 bot.ts

import type {
  CallToolResult,
  InputRequiredResult,
} from "@modelcontextprotocol/server";
import {
  acceptedContent,
  CLIENT_CAPABILITIES_META_KEY,
  createRequestStateCodec,
  inputRequired,
  McpServer,
} from "@modelcontextprotocol/server";
import * as z from "zod/v4";

const deployments: { env: string; at: string }[] = [];
let requestsServed = 0;

type DeployState = { step: "confirm"; env: string };

// set STATE_KEY in production so all instances share the secret
// lazy init: Workers forbids generating random values at module scope
let codec: ReturnType<typeof createRequestStateCodec<DeployState>> | undefined;
function stateCodec() {
  if (!codec) {
    const key = globalThis.process?.env?.STATE_KEY;
    codec = createRequestStateCodec<DeployState>({
      key: key
        ? new TextEncoder().encode(key)
        : crypto.getRandomValues(new Uint8Array(32)),
      ttlSeconds: 600,
    });
  }
  return codec;
}

const CONFIRM_SCHEMA = {
  type: "object" as const,
  properties: { confirm: { type: "boolean" as const } },
  required: ["confirm"],
};

// runs per request, keep it cheap
export function buildServer(): McpServer {
  requestsServed++;

  const server = new McpServer(
    { name: "updated-mcp-spec-bot", version: "1.0.0" },
    {
      cacheHints: {
        "tools/list": { ttlMs: 30_000, cacheScope: "public" },
      },
      requestState: { verify: (...a) => stateCodec().verify(...a) },
    },
  );

  server.registerTool(
    "list_deployments",
    {
      title: "List deployments",
      description: "List all deployments recorded by this server.",
    },
    async (): Promise<CallToolResult> => ({
      content: [
        {
          type: "text",
          text: deployments.length
            ? deployments.map((d) => `${d.env} @ ${d.at}`).join("\n")
            : "No deployments yet.",
        },
      ],
    }),
  );

  server.registerTool(
    "server_stats",
    {
      title: "Server stats",
      description:
        "How many requests this process served, each on a fresh server instance.",
    },
    async (): Promise<CallToolResult> => ({
      content: [
        {
          type: "text",
          text: `pid=${globalThis.process?.pid ?? "edge"} requestsServed=${requestsServed}`,
        },
      ],
    }),
  );

  server.registerTool(
    "deploy",
    {
      title: "Deploy",
      description:
        "Deploy to an environment. Requires confirmation: interactive clients get a prompt, others must pass confirm: true.",
      inputSchema: z.object({
        env: z.enum(["staging", "prod"]).describe("Target environment"),
        confirm: z
          .boolean()
          .optional()
          .describe("Set true to confirm, only after asking the user"),
      }),
    },
    async (
      { env, confirm },
      ctx,
    ): Promise<CallToolResult | InputRequiredResult> => {
      const caps =
        (ctx.mcpReq.envelope as Record<string, unknown> | undefined)?.[
          CLIENT_CAPABILITIES_META_KEY
        ] ?? server.server.getClientCapabilities();
      const canElicit = Boolean(
        (caps as { elicitation?: unknown } | undefined)?.elicitation,
      );

      if (canElicit) {
        const state = ctx.mcpReq.requestState<DeployState>();
        const confirmed = acceptedContent<{ confirm: boolean }>(
          ctx.mcpReq.inputResponses,
          "confirm",
        );
        if (!state || !confirmed?.confirm) {
          return inputRequired({
            inputRequests: {
              confirm: inputRequired.elicit({
                message: `Deploy to ${env}? This will go live.`,
                requestedSchema: CONFIRM_SCHEMA,
              }),
            },
            requestState: await stateCodec().mint({ step: "confirm", env }),
          });
        }
        const record = { env: state.env, at: new Date().toISOString() };
        deployments.push(record);
        return {
          content: [
            { type: "text", text: `Deployed to ${record.env} at ${record.at}` },
          ],
        };
      }

      // fallback for clients without elicitation support
      if (confirm !== true) {
        return {
          content: [
            {
              type: "text",
              text: `Deploy to ${env} needs confirmation. Ask the user, then call deploy again with confirm: true.`,
            },
          ],
        };
      }
      const record = { env, at: new Date().toISOString() };
      deployments.push(record);
      return {
        content: [
          { type: "text", text: `Deployed to ${record.env} at ${record.at}` },
        ],
      };
    },
  );

  return server;
}

A few things worth explaining here:

buildServer() runs on every single request. Not once at startup. Every request gets a brand-new McpServer instance.

If that surprises you, I get it. It surprised me too. But this is literally the canonical pattern from the SDK's own examples, and it's the whole point of the release.

Construction is just object creation and a handler map, microseconds of work. There's no protocol state to preserve anymore, so there's nothing to keep alive.

Per-request server construction, per-process resources. App state (our deployments array, the state codec, your DB pool in real life) lives at module level. The server instance is disposable.

The deploy tool never blocks. When it needs confirmation, it returns inputRequired(...) and the request is over. Done. Connection closed. The requestState token is the only thing that survives between rounds, and it round-trips through the client.

This means the client could tamper with it. That's why we seal it with createRequestStateCodec, so tampered or expired state gets rejected with a wire-level error before our handler even runs.

Notice the codec is lazily created on first use instead of at module level. That looks like a pointless indirection on Node. It's not. Cloudflare Workers forbids generating random values in global scope, and this exact line is what lets the same file run on both platforms. Same story with the globalThis.process?. guards: Workers has no process global by default.

So the tool reads the client's declared capabilities from the per-request envelope (that's the CLIENT_CAPABILITIES_META_KEY lookup, with a legacy-connection fallback) and if:

Client supports elicitation then the full MRTR confirmation flow
Client doesn't then the tool accepts an optional confirm: true argument, and without it, it returns a plain instruction: "Ask the user, then call deploy again with confirm: true"

Step 3: The Node entry

Create server.ts. This is everything Node-specific:

// 👇 server.ts

import { serve } from "@hono/node-server";
import { createMcpHonoApp } from "@modelcontextprotocol/hono";
import { createMcpHandler } from "@modelcontextprotocol/server";
import { buildServer } from "./bot.js";

const handler = createMcpHandler(buildServer);

// in production set ALLOWED_HOSTS to your public domain
const allowedHosts = process.env.ALLOWED_HOSTS?.split(",").map((h) => h.trim());
const app = createMcpHonoApp(allowedHosts ? { allowedHosts } : {});
app.get("/healthz", (c) => c.text("ok"));
app.all("/mcp", (c) => handler.fetch(c.req.raw));

const port = Number(process.env.PORT ?? 3000);
const hostname = process.env.HOST ?? "127.0.0.1";
serve({ fetch: app.fetch, port, hostname }, () => {
  console.error(`updated-mcp-spec-bot listening on <http://$>{hostname}:${port}/mcp`);
});

That's it. createMcpHandler gives you a standard fetch-style handler, and Hono is just routing. createMcpHonoApp() validates Host/Origin headers (DNS rebinding protection) and only allows localhost out of the box, so the ALLOWED_HOSTS env var is there for when this runs behind a real domain.

Everything is env-driven (PORT, HOST, ALLOWED_HOSTS, STATE_KEY) because that's what a VM or a PaaS like Railway wants. We won't use this file for the Cloudflare deploy, but it's your path if you'd rather run this on Node anywhere.

Step 4: The client

Create client.ts:

// 👇 client.ts

import {
  Client,
  StreamableHTTPClientTransport,
} from "@modelcontextprotocol/client";

const url = process.env.MCP_URL ?? "<http://127.0.0.1:3000/mcp>";

const client = new Client(
  { name: "mcp-demooo-client", version: "1.0.0" },
  {
    capabilities: { elicitation: { form: {} } },
    versionNegotiation: { mode: "auto" }, // use 2026-07-28 when the server does
  },
);

// The elicitation handler: in a real app this renders a confirm dialog.
// Here we auto-accept and log what the server asked.
client.setRequestHandler("elicitation/create", async (request) => {
  const { message } = request.params as { message: string };
  console.log(`\n[elicitation] server asks: "${message}" -> answering yes`);
  return { action: "accept", content: { confirm: true } };
});

await client.connect(new StreamableHTTPClientTransport(new URL(url)));
console.log(
  `connected, negotiated protocol: ${client.getNegotiatedProtocolVersion()}`,
);

const tools = await client.listTools();
const { ttlMs, cacheScope } = tools as { ttlMs?: number; cacheScope?: string };
console.log(`tools/list: ${tools.tools.map((t) => t.name).join(", ")}`);
console.log(`cache hints: ttlMs=${ttlMs} cacheScope=${cacheScope}`);

await client.listTools();
console.log("second listTools served from cache");

const before = await client.callTool({ name: "list_deployments" });
console.log(
  `list_deployments: ${(before.content[0] as { text: string }).text}`,
);

const result = await client.callTool({
  name: "deploy",
  arguments: { env: "prod" },
});
console.log(`deploy: ${(result.content[0] as { text: string }).text}`);

const stats = await client.callTool({ name: "server_stats" });
console.log(`server_stats: ${(stats.content[0] as { text: string }).text}`);

const after = await client.callTool({ name: "list_deployments" });
console.log(`list_deployments: ${(after.content[0] as { text: string }).text}`);

await client.close();

⚠️ Don't miss versionNegotiation: { mode: 'auto' }. Without it, the client negotiates the legacy 2025-11-25 protocol and the MRTR flow fails. This took me half an hour to debug.

Notice the elicitation handler is a completely normal elicitation/create handler, the same one you'd write for the old flow. The SDK's auto-fulfillment engine routes the embedded MRTR request through it and retries the tool call for you. Your code doesn't even see the round trip.

Step 5: Run it

In two terminals (better with tmux), run the following:

In the first terminal:

npx tsx server.ts

And in the other:

npx tsx client.ts

This is the kinda output you'd get:

connected, negotiated protocol: 2026-07-28

tools/list: list_deployments, server_stats, deploy
cache hints: ttlMs=30000 cacheScope=public

second listTools served from cache
list_deployments: No deployments yet.

[elicitation] server asks: "Deploy to prod? This will go live." -> answering yes
deploy: Deployed to prod at 2026-08-01T08:02:45.601Z

server_stats: pid=159984 requestsServed=6
list_deployments: prod @ 2026-08-01T08:02:45.601Z

Every line here demonstrates a spec feature, and I designed it that way:

2026-07-28: we're on the new protocol, not the legacy fallback
ttlMs=30000 + served from cache: the second listTools() never touched the network
The elicitation line, then the deploy: that was two tools/call POSTs. First one returned input_required and closed. Second had the answer plus the sealed requestState. No stream was ever held open.
requestsServed=6: six requests, six fresh server instances, one process. Under a load balancer, those six could've hit six different machines. How cool is that?
The final list_deployments: app state survived even though protocol state didn't.

And the math is here: 4 tool calls, plus 2 listTools() where only 1 hit the wire, plus 1 extra round for the MRTR retry = 6 server builds.

Step 6: Look at the raw wire

Let's see the "no handshake" thing. One bare curl, with no initialization:

Run the following command:

curl -s -X POST http://127.0.0.1:3000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "MCP-Protocol-Version: 2026-07-28" \
  -H "Mcp-Method: tools/call" -H "Mcp-Name: list_deployments" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"list_deployments","arguments":{},"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28","io.modelcontextprotocol/clientInfo":{"name":"curl","version":"1.0"},"io.modelcontextprotocol/clientCapabilities":{}}}}'

btw, this curl command was suggested by Claude.

Here's the result you get back:

{
  "result": {
    "content": [{ "type": "text", "text": "prod @ 2026-08-01T08:02:45.601Z" }],
    "resultType": "complete",
    "_meta": {
      "io.modelcontextprotocol/serverInfo": {
        "name": "updated-mcp-spec-bot",
        "version": "1.0.0"
      }
    }
  },
  "jsonrpc": "2.0",
  "id": 1
}

Two things worth noticing in that response.

The result already shows the prod deployment, because I ran this curl against the same server process the client just deployed through.

A totally separate client, no handshake, no session, and it reads the record the TypeScript client wrote. App state persists, protocol state doesn't.

Look at those headers. Mcp-Method and Mcp-Name are right there for your gateway to route on. And the _meta makes the request fully self-describing.

The DX here is genuinely good.

Deploying to Cloudflare Workers

We're deploying this to Cloudflare Workers, and it costs nothing: the free plan gives you 100,000 requests a day and a *.workers.dev subdomain, no credit card needed.

Why Workers? Because it's the natural way for a stateless MCP server. createMcpHandler returns a fetch-style handler, and fetch handlers are literally what Workers runs. The entire platform difference fits in one tiny file.

⚠️ Cloudflare has quick-start MCP templates (npm create cloudflare -- --template=cloudflare/ai/demos/remote-mcp-authless). As of writing, Cloudflare's own docs warn that these still scaffold the deprecated McpAgent path and say, "Do not use that path for a new server." It's the old stateful world, and it doesn't speak 2026-07-28. Skip the template. createMcpHandler is the recommended path, and it's what we're already using.

Step 1: The Worker entry

Create worker.ts:

// 👇 worker.ts

import { Hono } from "hono";
import { createMcpHandler } from "@modelcontextprotocol/server";
import { buildServer } from "./bot.js";

const handler = createMcpHandler(buildServer);

const app = new Hono();
app.get("/healthz", (c) => c.text("ok"));
app.all("/mcp", (c) => handler.fetch(c.req.raw));

export default app;

Eleven lines. Same buildServer, same tools, same MRTR flow.

Two deliberate differences from the Node entry:

Plain new Hono() instead of createMcpHonoApp(). The Host validation in createMcpHonoApp is DNS rebinding protection for localhost servers. Behind Cloudflare's edge, it just gets in the way.
No serve(...). Workers calls your exported fetch handler itself.

Step 2: The wrangler config

Wrangler is Cloudflare's CLI for Workers. It bundles your TypeScript (no build step needed), runs it locally on the real production runtime, manages secrets, and deploys.

npm install -D wrangler

Create wrangler.jsonc:

{
  "name": "updated-mcp-spec-bot",
  "main": "worker.ts",
  "compatibility_date": "2026-07-01",
  "compatibility_flags": ["nodejs_compat"]
}

The nodejs_compat flag fills in Node-ish globals so npm packages behave.

Step 3: Test on the real runtime, locally

npx wrangler dev

This runs worker.ts on workerd, the same engine that runs in Cloudflare production, at http://localhost:8787. Point the client at it:

MCP_URL=http://localhost:8787/mcp npx tsx client.ts

Same full output as the Node run: 2026-07-28 negotiated, cache hints, the MRTR deploy round-trip. Except one line:

server_stats: pid=1 requestsServed=6

pid=1. That's the edge runtime saying hello. 🫡

Step 4: Deploy it

Create a free account at dash.cloudflare.com/sign-up if you don't have one, then:

npx wrangler login

Set the production requestState secret (this is the shared HMAC key, so every edge instance can verify tokens minted by any other):

openssl rand -hex 32          # copy the output
npx wrangler secret put STATE_KEY   # paste it when prompted

⚠️ One gotcha from my own run: the key must be at least 32 bytes or the codec throws at startup. openssl rand -hex 32 gives you 64 hex characters, which is plenty.

And ship it:

npx wrangler deploy

First deploy asks you to pick your free workers.dev subdomain. Ten seconds later:

https://updated-mcp-spec-2026.<your-subdomain>.workers.dev

Your MCP server is live on Cloudflare's global edge.

Step 5: Verify from the outside

Run the Step 6 curl against the public URL (just swap the host), hit /healthz in a browser, and then the real proof:

MCP_URL=https://updated-mcp-spec-2026.<your-subdomain>.workers.dev/mcp npx tsx client.ts

Same output. Except now it's on the internet.

Connect a real agent to it, with no tunnel and no ngrok:

claude mcp add --transport http updated-mcp-spec-2026 \
  https://updated-mcp-spec-bot.<your-subdomain>.workers.dev/mcp

Run /mcp in a Claude Code session to see it connected, then ask it to "deploy to staging". Since Claude Code doesn't declare the elicitation capability yet, our capability-aware fallback kicks in: the tool tells the agent to confirm with you first, you say yes in chat, and the deploy lands.

Bonus: run npx wrangler tail while you do it and watch the requests land in your production logs live.

Here’s a small demo:

One caveat

Our deployments array lives in memory, and on Workers, memory is extra ephemeral: isolates spin up and down per location, so two requests might see different histories. That's not a bug in the demo; it's the whole lesson of the spec, one more time. Protocol state is gone by design, and app state belongs in real storage. On Cloudflare, that's KV, D1, or Durable Objects.

Where Composio fits

What we just built is one server with three tools. Real agents need Gmail, Slack, Notion, GitHub, Linear, and fifty other things.

You could build and deploy a similar server for every one of those. Handle each app's OAuth. Keep up with every API change. Run all that infra.

Or you point your agent at Composio, which gives you 1000+ apps behind a single MCP endpoint:

https://connect.composio.dev/mcp

Build custom MCP servers (like the one we built) for your own domain logic, and let Composio be the app layer for everything else.

Conclusion

The 2026-07-28 spec is a breaking release, and it's the good kind of breaking.

MCP servers are now boring HTTP services. Deploy them like you deploy everything else: stateless, load-balanced, cacheable, autoscaled. The handshake is gone, sessions are gone, and interactive tools work anyway thanks to MRTR.

If you're starting a new server today: use SDK v2, use the createMcpHandler(buildServer) factory pattern, keep resources at module level, seal your requestState, and split your logic from your platform entry. We went from localhost to Cloudflare's global edge with an eleven-line file, and the same split works for Railway, Render, Fly, or a plain VM through server.ts.

If you have existing servers: you've got a 12-month window on everything deprecated. Use it.

Now your MCP server finally gets to be just another web service. 🤌

ℹ️ You can find the entire source code in the repository.

Shrijal AcharyaFollow

SDE • GOLD @Microsoft Student Ambassador • Prev Lead Collab and Dev-Team Lead @oppiaorg • Mail for collaboration

How to connect MCP servers to Slackbot

Shrijal Acharya — Sat, 18 Jul 2026 11:57:47 +0000

Slackbot recently added support for MCP, which means you can now connect it with external apps and let it take actions across your work tools directly from Slack

But the native app list is still limited. By the time of writing this post, there's just about 20 apps that you can connect from the Slack marketplace.

And for most teams, that's not enough.

But luckily, Slack allows you to set up or use your custom MCP servers and not have to be limited by the number of apps available in marketplace.

That's where Composio helps you. It can connect your slack bots to 1000+ apps that you can use.

In this guide, we’ll go through how to connect Slackbot with Composio’s MCP server in 3 steps.

ℹ️ The steps will be pretty much the same with other MCP servers as well.

What We're Building

Once this is set up, you can ask Slackbot things like:

Find my latest unread Gmail emails.
Search my Notion workspace for launch notes.
Check my Google Calendar for meetings tomorrow.

And a bunch more. Imagine all the stuff you can do with 1000+ apps. 😵‍💫

I'll leave the rest to your imagination...

Slackbot sends the request to Composio Connect, Composio finds the right tool, asks you to connect the app if needed (one time), and then executes the action.

Prerequisites

Before we begin, make sure you have:

A Slack workspace with Slackbot MCP client access (comes with Business+ and Enterprise plan)
Permission to create or configure a Slack app.
A Composio account.

That's it.

Step 1: Get the Composio Connect MCP URL

First, you need the MCP server URL from Composio.

For this setup, use Composio Connect:

https://connect.composio.dev/mcp

This is Composio’s hosted MCP server that gives your AI agent access to 1,000+ apps with just 7 meta-tools that let the slackbot discover what's available, authorize apps on demand, and execute tools across apps in parallel through a single connection.

You don’t need to create a custom MCP server for this guide.

Composio also supports custom MCP servers for more scoped project-specific use cases, but those can require API-key-based auth. For Slackbot, Composio Connect is the simpler path because it works with OAuth-based MCP client flows.

Step 2: Add Composio Connect to a Slack App

Now, we need to register the Composio MCP server inside a Slack app.

Go to the Slack developer dashboard and create a new app.

Once the app is created:

Open your Slack app.
In the left sidebar, go to Features.
Click MCP Servers.

Click Get Started.

Now fill in the MCP server details.

Use:

Name: Composio (or anything you wish)
URL: https://connect.composio.dev/mcp
Auth Type: Dynamic Client Registration

For the auth type, select Dynamic Client Registration.

ℹ️ This is the right option for MCP servers that support OAuth discovery and client registration. Slack handles the client registration automatically, so you don’t need to manually create OAuth credentials first.

Now, if you click the three dots and then Tools, you should see that it currently cannot fetch the tools because the MCP server uses a dynamic connection and must be installed in your workspace first.

So, now head over to the Install App tab, and install it to the workspace you selected when creating the app.

If your workspace requires approval, send the app request to your admin.

Step 3: Connect Composio inside Slackbot

Once your Slack app is installed and approved, open a DM with Slackbot.

Then, just type in a prompt that requires using the app, Slack will use the correct app automatically for you.

Click Connect, and you’ll be taken to a confirmation page.

Click on Continue, and then confirm it on the Composio end.

If everything went well, you should see that your account is connected.

After that, Slackbot should be able to discover Composio’s MCP tools.

Start with a simpler test:

What tools are available from Composio?

Once that goes through, now try an actual app action.

For example:

Send a mail to x@y.com saying 'Hi, from Composio 👋 inside Slackbot'

If Gmail is not connected yet, Composio should generate an OAuth link for you to connect it. Once you approve it, the connection persists for future use. So, you don't have to repeat this step again and again.

Slackbot may ask you to approve the action before it writes data to another app.

That's expected. Once connected, Slackbot can use that app through Composio.

Voilà, you've successfully connected Slackbot to Composio MCP. 🎊

Here’s a quick workflow for initiating a connection and running an actual app action:

Final Thoughts

Slack's own marketplace is good, and if it covers all the tools you require, you can completely stick to it.

But for some of you, that's simply not enough. I hope this helps overcome that problem.

So instead of jumping between different tools, you can ask Slackbot to find information, create tasks, update records, and run actions across your apps from inside Slack.

This is a much-needed quality-of-life improvement for teams that already live in Slack.

Slackbot gives you the interface. MCP gives you the protocol.

And Composio gives you the app layer. 👌

Shrijal AcharyaFollow

SDE • GOLD @Microsoft Student Ambassador • Prev Lead Collab and Dev-Team Lead @oppiaorg • Mail for collaboration

Claude Opus vs Kombai in 3 Real-World Frontend AI Tests 🚀

Shrijal Acharya — Tue, 02 Jun 2026 16:06:41 +0000

Frontend automation has been getting pretty wild lately. 🫠

A few months ago, this comparison would have been much easier to frame.

On one side, you had the Claude Opus lineup, one of the strongest coding model lineups available, running through tools like Claude Code with Figma MCP support.

On the other side, you had Kombai, a frontend-focused AI coding agent that was mostly known for turning Figma files, screenshots, prompts, and existing designs into clean frontend code.

But Kombai has changed quite a bit since then.

It is not just an AI coding agent for frontend anymore. With the newer updates, especially Design Mode, Kombai is now trying to cover a much bigger part of the frontend workflow.

That changes the comparison a little.

If we only compare it on code generation, we are kind of testing just one part of what Kombai does now. It can also help create UI designs, iterate on them visually, and then move those designs into code inside the same IDE workflow.

I will go deeper into that later in the Kombai section, because there is quite a bit to talk about there.

For this post, though, I still want to focus mainly on the part that matters most in real projects:

🤔 Can it work inside an existing codebase and ship good frontend changes without breaking things?

Because at the end of the day, that's the actual pain point of frontend.

If you’re interested, we already compared the older version of Kombai and Figma MCP here: Figma MCP vs Kombai

So in this post, we’ll look at what has changed with Kombai, touch on Design Mode, and then compare it with Claude Opus on 3 real open-source projects.

Let’s find out which one actually feels more useful. 👇

TL;DR

If you just want the takeaway, here’s the quick rundown:

Kombai is not just a Figma-to-code tool anymore. It is now more of an all-in-one design and frontend coding agent. You can generate designs, iterate on a canvas, visually edit UI, use Figma as input, work with your real codebase, and then ship great frontend code from inside your IDE.

Design Mode is the biggest new change. It gives Kombai an infinite canvas, Style Guides, Themes, reusable Blocks, CSS-level visual editing, and a one-click flow from design to code. It is still early, but it changes how you should think about Kombai.

For the coding tests in this post, Kombai was still clearly stronger overall. In real open-source projects, it handled frontend implementation, codebase understanding, UI quality, and integration better than Claude Opus in most cases.

Claude Opus 4.6 with Figma MCP is still totally fine, especially if you already work mostly in the CLI with tools like Claude Code, Codex, or whatever else. You can absolutely stick with that setup and still get solid results. It works well.

The bigger point is this:

If you are already comfortable in the CLI, there is no real reason to switch just for this. A good model with Figma MCP is more than enough for a lot of use cases.

But if you use Cursor, VS Code, Windsurf, Trae, Antigravity, Kiro, or another GUI IDE, you should seriously try Kombai at least once.

It is just a very good tool for frontend work.

I’ve been using Kombai for more than 10 months, and it still feels kind of wild sometimes. Honestly, I have Cursor installed on my machine for Kombai alone. That’s how useful it has been for me.

Brief on Kombai

💁 Kombai is an all-in-one design and frontend coding agent for building production-ready frontends

Kombai used to be easy to describe as a frontend AI agent.

This is still true, but now its just a small part of what it can do.

The better way to describe it today is this:

It's an agent specially designed for frontend unlike usual coding agents like Claude Code, Codex or anything else.

That specialization shows up in a few places:

It understands frontend stacks and component libraries
It can parse Figma designs natively
It can work with existing components, tokens, hooks, and design systems
It can visually inspect and edit the browser output
It can generate UI designs before writing code
It focuses more on production-ready frontend changes, and not a generic app generation

The biggest recent update is the Design Mode.

With Design Mode, Kombai can generate UI designs directly inside your IDE. You can start from a prompt, image, website, or Figma reference, and Kombai creates editable UI designs that you can change visually.

Here’s a quick demo of Kombai’s Design Mode in action by Beau Carnes (core team member of freeCodeCamp).

It also has proper design-system-ish primitives now:

Style Guides for controlling the overall visual direction
Themes for reusable design tokens
Blocks for reusable design elements
Variants for trying multiple design directions
A CSS editor for visual tweaks
and many more...

These designs live as .canvas files in your repo, which means they can be version-controlled with git like your code.

Once you like a design, you can hit Code design, and Kombai moves it into Code Mode, reads your project, understands your stack, reuses relevant components and tokens, and codes it into working frontend.

That is what makes it feel different from tools like v0, Lovable, or Bolt.

Those tools are great for generating prototypes or apps from scratch. Kombai is more focused on your existing frontend repo. It works inside your editor and tries to build the thing using the stack you already have.

Check out this intro demo Kombai 2.0. The first AI design engineer.

// Detect dark theme var iframe = document.getElementById('tweet-2061825199247614316-192'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2061825199247614316&theme=dark" }

With Kombai, you can not only create designs, but also get the safety of knowing that no backend code is ever touched, which ensures your business logic is not mistakenly changed.

You can add it right inside your editor. It works with VSCode, Cursor, Windsurf, and Trae. Just grab it from the extension marketplace, launch it, and you’re ready to go.

With Kombai, you can:

Generate UI designs from scratch using Design Mode.
Turn Figma designs into code without setting up Figma MCP separately.
Turn Figma designs into code (React, HTML, CSS, etc.) using the component library your project already uses.
Use screenshots, images, websites, or natural language as input.
Reuse existing components, hooks, design tokens, and frontend conventions from your repo.
Visually inspect and edit your UI through the Kombai Browser.
Work with a frontend-smart engine that understands 30+ libraries, including Next.js, MUI, and Chakra UI.
Most importantly, preview the changes in a sandbox so you can approve or reject the change before committing it to the files.

Head to the docs to get started and find the setup for your editor.

You can be up and running in under a minute:

Install the extension for your editor
Sign in and connect your project
Pick the mode you need: Code, Plan, Design, etc.
Paste a Figma link or describe what you want to build
Paste a Figma link, describe what you want, attach an image, or work from your existing code
Review the output and commit your code

If you spend most of your time on the frontend, this is a no-brainer.

Now comes with 100 award winning landing pages that you can make your own.

// Detect dark theme var iframe = document.getElementById('tweet-2061825201730695550-647'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2061825201730695550&theme=dark" }

One Important Note Before the Tests

Kombai now has a much bigger features and design support than it did before.

But this post is still mainly a frontend coding and implementation comparison.

So, I am not going to deeply test Design Mode here. That deserves its own separate post because the right comparison there would probably be against tools like v0, Lovable, Bolt, and maybe Figma-based workflows.

For this post, the test is still:

Can it understand a real codebase?
Can it preserve functionality?
Can it implement a feature cleanly?
Can it match or improve UI quality?
Importantly, can it work in a large real-world existing project?

That is exactly what we are going to test here.

Test Workflow

In this model test, I’ll be using Claude Opus 4.6 with everyone’s favorite CLI coding agent, Claude Code, along with Figma MCP support.

💁 Just in case you're interested in how to add MCP support to Claude Code, you can view the guide here.

For Kombai, the choice mostly comes down to the IDE, so I’ll go with Cursor. It does not really matter much which IDE you use, though. VS Code would work just fine too.

We’ll test both tools on three decently complex tasks in real-world open-source projects with hundreds of thousands of LOC.

A frontend-heavy task with Figma
A frontend + backend task with more implementation complexity
A task that relies more on codebase understanding than implementation complexity

I’ll compare both tools on:

time to complete the task
quality of the code
how closely the final output matches the given design or feature intent

💡 Note: I’ll share the source code changes for each task from both tools in a .patch file. That way, you can easily reproduce them on your local system by cloning the repository and applying the patch with git apply <path_file_name>.

Note: As I’m using a Claude plan and not API usage, price is roughly estimated based on the output tokens.

Real-World Coding Comparison

The entire test in this blog is going to be on top of real-world open-source projects that are used by thousands of people, not toy projects, but ones with thousands of LOC.

All three tests are going to be on 3 different projects, all open-source, of course!

Let's start with an easier one, a shadcn template repository with Figma MCP.

Test 1: Rebuild an Open-Source Project UI (with Figma MCP)

For this test, I'll be comparing both of them on a Figma file, giving them access to a Figma MCP server.

You can find the Figma design template for this test here: Dashboard

Prompt:

Implement the provided Figma dashboard design in this existing Next.js + shadcn/ui dashboard codebase.

<figma_url>

Constraints:

1. Preserve all existing functionality exactly.
2. Do not break routing, state, existing interactions, or responsiveness.
3. Replace the current dashboard presentation layer with a UI that closely matches the Figma design.
4. The Figma is food-delivery themed. Translate only the content domain, not the visual system.
5. Keep the layout, structure, and styling language of the Figma as intact as possible.
6. Replace food-specific labels and data with content that fits a generic admin dashboard.
7. Reuse existing logic and data bindings wherever possible.
8. Avoid adding fake backend logic.
9. Keep the implementation production-quality and componentized.

Focus areas:

- sidebar
- header/top bar
- summary metric cards
- chart section
- table/list section
- filters/search/actions if present

Claude Opus 4.6 (with Figma MCP)

Here's the response from Claude Opus 4.6:

You can find the code it generated here: Opus 4.6 source Code

Opus did a pretty good job here.

Given the Figma itself was not even made for a generic admin dashboard in the first place, there was obviously some room where it had to improvise a bit on its own.

But the main thing I was testing for here was not just whether it could make the frontend look close enough. The actual ask in the prompt was to preserve and support the existing functionalities as well. That part just was not really there. It did not add the interactivity support properly, which was the whole point of the test after all.

So visually, sure, it looks alright. But if the interaction layer is missing, that is a pretty big miss for this kind of task. If it had nailed that part too, this would have been a really strong result. But it didn’t.

Output token Cost: ~$0.125
Duration: 9 minutes 1 second
Output Token Usage: ~8K
Code Changes: 6 files changed, 420 insertions(+), 422 deletions(-)

Kombai

Here's the response from Kombai:

You can find the code it generated here: Kombai source Code

Kombai nailed this one. Even better how accurately it finds out the tech stack.

The frontend turned out really good. More importantly, the interactivity is there, which is the main thing being tested other than how close it comes to the Figma design. It actually respected the fact that this is an existing app, with behavior that still needs to work.

And if I compare both the UI itself and the overall build quality from a production POV, Kombai clearly did better here.

That said, there was one issue I noticed while working with Kombai.

After it finishes the implementation, it has this nice default feature where it opens up a browser preview and lets you chat there to fix smaller things quickly. In theory, that sounds great. In practice, for apps that require authentication, which this one did, it falls apart. Google OAuth simply flags it as unsafe, so you cannot log in there at all.

So yeah, they definitely need to work on that.

Still, overall, the user experience and the actual result were top notch.

Duration: ~12 minutes
Code Changes: 12 files changed, 940 insertions(+), 532 deletions(-)

I noticed the project also has standalone Kanban board support, so why not quickly test it on this as well?

Prompt:

Completely refactor the Kanban board UI in this open source project based on this Figma:

<figma_url>

This should be a real UI redesign, not minor styling tweaks. Study the Figma closely and make the board feel much more polished, modern, clean, and cohesive. Improve layout, spacing, typography, hierarchy, cards, columns, controls, interaction states, and responsiveness.

Preserve functionality, but refactor components and styling where needed so the code is cleaner and the design is more consistent. Focus heavily on UI quality and make the final result feel much closer to the Figma overall.

Claude Opus 4.6 (with Figma MCP)

Here's the response from Claude Opus 4.6:

You can find the code it generated here: Opus 4.6 source Code

Opus did pretty well here too.

The good part is that functionality was not broken. The board still works, and that matters a lot for a refactor like this. The redesign itself is also nice. It is clearly better than before.

That said, if I look closely at the actual design match, it seems to miss a little here and there. It does not feel quite as locked in to the Figma as the best result should.

Cost: negligible
Duration: 5 minutes 26 seconds
Code Changes: 5 files changed, 162 insertions(+), 56 deletions(-)

Kombai

Here's the response from Kombai:

You can find the code it generated here: Kombai source Code

Again, Kombai was excellent here.

Honestly, this one came out awesome. It matches the Figma really well, preserves the expected behavior, and feels like a proper redesign.

Code Changes: 4 files changed, 362 insertions(+), 72 deletions(-)
Duration: ~6 minutes

Test 2: Add a Feature in Uptime Kuma

There's this open-source project that's super popular for the self-hosted monitoring service Uptime Kuma, with over 84K stars on GitHub.

Here's an existing issue on the project that we will try to build: Calendar Graph

Prompt:

Add a heatmap-style uptime history section to the public status page in this existing Uptime Kuma codebase.

Constraints:

1. Keep all existing functionality working.
2. Do not break the public status page, responsiveness, or current monitor behavior.
3. Build this as a proper feature inside the existing architecture.
4. Reuse existing logic and data flow wherever possible.
5. Avoid fake backend logic or hardcoded mock data.
6. Make the heatmap work independently for each service.
7. Keep the UI consistent with Uptime Kuma’s existing style.
8. Make the implementation clean and production-ready.

Focus areas:

- status page
- per-service uptime history
- heatmap UI
- state management
- backend integration

Claude Opus 4.6 (with Figma MCP)

Here's the response from Claude Opus 4.6:

You can find the code it generated here: Opus 4.6 source Code

Opus actually got the overall feature in place, and to be fair, the core thing does work.

That said, there is a bug.

When you try to change the monitoring duration for one service, it also changes it for all the others. That is obviously not how this should behave, and for a feature like this, that kind of state handling bug is a big issue, of course!

Other than that, there is not a whole lot to complain about. The core functionality works, just with some caveats, and that feels fair. I do not think it is realistic to expect the model to get every detail perfectly right in one shot on a non-trivial codebase.

Output token Cost: ~$0.12
Duration: 13 minutes 5 seconds
Output Token Usage: ~7.5K
Code Changes: 4 files changed, 784 insertions(+)

Kombai

Here's the response from Kombai:

You can find the code it generated here: Kombai source Code

Kombai did this one properly.

The feature is implemented correctly, the behavior is right, and even the bug that showed up in the Claude Opus implementation is not there at all.

That part matters because this is not just about working on the UI. It's about putting the feature into an existing project.

Duration: ~10 minutes
Code Changes: 4 files changed, 809 insertions(+), 1 deletion(-)

Test 3: Add a Feature to Chatwoot

This is a little different from the other two tests.

This was more of a test to see how well the model actually understands the codebase and less about generating or working on the UI heavily. The code change isn't going to be huge, but it tests the two on how good they are at understanding the codebase and adding a feature on top.

Prompt:

Add a new "Participating" tab to the chat/conversation list in this existing Chatwoot codebase.

The goal is to let users quickly view conversations they are participating in, while keeping the implementation fully aligned with how Chatwoot already handles conversations, tabs, filtering, permissions, and dashboard state.

Constraints:

1. Preserve all existing functionality exactly.
2. Do not break existing conversation list behavior, routing, filters, permissions, or dashboard interactions.
3. Add "Participating" as a proper tab in the existing chat list UI, not as a separate temporary view.
4. Make sure the tab only shows conversations the current user is participating in.
5. Reuse existing backend, frontend, and store patterns wherever possible.
6. Avoid hacks, fake data, or disconnected logic.
7. Keep the implementation production-quality and consistent with the current Chatwoot UI.
8. Ensure permissions and visibility rules continue to work correctly.
9. Make the feature feel like a native part of the product.

Focus areas:

- chat list tab integration
- conversation filtering
- backend query support
- participation-based logic
- store/state updates
- correct tab placement in the UI

Claude Opus 4.6 (with Figma MCP)

Here's the response from Claude Opus 4.6:

You can find the code it generated here: Opus 4.6 source Code

Opus nailed this one.

This is exactly the kind of test where you are not making massive frontend changes, but instead need a solid understanding of how the app actually works so the feature fits naturally into the existing codebase.

The only issue I noticed was some UI flickering. But that looked more like a minor issue than anything wrong with the feature implementation itself.

Other than that, it worked perfectly.

Output token Cost: ~$0.02
Duration: 12 minutes
Output Token Usage: ~1.3K
Code Changes: 13 files changed, 52 insertions(+), 11 deletions(-)

Kombai

Here's the response from Kombai:

You can find the code it generated here: Kombai source Code

There was no flickering here, and Kombai also placed everything where it actually made sense. Claude got the logic right too, but the tab placement felt a bit off. Kombai just tied it together better, and the whole thing felt a bit cleaner.

So overall, both did well here, but Kombai’s result felt cleaner and better integrated.

Code Changes: 7 files changed, 30 insertions(+), 9 deletions(-)
Duration: ~8 minutes

Final Verdict

So, what’s the takeaway?

After testing both on real tasks in real codebases, Kombai was clearly stronger for frontend work.

That is not to say Claude Code with Opus 4.6 is bad. Far from it.

It is one of the strongest coding models available right now, and it can do some serious work. In some cases, especially the Chatwoot test, it understood the codebase really well and shipped something that was genuinely solid.

Claude Opus lineup is too good for general coding.

Kombai is a frontend specialized tool.

And for frontend-heavy work, that specialization really shows.

That said, I do not think you should take my word for it blindly.

Also, this does not mean Kombai will win every project or every workflow.

If you are doing backend-heavy work, infra changes, full-stack architecture, or mostly CLI-based development, Claude Code still makes a ton of sense.

The design side is also getting more interesting now. I did not fully test Design Mode in this post, because that deserves a separate comparison.

Honestly, the best way to judge tools like this is to try them yourself. A comparison like this can give you a rough idea, but it really clicks only when you use them on your own codebase and see how they actually feel.

That’s all for this one. Thank you for reading! ✌️

Claude Code vs. OpenCode without the hype

Shrijal Acharya — Thu, 21 May 2026 13:55:18 +0000

Everyone wants a coding agent now.

Not a chatbot that explains code.

An actual agent that can read your repo, edit files, run commands, use tools, and keep moving while you supervise.

Claude Code and OpenCode are two of the most interesting takes on that idea.

Claude Code is the polished Anthropic-native route.

OpenCode is the open-source route for people who want more model choice, more control, and a setup they can tweak.

And that difference matters more than it looks.

What is OpenCode

ℹ️ Open-source coding agent with model and tool control

OpenCode is an open-source coding agent for developers who want more control over their AI coding setup.

It runs in the terminal, IDE, and desktop, and lets you bring your own model instead of locking you into one provider. Claude, GPT, Gemini, local models, and 75+ other providers are supported. That is probably the biggest reason people care about it.

It also comes with the things you expect from a serious coding agent now: LSP support, multi-session workflows, project memory through AGENTS.md, MCP tools, custom agents, plugins, and editor support, and maybe a bunch more.

So the pitch is not just “AI in your terminal.”

That undersells it.

OpenCode is closer to a coding-agent workbench. You bring the model, the provider, the editor, the agents, and the workflow. OpenCode gives you the open layer that ties it all together.

Not everyone needs that level of control.

But some developers absolutely do.

💁 OpenCode is for developers who want to tweak every single detail of their coding agent.

That is what makes it interesting next to Claude Code.

What is Claude Code

ℹ️ Anthropic’s polished coding agent for your terminal.

Claude Code is Anthropic’s coding agent that lives in your terminal.

The idea is pretty same here, it can read your codebase, edit files, run commands, handle Git stuff, and all through prompts.

The big difference is that Claude Code is built around Claude.

That sounds obvious, but it matters.

You are not coming here to mix and match ten different model providers. You are coming here because you trust Anthropic’s models, and you want the cleanest experience around them.

Claude Code also comes with a lot of serious agent features: project memory through CLAUDE.md, slash commands, permissions, hooks, MCP, plugins, custom subagents, and IDE integrations.

Claude Code is closer to a Claude-native coding environment. The model, the agent loop, the tool use, the permissions, and the workflow all come from the same Anthropic-shaped box.

Less DIY.

But there is also a small shift happening.

Some developers are starting to move from Claude Code to OpenCode or OpenAI’s Codex for one simple reason: usage limits.

Claude Code is great, but when you are deep in a coding session, hitting limits feels brutal. And for heavier users, even the $200 Claude Max plan does not always feel like enough.

That is why OpenCode and Codex are tempting. Also read: Claude Code vs. Codex: Detailed breakdown

When Claude hits the wall, people still need a way to keep shipping.

💁 If you're an Anthropic fanboy, and don't care about other models, stick to Claude Code.

High Level Architecture

ℹ️ How both the agents work

At a high level, both OpenCode and Claude Code follow the same basic agent loop.

You give it a task.

It looks at the repo.

It decides what files, commands, or tools it needs.

It takes an action.

Then it reads the result and keeps going.

ℹ️ This is the highest-level architecture of a coding agent. A few details change from tool to tool.

That loop is the boring part.

The interesting part is everything around it.

Here is a tiny example of that loop in practice.

I gave Claude Code and OpenCode the same small task in a demo word-count repo:

Add a --json flag to a word-count CLI, update the tests, and run them.

The interesting part is not the feature. It is watching both agents go through the same shape: understand the repo, plan the change, edit the files, and run the tests.

Claude Code wraps that loop in Anthropic’s own product system. You get Claude, project memory through CLAUDE.md, permissions, hooks, MCP, plugins, Claude skills, and subagents in one single setup.

OpenCode takes a more open route. It gives you the agent runtime, but lets you bring different models, providers, agents, tools, and workflows. Its docs split agents into primary agents and subagents, and let you configure specialized assistants with custom prompts, models, and tool access.

So architecturally, the difference is not that one is an agent and the other is not.

They both are.

The real difference is who controls the harness around the agent.

Claude Code gives you Anthropic’s harness.

OpenCode gives you a harness you can inspect, and configure.

Context, memory and tool use

Both Claude Code and OpenCode are doing the same basic thing: they build a giant prompt, stuff it with repo context, tool definitions, memory files, recent messages, and tool results, then ask the model what to do next.

The difference is how much of that system you control.

Claude Code is more vertically integrated here. It is built around Anthropic models, so it can take advantage of Anthropic-specific stuff like prompt caching, native tool calls, and Claude’s own long-context behavior.

That matters.

Tool definitions, system prompts, and CLAUDE.md can be cached between turns, which makes long coding sessions cheaper and faster than they would be if Claude had to re-read everything from scratch every single time.

OpenCode takes a different route.

It does not assume one model or one provider. Instead, it reads the model’s context limit from the provider metadata and builds the session around that. So the same OpenCode setup can run with Claude, GPT, Gemini, Qwen, local models, or whatever else you plug in.

That flexibility is the whole point.

But it also means OpenCode has to normalize all the weird provider differences: tool call IDs, cache support, model limits, and tool-calling parts.

Claude Code gets to optimize deeply for Claude.

OpenCode has to work with everyone.

Memory works the same way.

Claude Code uses CLAUDE.md as the main project memory file. It can also load nested CLAUDE.md files, user-level memory, and auto-memory. So it feels more like the agent has a built-in memory system.

OpenCode uses AGENTS.md.

That is more portable. You can commit it to the repo, share it with the team, and use it as a general agent instruction file instead of something tied to one vendor. OpenCode can even fall back to CLAUDE.md, which makes migration easier.

At some point, every agent runs out of context.

Claude Code handles this by compacting the conversation. Older tool outputs are cleared first, then the session gets summarized if needed. That is why Claude Code has commands like /context and /compact.

OpenCode is a bit more explicit. It checks whether the session is close to the model’s context limit, keeps a buffer for output, and then prunes old tool outputs before doing a full summary. The important bit is that OpenCode stores the raw history in SQLite, so pruning does not mean the data is gone forever.

Tool use follows the same pattern.

Claude Code gives you a polished default toolbelt: read, write, edit, grep, glob, bash, web fetch, todo tracking, MCP, hooks, skills, and subagents.

OpenCode gives you a smaller but more configurable tool system: read, write, edit, patch, bash, grep, glob, web fetch, task, todo, skills, MCP, custom tools, and experimental LSP support.

The difference is who controls the tool layer.

Subagents and task delegation

Subagents are basically how coding agents avoid stuffing everything into one giant conversation.

Instead of making the main agent do every task itself, it can delegate a smaller job to another agent with its own context window, prompt, tools, and permissions.

Claude Code and OpenCode both follow the same basic pattern here.

The parent agent calls a Task or task tool.

A child agent spins up.

It does the work in isolation.

Then it returns one final message back to the parent.

Main Agent
  |
  | calls Task / task
  v
Subagent
  - own context window
  - own prompt
  - own tools
  - own permissions
  |
  | returns final result only
  v
Main Agent continues...

That part is important. The parent usually does not see the full subagent conversation. It gets the result, not the whole reasoning.

Claude Code has the more polished version of this.

It ships with built-in agents like Explore, Plan, and general-purpose. Explore is mostly read-only and useful for repo research. Plan helps gather context during planning. general-purpose is for broader work.

You can also define custom agents in .claude/agents/ with YAML frontmatter for things like tools, model, permissionMode, maxTurns, skills.

That means you can do stuff like:

Use a fast Haiku-style agent for repo search.

Use a stronger model for code review.

OpenCode has a similar shape, but it is more transparent.

It has primary agents and subagents. Primary agents handle the main chat, while subagents are called through the task tool or @ mentions.

Custom agents can live in .opencode/agents/*.md or inside opencode.json, with fields like mode, model, temperature, steps, prompt, and permission.

The interesting part is that OpenCode stores subagents as real child sessions in SQLite. So delegation is not just a hidden prompt trick. It is represented in the session model with its own messages, permissions, and snapshots.

That fits OpenCode’s whole philosophy.

Claude Code gives you a cleaner subagent experience.

OpenCode gives you a more inspectable one.

Permissions, safety, and control

This is where the two are very different.

Claude Code is more conservative by default. It has permission modes, allow/ask/deny rules, hooks, and sandboxing around Bash.

So you can allow boring commands like tests, deny obvious footguns like .env reads or curl | sh, and ask before anything risky.

The important part is that Claude Code has multiple safety layers.

Permissions decide what Claude is allowed to do.

Something like:

{
  "permissions": {
    "allow": ["Bash(npm run test *)", "Bash(git status *)"],
    "deny": ["Read(./.env)", "Read(./secrets/**)", "Bash(curl *)"]
  }
}

Hooks can intercept tool calls before or after they run and sandboxing gives Bash an OS-level boundary.

OpenCode is simpler.

Most of the control lives in one permission object inside opencode.json. You can set rules for bash, edit, read, task, webfetch, and other tools from the same place.

That is clean, but OpenCode is also more permissive by default. You are expected to configure the rules yourself.

Something like:

{
  "permission": {
    "*": "ask",
    "bash": {
      "*": "ask",
      "git status *": "allow",
      "git push *": "deny",
      "rm *": "deny"
    },
    "edit": {
      "*": "deny",
      "packages/web/src/**/*.tsx": "ask"
    }
  }
}

It does have some smart checks, especially for Bash. OpenCode parses shell commands with tree-sitter (the same thing you have inside NeoVim), so it can detect risky commands like rm, mv, chmod, or paths outside the project more carefully than plain string matching.

But there is no native sandbox like Claude Code.

The bigger OpenCode power feature is plugins. Plugins can intercept tool execution, add custom tools, and change agent behavior.

That makes OpenCode way more hackable.

What the Claude Code leak tells us

The interesting part of the leak is what it showed about coding agents.

A lot of Claude magic is in the harness around the model: context management, tool descriptions, prompt caching, permissions, compaction, subagents, and the agent loop.

OpenCode does pretty much the same. It is not trying to clone some impossible model-level feature. It is trying to build a different harness around similar idea.

OpenCode’s advantage is that the harness is open, inspectable, and replaceable.

Another thing that's clear is that the future is not just about better models, but the system around them.

Better context control.

Better tool boundaries.

Better memory.

Better permissions.

That is why this comparison is even interesting. Claude Code and OpenCode are not just two CLIs. They are two different answers to the same question:

❓ How much of the agent stack should be final, and how much should developers be able to control?

So, which should you pick?

There is no clever answer here.

Pick Claude Code if you want the cleanest Claude-native coding agent experience.

Pick OpenCode if you want more control.

Personally, I still love Claude Code.

I really do.

Anthropic models are banger, especially for coding. The problem is that the limits have started to piss me off. When you are deep in a coding session and the limit hits, it completely breaks the flow.

But there is some relief now.

On May 6, Anthropic announced a new compute partnership with SpaceX and doubled Claude Code’s 5-hour limits for Pro, Max, Team, and seat-based Enterprise users. They also removed peak-time limits for Pro and Max users.

That makes Claude Code a lot easier to recommend again.

I am personally still mostly stuck with Claude Code because the experience is just that good.

But I use OpenCode when I want to try newer models like Kimi, OpenAI models, or local models. That is where OpenCode makes more sense to me. And by no means, it is to say that you can't use Anthropic models in OpenCode, you can, and that makes it even better.

Final thoughts

Claude Code and OpenCode are both useful, but for different reasons.

Claude Code is the one I’d pick if I just want the agent to work without thinking too much about setup. It feels cleaner, and better for getting into a repo quickly.

OpenCode is more for when you want control. Different models, different providers, more ways to shape the workflow around how you actually code.

I wouldn’t overthink it.

If you hate setup and love Anthropic, use Claude Code.

If you want more flexibility and less vendor lock-in, use OpenCode.

That’s really the whole comparison.

Shrijal AcharyaFollow

SDE • GOLD @Microsoft Student Ambassador • Prev Lead Collab and Dev-Team Lead @oppiaorg • Mail for collaboration

Kimi K2.6 vs. Claude Opus 4.7 in a Weird Game Coding Test ✅

Shrijal Acharya — Tue, 05 May 2026 13:04:03 +0000

Kimi K2.6 has been getting a lot of love lately, especially from devs who want a strong coding model without paying premium model prices every time they run a big prompt.

So I wanted to see how good this model actually is. But this time, I wanted to compare it with something much heavier, the developers darling Claude Opus 4.7.

On paper, Claude Opus 4.7 and Kimi K2.6 are very different models.

One is a premium frontier model from Anthropic. The other is Moonshot AI's much cheaper open model for coding and agentic tasks.

The pricing difference is pretty wild too. Claude Opus 4.7 costs $5/M input tokens and $25/M output tokens. Kimi K2.6 is listed at $0.95/M input tokens and $4/M output tokens, with cached input going even lower at $0.16/M tokens.

That is a pretty big gap.

So in this article, we'll see how the cheaper model, Kimi K2.6, does against Claude Opus 4.7.

For the test, I gave both models the same coding task: build a small Minetest (similar to Minecraft) bounty board with a TypeScript backend, then extend it with Google Sheets logging through Composio.

TL;DR

If you want the quick take, Claude Opus 4.7 clearly won this test, but it was painfully expensive.

Opus was better at the real task. The local build was cleaner, and it was the only one that got the real Google Sheets integration working.
Kimi did pretty well in Test 1. It got the local bounty board working for way less money, but it needed more debugging.
Test 2 changed the whole comparison. Opus was expensive, but it finished. Kimi just could not put it all together.

The cost difference was wild though.

For the first local bounty board test, Opus cost around $3.59, while Kimi came in at around $0.39. That is a huge gap. For the basic version, Kimi honestly did pretty well for the price.

But once the task got a little more real, the gap became way more obvious.

Opus got it working, even though it needed a little back and forth. The Google Sheets sync worked, and the project was modular enough that I could test the whole flow with two curl requests without even opening the game.

The painful part is that the Composio run alone cost $16 and took around 28min 52sec API time.

Kimi, on the other hand, burned 135k+ tokens, took around 25 minutes, cost around $5.03, and still did not really get any closer.

👀 So yeah, Kimi K2.6 is a usable and interesting cheaper model. But in this test, it could not really come close to Opus 4.7 for real-world coding.

Evaluation

I treated this like a real project, not a benchmark chart. Both models got the same prompts, and I compared the results based on whether it actually worked, how clean the code was, how much debugging it needed, how long it took, and how much it cost. That last one matters a lot here.

Setup

Same tasks and prompts for both models (Test 1: local-only bounty board, Test 2: real Composio Google Sheets sync).
Same target architecture: Minetest/Luanti Lua mod + TypeScript backend.
Same success criteria: /bounty flow works in-game, backend APIs behave correctly, and in Test 2 the completion is appended to Google Sheets via Composio.

What I measured

Functional correctness (most important): Did it work end-to-end with real verification?
- Local run: could a player generate, progress, and complete bounties without breaking state?
- Backend: did /health, /api/bounty/generate, /api/bounty/complete, and /api/leaderboard return the expected shapes?
- Test 2: did the Google Sheets append succeed, and could I validate it from the API without needing to be in the game?
Code quality and structure: modularity, clarity, and whether the repo was easy to reason about and test.
Debug burden: how many follow-ups were needed, how confusing the failure modes were, and whether issues were “real bugs” vs. “misconfiguration traps.”
Time: API time and wall time for each run.
Cost and token usage: input, output, cache behavior, and total run cost.
Practical ergonomics: whether I could validate quickly (for example, testing the full backend + Composio flow with curl).

How I verified outcomes

Test 1: ran the backend locally, joined a local Minetest world, used /bounty, and confirmed task tracking, rewards, and leaderboard persistence.
Test 2: verified the end-to-end sync by generating and completing a bounty via the backend API, and confirming a successful Google Sheets append through Composio.

Scoring approach

This was not a “unit test leaderboard” benchmark. It was a real build-and-ship check.

A model “wins” when the project works with minimal intervention.
A model “loses” when it cannot reach a working state in reasonable time/cost, even if parts of the code look promising.

Coding Test

For this test, I used the following CLI coding agents:

Claude Opus 4.7: Claude Code, Anthropic's terminal-based agentic coding tool

Kimi K2.6: OpenCode via OpenRouter

ℹ️ This is a practical coding test, so both models get the same prompt. I will compare time taken, code quality, token usage, cost, and all that stuff.

What are we building?

For this test, I wanted something small enough to verify properly, but still weird enough to show how each model handles an unusual idea.

So, we're building a simple Minetest/Luanti bounty board.

A player can join a local world, run /bounty, get a task like mining dirt or placing torches, and receive a reward after completing it.

After that, the backend records the completion, logs it to Google Sheets through Composio.

I get it, the concept is a little unusual on purpose.

Test Prompts

Both models received the same prompts for each test.

Test 1 Prompt: Local Bounty Board Prompt
Test 2 Prompt: Real Composio Integration Prompt

Test 1: Local Bounty Board

This first test is the basic version of the idea.

No external tools, no Composio. Just the game, the backend, and the local bounty flow working properly.

The goal was simple. A player runs /bounty, gets a task, completes it inside the game, and the backend tracks the progress without everything falling apart.

Claude Opus 4.7

Claude Opus 4.7 handled the first test really well.

The local bounty board worked end to end. It built the TypeScript backend, the Minetest/Luanti Lua mod, the command flow, progress tracking, rewards, and leaderboard persistence without needing a bunch of follow-up fixes.

The file structure also felt nice, which I had specifically asked for in the prompt:

The backend was built with Express, Zod, and Vitest. It also handled the boring stuff properly, which honestly matters a lot here:

npm test passed with 11/11 tests
npm run build passed cleanly
/health, /api/bounty/generate, /api/bounty/complete, and /api/leaderboard returned the right response shapes
incomplete bounty completions returned a clean 400

It created the Lua files cleanly, used minetest.request_http_api(), handled secure.http_mods, tracked digging and placing, stored player bounty state, and handled inventory rewards properly.

The whole run took around 12 minutes of API time, with about 23 minutes wall time. That is a bit longer than a quick web app build test, but for this kind of cross-stack project, it felt fair.

The cost came out to $3.59, which is definitely not cheap. But to be fair, the output was actually useful. It added a lot of code, but most of it was real implementation, not random filler like CONTRIBUTING.md, INSTALLATION.md, and all those extra files models sometimes create for no reason.

You can find the code it generated here: Claude Opus 4.7 Code

Here's the demo:

Cost: ~$3.59
Duration: 12min 3sec API time, 23min 53sec wall time
Code Changes: +1,688 lines, 0 lines removed
Token Usage:
- Input: 65
- Output: 54.8k
- Cache read: 2.8M
- Cache write: 129.8k

Quick Verdict

It worked end to end without much tweaking. I only had to configure ~/.minetest/minetest.conf and add this line:
secure.http_mods = bountyboard
Pretty much everything else was smooth. Great quick MVP.

I noticed one small issue: mine_node bounties can be farmed by placing and then re-mining the same blocks, because vanilla Minetest does not track who placed a node.

But that's fine. That is not really a code issue here.

Kimi K2.6

The core idea worked. Kimi created the TypeScript backend, the Minetest/Luanti mod, the bounty commands, task tracking, completion flow, rewards, and leaderboard logic.

The backend side looked solid enough. It used Express, Zod, and Vitest, and the main routes were there:

/health
/api/bounty/generate
/api/bounty/complete
/api/leaderboard

It also created the Lua mod files properly and handled the basic /bounty flow inside Minetest. The code was not bad either. I just felt like it was not as clean or modular as what Opus 4.7 wrote.

But there was one really irritating issue.

Somehow, Kimi wrote the global Minetest config in ~/.minetest/minetest.conf with this:

secure.http_mods = bountykimi

But then it also created a world config and added a different mod name there.

So when I loaded the world, Minetest used the world-level config. That basically overrode the global config behavior I was expecting. Because of that, the HTTP API was not enabled for the actual mod that was running.

This took me more than half an hour to debug.

And honestly, because I do not have much experience with Minetest config behavior, this was super annoying.

The run itself was much cheaper and faster than Opus. Kimi used around 52k context tokens, took about 9 minutes 27 seconds, and cost around $0.39.

That price difference is pretty wild. Opus cost around $3.59 for the first test, while Kimi came in under $0.40.

You can find the code it generated here: Kimi K2.6 Code

Here's the demo:

Cost: ~$0.39
Duration: ~9min 27sec
Code Changes: +4,671 lines, 0 lines removed
Context Used: 52,073 tokens
Context Window Used: 20%

Quick Verdict

The local bounty board idea worked, the code was usable, and the model clearly understood the Lua + TypeScript setup. But if you notice, Kimi wrote more than twice as much code as Opus 4.7.

The main problem was the Minetest config mess. It added secure.http_mods = bountykimi globally, but then created another world-level config with a different mod name, which made debugging way more painful.

So yeah, Kimi passed the first test, but not as smoothly as Opus.

Test 2: Real Composio Integration

Now this is where the actual test, and the fun, begins.

The custom mod is ready, so now it is time to integrate Composio and give the game a quick agentic touch.

The idea is simple. As players progress through the game, their bounty completions get logged into Google Sheets with Composio.

Claude Opus 4.7

Claude Opus 4.7 did manage to add the real Composio integration, but this one was not as smooth as Test 1.

The backend could sync bounty completions to Google Sheets. The nice thing is that I did not even need to open Minetest to test whether it was working. Because the project was structured cleanly, I could test the whole backend flow with just two curl requests.

First, generate a bounty:

curl -s -X POST http://localhost:8787/api/bounty/generate \
  -H 'Content-Type: application/json' \
  -d '{"player":"singleplayer","availableTasks":["collect_item"]}' \
  | tee /tmp/b.json | jq

Then complete it:

curl -s -X POST http://localhost:8787/api/bounty/complete \
  -H 'Content-Type: application/json' \
  -d "$(jq -nc \
        --argjson b "$(jq .bounty /tmp/b.json)" \
        --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
        '{player:"singleplayer", bounty:$b, progress:{current:$b.target.count, required:$b.target.count}, completedAt:$ts}')" \
  | jq

And if everything is configured correctly, the second response looks like this:

{
  "ok": true,
  "message": "Bounty completed.",
  "leaderboard": {
    "player": "singleplayer",
    "points": 8,
    "completedBounties": 1
  },
  "sync": {
    "googleSheets": {
      "ok": true,
      "message": "Google Sheets row appended."
    }
  }
}

This is one of the things I love about Opus the most. It usually creates a pretty modular setup. The game mod, backend logic, and external sync were separated well enough that I could test the Composio part directly from the API without needing to run around inside the game every time.

It did run into a dev server issue where the tsx command was parsing watch incorrectly and treating it like the entry file.

After a bit of back and forth, it fixed the error. It eventually built a small runtime env loader and adjusted the config import so the backend could read the environment properly before the rest of the app booted.

After that, the build worked and the Google Sheets sync started working.

But that cost was painful. It literally cost me around $16. Like, actually :(. If you are not watching usage, this thing can make you broke real fast.

It took 28min 52sec API time, and about 1hr 17min wall time.

Apart from that, the code did work. But it cost way more than I expected for one run.

You can find the code it generated here: Claude Opus 4.7 Code - Composio

Here's the demo:

Cost: $16.03
Duration: 28min 52sec API time, 1hr 17min 40sec wall time
Code Changes: +1,848 lines, 507 lines removed
Token Usage:
- Input: 100.2k with Claude Haiku 4.5, plus Opus usage shown in the session
- Output: 3.2k with Claude Haiku 4.5, 123.3k with Claude Opus 4.7
- Cache read: 22.3M
- Cache write: 269.3k

Quick Verdict:

Claude Opus 4.7 got the real Composio integration working, especially the Google Sheets logging.

How cool is that? You add a custom agentic feature inside a game. A literal public game.

So yes, it worked. But $16 for this one run hurt.

Kimi K2.6

Kimi K2.6 did not do well on this test. It was pretty much busted.

From the start, it ran into a bunch of errors. The dev server broke, tests were failing, and even after a little handholding, it only managed to fix part of the test situation.

It eventually got past some of those failures, but the bigger problem was the actual Composio implementation. It did not seem fully sure how to wire the integration cleanly into the existing backend.

I had to stop and help again and again, but it still could not make meaningful progress with the build.

After spending more than 25 minutes and burning over 130k tokens, there was still no real progress. At that point, I had to stop the run.

Why on earth is it reading a version.txt file?

So yeah, I am calling this one a fail for Kimi K2.6.

Cost: ~$5.03
Duration: ~25min
Token Usage: 135,109+

Quick Verdict:

Kimi K2.6 failed this test.

It got stuck around tests, build issues, and the real Composio implementation. Even with manual help, it could not get the integration into a clean working state.

For the local bounty board, Kimi was surprisingly usable. But once the task moved into real external integration, it struggled a lot more.

Final Thoughts

Both Claude Opus 4.7 and Kimi K2.6 were pretty solid in this test, at least for the local version.

The task was not that simple either. It had Lua, TypeScript, SDK docs, backend logic, game commands, and the full flow had to work end to end.

Plus, the idea itself is not that common. Building an AI agent concept inside a custom game mod is not easy, and it is definitely not a one-shot thing, so props to both models.

Opus 4.7 did better overall. The code was cleaner, and as usual, Anthropic models are pretty good at that.

The only thing I hate with Anthropic models is the session limit.

I absolutely hate how little session usage you get. Opus 4.7 especially just eats through it completely in like 3 to 5 prompts.

Kimi K2.6 is an interesting model. Open models have not always been the best in my experience with real-world projects, but with every new model, my expectations rise a little.

Let's see where Kimi K2.6 goes from here.

Shrijal AcharyaFollow

SDE • GOLD @Microsoft Student Ambassador • Prev Lead Collab and Dev-Team Lead @oppiaorg • Mail for collaboration

My speaker broke, so I built a LAN speaker

Shrijal Acharya — Tue, 28 Apr 2026 14:29:00 +0000

What's happening?

This was around a year back when I started the project, after a small speaker that I had broke out of nowhere. Won't connect.

I used it to listen to music every night before going to sleep (not sure if anybody else does the same, but it's one of my only fixed routines since childhood :D).

The idea

Okay, so what's the idea?

I had a thought. Why not build something resembling a speaker? Although I had never worked on a project that required working with audio and all that, I knew what I'd use to implement something like that, which was WebSocket (I'll explain why I chose it in a moment).

And I did start on it.

The irony is that I had just finished Tour of Go, Let's Go, Let's Go Further, and had just started on 100 Go Mistakes and How to Avoid Them.

So... I had to do this in Golang.

I had quite a good idea of how to work with Golang, so it had to be done with it, also because I wanted to get away from the JavaScript ecosystem (the same Node.js, React, Next.js, yada yada was just too much).

Why WebSocket

So, let's come to the plan. Why WebSocket?

The main reason is that when I had the idea, it was WebSocket that came to my mind, and it's what I thought of before even starting to code the project.

And WebSocket kind of makes sense as well. This was my thought process.

Lets you connect multiple connections from one device? Yes
Is synchronous enough when streaming over multiple devices in LAN? Yes
Something I've worked with a lot? Totally Yes

That's pretty much the reason I chose WebSocket. It might not be the best option, but it's what I had and something I was confident with when I started.

One more thing I'll add here: since I'm streaming audio frames sequentially (PCM chunks, one after another), I actually need them to arrive in order.

WebSocket runs over TCP, so that's handled for me. I don't have to think about it. That alone made it feel like the right call.

Why not WebRTC or UDP?

Let me tell you something pretty frankly: I never even knew there was something like WebRTC. I came across this term while building the project halfway through it.

So, why not UDP?

Okay, so UDP is connectionless, which means there's no guarantee that every packet arrives, and more importantly, no guarantee they arrive in order. For something like video calls or games, that's totally fine. You drop a frame, move on, nobody cares.

But for audio? If a chunk goes missing or arrives out of order, you either get silence or a glitch.

And since I'm streaming raw PCM (basically just a stream of bytes that represent sound), every single chunk matters. You can't just skip them.

So UDP was out as well.

And WebRTC? Also, WebRTC is mostly built around browser-to-browser, peer-to-peer stuff. My setup is a server broadcasting to multiple clients over LAN, which isn't really what WebRTC is designed for. So even if I knew about it from day one, I'm not sure it would've been the right fit anyway.

WebSocket was fine. It worked. And sometimes that's enough. I didn't want to overengineer.

👀 I learned about all these new terms like PCM, WebRTC, and all that stuff during the build. So, I might say something wrong. I'm not really that familiar with them. So, just hit me in the comments if so.

The Architecture

Okay, so the high level is pretty simple.

There's a server and there are clients. The server is where the music is stored, and the clients are the devices that play it (could be the same device).

Here's what actually happens when you press play:

The server takes the MP3 file, decodes it into raw PCM (basically just bytes of sound data), and starts broadcasting those bytes over WebSocket to every connected client. No client-side decoding. The server does all of that.

Here's a high level architecture:

The tricky part is sync. If you just start streaming, each client will start playing at slightly different times, and it'll sound like a little echo.

So what I did is, before playback starts, the server sends a timestamp to all clients saying, "Start playing at exactly this moment in time." Every client gets that timestamp, buffers the audio frames, and waits. When the clock hits that time, everyone starts together.

It works because on a LAN with NTP, all the device clocks are usually within a millisecond or two of each other. Close enough that you can't tell the difference (much).

That's the whole thing, honestly.

Server decodes -> broadcasts -> clients sync -> play

What I still don't know

Okay, so during the build, I ran into something I didn't fully understand.

Turns out your speaker doesn't play audio the exact moment you write data to it. There's some time it spends sitting in a system buffer before it actually comes out. And that delay varies by OS and audio system. On Linux, it's something around 50ms, I feel.

Honestly, I had no idea there was something like this that you need to account for. This thing was completely debugged by GPT-5.4. There's a hardcoded 50ms constant in the code that is counted when clients actually start playing.

There's also something in the code that keeps checking, roughly every second, whether the audio playing on your device is slightly ahead or behind.

If it is, it quietly adds or removes a tiny bit of audio to bring it back in line. So small you'd never hear it.

Both of these kind of work. I tested it. I just couldn't tell you exactly why the numbers are what they are.

How I usually run it

Since my plan is to use my whole laptop as a "speaker", I usually have the server and client on the same system (my personal laptop).

And I usually have it like this:

Start the server:

gophercast serve # now follow the TUI setup....

Connect clients (each in a tmux split). Usually, I have around 2 clients when running on the same machine. More than 2 kind of distorts the audio quality.

gophercast play --host <ip_from_serve> --port 8080 --name "client 1"
gophercast play --host <ip_from_serve> --port 8080 --name "client 2"

The steps are the same when connecting from multiple machines. Just make sure that all of them are connected to the same LAN.

Here's a quick demo on a single machine, working as the speaker.
(Ignore the audio quality)

My take

I'm pretty happy with how this worked out.

This was my very first time working with Charm's Bubbletea and the whole audio stuff.

I started this project just because I wanted to DIY. It was just the perfect time: my speaker broke, and I had somehow finished learning basic Go programming.

There's probably still a ton of bugs. I've tried to test most of it. I just picked the tool I knew and figured it out as I went.

It plays music across multiple devices in sync, or the same one if you connect through multiple terminals. My laptop, another computer, whatever's connected, all playing together.

That's what I wanted. That's what it does.

(I haven't tested how well it works on Mac, so I can't tell there. Also, I'm not sure Windows will work either due to oto's limitation and our hardcoded 50ms delay.)

You can find the repo here: shricodev/gophercast

🚀 Want to build such cool stuff? You learn hands on here: CodeCrafters

Shrijal AcharyaFollow

SDE • GOLD @Microsoft Student Ambassador • Prev Lead Collab and Dev-Team Lead @oppiaorg • Mail for collaboration