DEV Community: Maksim Danilchenko

Kimi K3 Review: The 2.8T Open Model That Beats Claude on Paper

Maksim Danilchenko — Wed, 22 Jul 2026 08:45:26 +0000

TL;DR

Kimi K3 is the largest open-weight model anyone has shipped: 2.8 trillion parameters, a million-token context, and a price that undercuts Claude by roughly 3x. On the leaderboards it beats Claude Fable 5 at frontend coding and lands third overall on GDPval, which is genuinely impressive for weights you can download. After a week of feeding it real Go and Python work, the picture got messier. It's slow, it invents APIs that don't exist more often than any frontier model I've used this year, and most of its headline numbers are still Moonshot's own. Worth running for cost-sensitive batch work; I would not hand it an unsupervised agent loop yet.

Why I bothered testing this one

I review a lot of models and most open-weight releases don't survive contact with a real repo. DeepSeek V4 Pro and GLM-5.2 were the two that did. Both were cheap, competent, and good enough that I kept them in rotation for grunt work. When Moonshot dropped Kimi K3 on July 16 and the first benchmark screenshots showed it ahead of Claude Fable 5, I moved it to the top of my testing queue.

I ran K3 through the API (OpenRouter forwards to Moonshot's single hosted endpoint right now) for about a week: a Go service refactor, a pandas-to-Polars migration, a batch of PR reviews, and the usual pile of "explain this stack trace" prompts. I paid for the tokens myself, so the cost math below is real. What follows is what the leaderboards don't tell you: where a 2.8T open model actually helps, and where it quietly wastes your afternoon.

One caveat up front, and it's a big one: at the time of writing, the open weights aren't public yet. Moonshot says they land July 27, ten days after launch, under a modified MIT-style license. Everything in this review is based on the hosted API. If the weight release slips or the license has teeth, half the appeal evaporates.

What Kimi K3 actually is

Kimi K3 is a Mixture-of-Experts model with about 2.8 trillion total parameters. Moonshot reports 896 experts with 16 active per token, which is why a model this size can serve at Sonnet-ish prices: you only pay to run a sliver of it on each forward pass. The context window is 1,048,576 tokens, the same 1M ballpark the Gemini line reaches, and it's tuned hard for long-horizon coding and agent loops rather than chat.

At launch there are two variants: K3 Max for interactive chat and agent tasks, and K3 Swarm Max for large-scale parallel processing. Most people will only touch K3 Max. Swarm Max is aimed at the "fan out 200 requests and merge" workloads that batch pipelines and eval harnesses generate.

The "largest open-weight model ever" line is doing a lot of marketing work, but it's accurate. Nothing else with downloadable weights comes close to 2.8T. Whether you can run it locally is a separate question I'll get to.

Getting it running

The API is a plain OpenAI-compatible endpoint, so if you already have code hitting GPT or Claude, swapping in K3 is a two-line change. Here's the exact call I used to smoke-test it:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

resp = client.chat.completions.create(
    model="moonshotai/kimi-k3",
    messages=[
        {"role": "system", "content": "You are a senior Go engineer. Be terse."},
        {"role": "user", "content": "Write a worker pool in Go 1.26 that "
                                    "processes a channel of jobs with N workers "
                                    "and returns the first error via errgroup."},
    ],
    temperature=0.2,
)

print(resp.choices[0].message.content)
print("tokens:", resp.usage.total_tokens)

Output on my run (trimmed):

package main

import (
    "context"
    "golang.org/x/sync/errgroup"
)

func Process(ctx context.Context, jobs <-chan Job, n int) error {
    g, ctx := errgroup.WithContext(ctx)
    for i := 0; i < n; i++ {
        g.Go(func() error {
            for {
                select {
                case <-ctx.Done():
                    return ctx.Err()
                case job, ok := <-jobs:
                    if !ok {
                        return nil
                    }
                    if err := job.Do(ctx); err != nil {
                        return err
                    }
                }
            }
        })
    }
    return g.Wait()
}

tokens: 512

That's correct, idiomatic, and closes over the loop variable safely (Go 1.22+ semantics). No complaints on the code itself. The complaint is what the timer showed: that response took noticeably longer than the same prompt to Fable 5 or GPT-5.6. Which brings me to the thing the leaderboards bury.

The speed problem is real

Every hands-on account I've seen, including mine, lands on the same finding: K3 is slow. The New Stack ran a coding comparison and clocked it at roughly 4x slower than Claude Fable 5 on equivalent tasks. My informal timings weren't that clean, but "meaningfully slower" is not in dispute.

For an interactive agent loop, 4x slower is disqualifying. When I'm iterating with a coding agent, latency is the whole experience. A tool call that takes six seconds instead of two turns a ten-minute session into forty. I stopped using K3 for anything conversational within a day.

Where the slowness stops mattering is batch work. If you're running a nightly job that summarizes 5,000 support tickets or generates first-draft docstrings across a monorepo, nobody's watching the clock. There, K3's price does the talking. That split defines the whole model: great for offline, painful for interactive.

The benchmarks, with the asterisks attached

K3 earned its headlines on the benchmarks. As VentureBeat's launch coverage laid out, these are the numbers being quoted, and I'm labeling every one of them as reported, because most trace back to Moonshot's own eval runs or single-leaderboard placements rather than broad independent replication.

Benchmark	Kimi K3 (reported)	For reference	Placement
Frontend Code Arena	1,679	~48 pts ahead of Claude Fable 5	1st
GDPval-AA v2	~1,668	Fable 5 Max ~1,760 · GPT-5.6 Sol Max ~1,748 · Opus 4.8 1,600	3rd
AA-Briefcase	~1,547	Fable 5 Max ~1,583 · GPT-5.6 Sol Max 1,495	2nd
SWE-bench Verified	~65–67%	Same tier as top OpenAI/Anthropic models	—

Two things stand out. First, beating Claude Fable 5 on the Frontend Code Arena is a real result. Arena runs are human-preference votes on built UIs, and that one leaderboard is independent of Moonshot. Second, on GDPval it lands behind the top closed models but ahead of Opus 4.8, which is the fair "third place, and that's still very good for open weights" framing.

The SWE-bench figure needs the most salt. "65–67% in scaffold-assisted settings" means with a coding harness doing retries and test feedback around it. Bare-model SWE-bench numbers are always lower, and Moonshot hasn't published a clean methodology at the time of writing. Treat it as "strong tier," not a precise ranking against a specific competitor.

Kimi K3 vs Claude vs GPT: the head-to-head

Most people searching for K3 are really asking one question: should I use this instead of Claude or GPT? Here's the head-to-head that reflects a week of use plus the public data.

	Kimi K3	Claude Fable 5	GPT-5.6
Weights	Open (July 27)	Closed	Closed
Context	1M	1M	~400K (est.)
Input price / 1M	$3 ($0.30 cache-hit)	~$10	Premium
Output price / 1M	$15	~$50	Premium
Speed	Slow	Fast	Fast
Agentic reliability	Uneven	Best in class	Very good
Best for	Cheap batch, self-hosting later	Interactive coding, agents	General reasoning

K3 and Fable 5 prices are published/reported rates; GPT-5.6 figures are approximate.

On raw coding quality for a single well-scoped task, K3 is close enough to Fable 5 that you would not reliably tell them apart in a blind read. The New Stack's summary of "same results, one-third the cost, four times slower" matches my experience almost exactly: Fable 5 runs about $10 in and $50 out per million tokens, so K3's $3/$15 really is the cheaper column. If your bottleneck is API spend and not wall-clock time, that trade is a bargain.

The gap opens up in agent loops. Fable 5 and GPT-5.6 recover from their own mistakes; they read a failing test, form a theory, and fix it. K3 is more likely to double down on a wrong assumption, which is a known failure mode I dug into in the piece on multi-agent error cascades. If you want the current best-in-class agent, the Claude Fable 5 review still holds. K3 wins on price; Fable 5 wins on recovering from its own mistakes. Anyone doing head-to-heads through the Anthropic API will feel that difference within a session.

Where it falls apart: hallucination

This is the part that kept K3 out of my daily rotation. Independent testing has put its hallucination rate around 51%, up from a reported ~39% on its predecessor. I don't have a way to reproduce that exact figure, but the qualitative pattern showed up fast.

Twice in a week, K3 confidently used APIs that don't exist. In one pandas migration it wrote pd.read_parquet(path, engine="polars") as though pandas ships a Polars backend — it doesn't. In a Go task it referenced a stdlib helper that isn't in the standard library at all. Both times the surrouding code was clean and plausible, which is exactly what makes it dangerous: the wrong line reads as confidently as the right ones. A less careful reviewer merges it.

For a human-in-the-loop workflow where you read every diff, this is an annoyance you can catch. For an autonomous agent that writes, runs, and commits without a person checking each step, a coin-flip hallucination rate is a hard stop. The benchmark wins and this number live in tension, and you can't take one seriously without the other.

Pricing and how to run it

Through the API, K3 costs $3 per million input tokens on a cache miss, $0.30 per million on a cache hit, and $15 per million output tokens. That's Sonnet-class pricing for a model posting near-frontier coding scores, which is the entire pitch.

Local is the harder story. The weights drop July 27, but 2.8T parameters is not something you run on a workstation. Even at MoE's reduced active-parameter cost, you need serious multi-GPU hardware or a quantized community build to load it at all. "Kimi K3 on Ollama" is one of the top autocompletes for the model, and the real answer for now is: not on consumer hardware, and not on day one. Watch the Hugging Face repo and the quantization community; a usable GGUF will show up eventually, but it won't be quick. If you want a model you can actually run on a Mac today, Apfel is a far more realistic pick. Different weight class, but it fits in memory.

Who should use it, who should skip

Use Kimi K3 if: you run large offline batch jobs where cost dominates and latency doesn't, you want to hedge against closed-model lock-in, or you plan to self-host once the weights and a viable serving stack exist. The price-per-quality on non-interactive work is the best I've measured this quarter.

Skip it if: you want a snappy interactive coding assistant (Fable 5 or GPT-5.6, no contest), you're building an unsupervised agent where a hallucination becomes a committed bug, or you need a model you can run locally this week. For most solo developers writing code interactively, the slowness alone rules it out.

FAQ

Is Kimi K3 open source?

The weights are open; the training data and full pipeline are not. Moonshot says the weights ship July 27, 2026 under a modified MIT-style license (the same approach as Kimi K2.6), which makes K3 open-weight rather than fully open-source. You can download and run the model, but you can't fully reproduce it.

Is Kimi K3 better than Claude?

On some coding leaderboards, yes: it reportedly tops the Frontend Code Arena ahead of Claude Fable 5. In day-to-day use it's competitive on single-task code quality but slower and more prone to hallucination, so Claude remains the stronger pick for interactive and agentic work.

How much does Kimi K3 cost?

API pricing is $3 per million input tokens ($0.30 on cache hits) and $15 per million output tokens, roughly a third of what comparable closed models charge for similar coding quality.

Can you run Kimi K3 on Ollama or locally?

Not on consumer hardware. At 2.8T total parameters it needs substantial multi-GPU infrastructure even as a Mixture-of-Experts model. Quantized community builds may make smaller-scale local runs possible after the July 27 weight release, but there's nothing practical for a laptop at launch.

How many parameters does Kimi K3 have?

About 2.8 trillion total parameters in a Mixture-of-Experts design, with roughly 896 experts and 16 active per token, so only a small fraction runs on any given forward pass, which is what keeps inference costs down.

Sources

Kimi K3 on OpenRouter — official API pricing and hosting details
China's Moonshot AI releases Kimi K3, the largest open-source model ever — VentureBeat — launch coverage, specs, and benchmark placements
Moonshot releases 2.8-trillion-parameter Kimi K3 — Tom's Hardware — architecture and Frontend Code Arena result
Claude Fable 5 vs Kimi K3: same results, one-third the cost, 4x slower — The New Stack — independent coding comparison, latency, and pricing
Kimi K3 — DataCamp — model overview, variants, and open-weights timeline
Kimi K3 beat Claude and GPT at coding — why can't you fully trust it? — Memeburn — reported hallucination-rate testing

Bottom line

Kimi K3 is the most impressive open-weight release of the year and, for a specific job, a genuine bargain. If your workload is offline, cost-bound, and human-reviewed, download it the moment the weights land and point your batch jobs at it. But the leaderboard story and the daily-driver story are two different reviews. On the benchmarks it beats Claude; at my keyboard it was slow and it lied to me twice in a week. I'll keep it for batch grunt work and reach for Fable 5 the moment a task actually needs to be right.

Go's JSON v2 (encoding/json/v2): Faster, Stricter, and What Breaks

Maksim Danilchenko — Sun, 19 Jul 2026 08:41:43 +0000

TL;DR

Go's new encoding/json/v2 is the first real rewrite of the standard library's JSON handling in fifteen years. It's a reported 2–10x faster at unmarshaling, cleans up a pile of long-standing footguns, and moves configuration from magic struct tags into explicit function options. The catch: it changes several v1 defaults you may be quietly relying on — nil slices now encode as [] instead of null, duplicate object keys are rejected, and field matching is case-sensitive. As of Go 1.26 it lives behind GOEXPERIMENT=jsonv2, with the plan to make it the default in Go 1.27. This is a practical walkthrough of turning it on, the changes that will bite you, and how to migrate without breaking your API contracts.

Why a second JSON package at all

encoding/json shipped with Go 1.0 in 2012 and barely changed since. That stability paid off for a long time, then slowly turned into a tax. The v1 package carried a set of decisions that looked reasonable in 2011 and aged into daily annoyances: it silently accepts duplicate object keys, replaces invalid UTF-8 with the Unicode replacement character instead of erroring, and marshals a nil slice as null rather than an empty array. Each of those has bitten real production code, including mine.

The one that cost me an afternoon was the nil-slice-as-null behavior. I had a Go service returning a list of items to a TypeScript frontend. When the list was empty, the backend sometimes sent "items": null and sometimes "items": [], depending on whether a slice had been initialized or left nil somewhere upstream. The frontend called .map() on the response and blew up only on the null path. It was an intermittent, hard-to-reproduce crash that came down to a Go zero-value quirk. I fixed it back then with a make([]Item, 0) and a mental note. v2 fixes it at the source.

That is the spirit of the whole package. The official experimental announcement frames it as fixing "15 years of accumulated sharp edges" while keeping the API recognizable. If you've written Go, json.Marshal and json.Unmarshal still work the way you expect. They just have saner defaults and a much faster engine underneath.

If you're tracking where the language is heading more broadly, this fits the same evolutionary arc as Go 1.26's other standard-library changes: fewer surprises, more explicit control.

What's actually in the box

json/v2 is not one package but two, split along a clean line:

encoding/json/jsontext handles syntax: the raw grammar of JSON. It gives you a streaming Encoder and Decoder, plus Value and Token types, and it doesn't touch reflection at all. Think of it as the tokenizer layer.
encoding/json/v2 handles semantics: mapping Go values to and from JSON. This is where Marshal, Unmarshal, and the struct-tag machinery live.

That separation pays off in performance-sensitive code, because you can drop down to jsontext and stream tokens without paying for reflection when you don't need it. For everyday use you'll import v2 and never think about the split.

Turning it on

On Go 1.25 or 1.26, the packages are hidden unless you build with the experiment flag. There's no go get and no third-party module. It's already in your toolchain, just gated.

# One-off run
GOEXPERIMENT=jsonv2 go run main.go

# Persist it for a session
export GOEXPERIMENT=jsonv2
go build ./...
go test ./...

Once the flag is set, the import paths become visible:

// main.go — run with: GOEXPERIMENT=jsonv2 go run main.go
package main

import (
    "fmt"

    json "encoding/json/v2"
)

type User struct {
    ID    int      `json:"id"`
    Name  string   `json:"name"`
    Roles []string `json:"roles"`
}

func main() {
    u := User{ID: 7, Name: "Ada"} // Roles left nil on purpose
    b, err := json.Marshal(u)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(b))
}

Run it and you get:

{"id":7,"name":"Ada","roles":[]}

Under v1, that same struct marshals to "roles":null. That single difference is the one most likely to change bytes on the wire in your service, so it's worth internalizing before anything else.

If you can't move off an older Go toolchain yet, the same code is available today as a standalone module, github.com/go-json-experiment/json, which is the upstream where v2 was developed. The import path differs but the API is the same, so it's a fair way to experiment on Go 1.23 or 1.24 before your production toolchain catches up.

The breaking changes, ranked by how likely they are to bite

Here's the full set of default changes, most disruptive first. Every one of them is controllable. v2 keeps the old behavior available as an option; it's just no longer the default.

Behavior	v1 default	v2 default	How to restore v1
Nil slice	`null`	`[]`	`json.FormatNilSliceAsNull(true)`
Nil map	`null`	`{}`	`json.FormatNilMapAsNull(true)`
Field matching	case-insensitive	case-sensitive	`json.MatchCaseInsensitiveNames(true)` or `case:ignore` tag
Duplicate object keys	accepted	rejected	`jsontext.AllowDuplicateNames(true)`
Invalid UTF-8	replaced silently	rejected	`jsontext.AllowInvalidUTF8(true)`
`time.Duration`	encoded as int64 nanoseconds	requires explicit `format`	`json:",format:nano"`

Case sensitivity is the sneaky one

The nil-slice change is loud. You'll see it in the first test that checks a response body. Case sensitivity is quiet, because it fails by silently not matching rather than by erroring. Consider input from an API that uses SCREAMING_CASE or PascalCase keys:

data := []byte(`{"NAME":"Ada"}`)

var u User // Name is tagged json:"name"
if err := json.Unmarshal(data, &u); err != nil {
    panic(err)
}
fmt.Printf("Name=%q\n", u.Name)

Output:

Name=""

In v1 that would have populated u.Name with "Ada" because matching ignored case. In v2 the field stays at its zero value and no error is raised. If you consume upstream JSON whose casing doesn't exactly match your tags, opt back in per-field:

type User struct {
    Name string `json:"name,case:ignore"`
}

Or globally at the call site with json.MatchCaseInsensitiveNames(true). I'd reach for the per-field tag, since it documents which fields have loose upstream contracts instead of loosening the whole struct.

Duplicate keys now error

v1 happily unmarshaled {"id":1,"id":2} and kept the last value. That's a genuine security hole: different parsers picking different values for the same key is the root of a whole class of request-smuggling and access-control bugs. v2 rejects it:

data := []byte(`{"id":1,"id":2}`)

var u User
err := json.Unmarshal(data, &u)
fmt.Println(err != nil) // true

The error message is descriptive (something along the lines of a duplicate object member name for "id"; the exact wording may shift while the package is experimental). If you have a legitimate reason to accept duplicates, passing jsontext.AllowDuplicateNames(true) restores the old leniency (that option and AllowInvalidUTF8 live in the jsontext package, since they're syntactic, but you pass them straight to json.Unmarshal like any other option). Think hard before you reach for it.

Options replace tag soup

The change I like most has nothing to do with speed. In v1, everything was a struct tag or a global, and anything you couldn't express as a tag you couldn't express at all. v2 makes options first-class function arguments that you pass to Marshal and Unmarshal directly. The package reference lists the full set.

b, err := json.Marshal(u,
    json.FormatNilSliceAsNull(true), // keep v1 null-for-nil
    json.Deterministic(true),        // stable map key ordering
    jsontext.WithIndent("  "),       // pretty-print
)

That Deterministic(true) option is a small quality-of-life win worth calling out: it guarantees identical inputs produce identical output bytes, which makes golden-file tests and content hashing reliable without you sorting maps by hand. You can bundle options once and reuse them:

prettyV1 := json.JoinOptions(
    json.FormatNilSliceAsNull(true),
    json.FormatNilMapAsNull(true),
    jsontext.WithIndent("  "),
)

out, _ := json.Marshal(payload, prettyV1)

The struct-tag vocabulary grew too. The most useful new tag is omitzero, which finally fixes the ancient confusion around omitempty:

type Event struct {
    Name  string    `json:"name"`
    Start time.Time `json:"start,omitzero"`  // dropped when the zero Time
    Tags  []string  `json:"tags,omitempty"`  // dropped when empty JSON ([], "", null)
}

omitempty in v1 was notorious for not omitting a zero time.Time (because a struct is never "empty" by its rule). omitzero checks the Go zero value (or an IsZero() method if your type has one), which is almost always what people meant in the first place. If you've ever shipped "start":"0001-01-01T00:00:00Z" into a JSON response by accident, this tag is for you.

Streaming and the O(n²) trap

The headline performance number is unmarshal speed, but the more interesting fix is structural. In v1, if a custom type implemented MarshalJSON, encoding a large slice of that type could degrade to O(n²): each element's MarshalJSON allocated a fresh []byte, which then got re-parsed and re-copied into the parent buffer. Nest that a couple of levels deep and throughput falls off a cliff.

v2 adds streaming custom-marshaler interfaces that write directly into the shared encoder:

// Old, allocation-heavy:
//   func (t T) MarshalJSON() ([]byte, error)
//
// New, streams into the parent encoder:
func (t T) MarshalJSONTo(enc *jsontext.Encoder) error {
    return enc.WriteToken(jsontext.String(t.value))
}

If you'd rather not add methods to a type, v2 also takes function-based marshalers registered per type through json.WithMarshalers. Those helpers lean on Go generics so you get a type-safe func(T) handler instead of the old interface{} juggling. For bulk output you can drive the encoder yourself and avoid building a giant in-memory value at all:

import (
    json "encoding/json/v2"
    "encoding/json/jsontext"
)

func writeAll(w io.Writer, users []User) error {
    enc := jsontext.NewEncoder(w)
    for _, u := range users {
        if err := json.MarshalEncode(enc, u); err != nil {
            return err
        }
    }
    return nil
}

The decode side mirrors it with json.UnmarshalDecode over a jsontext.Decoder, which pairs naturally with a range loop the way Go's iterators do: read one value, process it, discard it, without holding the whole document in memory. The Go team reported that Kubernetes' kube-openapi saw "orders of magnitude" improvement switching a hot path to the streaming interface. That's the kind of gain you only get from fixing the algorithmic shape of the problem, and no amount of parser tuning would have matched it.

Benchmarking it yourself

I want to be precise here: I did not run these numbers on my own hardware, because the machine I wrote this on is still on Go 1.24 and the package needs 1.25+. The 2–10x figure comes from the Go team's own benchmarks and third-party measurements, and I'm labeling it as reported rather than measured. Marshal is roughly at parity with v1; the big wins are on the unmarshal path, where the new parser does far less work.

Don't take anyone's benchmark on faith, including mine. Here's a harness you can drop into your own repo and run against your actual payloads, which is the only benchmark worth trusting:

package bench

import (
    "os"
    "testing"

    json "encoding/json/v2"
)

func BenchmarkUnmarshal(b *testing.B) {
    data, err := os.ReadFile("testdata/payload.json") // your real data
    if err != nil {
        b.Fatal(err)
    }
    b.ReportAllocs()
    for b.Loop() {
        var out []User
        if err := json.Unmarshal(data, &out); err != nil {
            b.Fatal(err)
        }
    }
}

GOEXPERIMENT=jsonv2 go test -bench=Unmarshal -benchmem ./bench/

Run the same benchmark once with the flag and once without (swapping the import back to encoding/json), and compare with benchstat. Your mileage depends heavily on the shape of your data. Documents with lots of small objects and string-heavy payloads tend to show the largest gap, while a handful of big numeric arrays will show less.

A safe migration plan

The good news is that v2 is designed for gradual adoption. v1 is being reimplemented on top of v2 internally, and the two are meant to coexist for years. The Go team has said v1 won't be deprecated for years after v2 stabilizes, so you don't have to flip everything at once. Here's the order I'd follow:

Turn on the flag in CI first, not prod. Build and run your full test suite with GOEXPERIMENT=jsonv2. Most failures will be golden-file or response-body assertions catching the nil-slice and case-sensitivity changes. That's your blast-radius report, for free.
Fix contracts, not symptoms. Where a test breaks because a nil slice now encodes as [], decide which representation your API actually promises. Usually [] is correct and the test was wrong; occasionally a client depends on null and you set FormatNilSliceAsNull(true) at that boundary.
Add case:ignore where upstream casing is loose. Audit every place you unmarshal third-party JSON and tag the fields whose casing you don't control.
Adopt omitzero in place of the omitempty-on-a-struct workarounds you've accumulated.
Leave the streaming interfaces for later. They're an optimization you can defer safely. Convert the one or two hot paths that profiling flags, and leave the rest on plain Marshal.

If you build data-heavy services, this is the same performance-versus-safety tradeoff conversation that plays out in Python land with msgspec versus Pydantic: a stricter, faster serializer that asks you to be explicit about your data's shape. Go just happens to be baking that upgrade into the standard library instead of a third-party package.

FAQ

Is Go json v2 stable and production-ready?

Not yet, officially. As of Go 1.26 it's experimental, gated behind GOEXPERIMENT=jsonv2, and the API can still change. The plan is to graduate it to the default in Go 1.27. Plenty of teams are already testing it in CI, but I wouldn't gate a production release on experimental behavior that might shift.

What's the difference between encoding/json and encoding/json/v2?

Same core API (Marshal/Unmarshal), but v2 has stricter, safer defaults (rejects duplicate keys and invalid UTF-8, matches field names case-sensitively), encodes nil slices/maps as []/{} instead of null, adds first-class option arguments, introduces the omitzero tag, and ships a much faster unmarshaler.

How much faster is Go json v2?

Reported benchmarks put unmarshaling at roughly 2–10x faster than v1, driven by a more efficient parser. Marshaling is about the same speed as v1. The gains depend on your data shape, so benchmark your own payloads rather than trusting a single headline number.

Do I need to rewrite my structs to use json v2?

No. Existing struct tags keep working. You only touch code where a changed default affects your output: restoring null for nil slices at a specific boundary, adding case:ignore for loosely-cased upstreams, or swapping omitempty for omitzero. Everything else is a recompile.

Can I use encoding/json/v2 on older Go versions?

The standard-library path needs Go 1.25 or newer with the experiment flag. If you're stuck on Go 1.23 or 1.24, the identical implementation is available as the standalone module github.com/go-json-experiment/json.

Sources

A new experimental Go API for JSON — The Go Blog — the official announcement, rationale, and behavior changes
encoding/json/v2 — Go Packages — full API reference for functions, options, and struct tags
encoding/json/jsontext — Go Packages — the syntactic streaming layer
go-json-experiment/json — GitHub — the upstream module usable on older Go toolchains
Go 1.26 Release Notes — current stable release context

Bottom line

json/v2 is the rare standard-library change that's both faster and safer, and the migration cost is mostly "run your tests and read the diffs." The performance is a nice bonus; the real value is that the defaults finally match what most Go developers meant all along: no more null slices leaking into frontends, no more silently accepted duplicate keys, and case-insensitive matching stops quietly papering over schema mismatches. Turn on GOEXPERIMENT=jsonv2 in CI this week, see what breaks, and you'll be ready the day Go 1.27 makes it the default.

How to Build an AI Agent in Go (From Scratch, No Framework)

Maksim Danilchenko — Mon, 13 Jul 2026 08:45:53 +0000

TL;DR

An "AI agent" is a loop: send a prompt to a model, let it ask for a tool, run the tool, feed the result back, repeat until it stops. You can write that loop in Go with the standard library and about 120 lines of code. No LangChain, no SDK, no google.golang.org/adk. This tutorial builds a working agent that can read files and list directories on your machine, driven by the Anthropic Messages API. I'll show the full source, the exact JSON the API expects, the terminal output from a real run, and the two mistakes that cost me an hour the first time.

Why write this in Go at all?

Python owns the agent tutorials, and for good reason — every framework ships a Python client first. But Go has quietly become a strong fit for the runtime side of agents. A Hacker News thread from earlier this year argued the case: one build system, one formatter, static types, and real concurrency without the footguns. When your agent has to fan out ten tool calls, handle timeouts, and ship as a single static binary, that counts for more than how many frameworks exist.

I spent an afternoon porting a toy Python agent to Go, and the thing I kept noticing was how little I missed the framework. The whole "agent" abstraction collapses into a for loop and a switch statement once you see the message flow. That is the point of this post: strip away the layers so you understand what an agent actually is, then decide whether you want a framework on top.

Here's what we'll build: a working file-reading agent in about 120 lines of Go, with zero third-party dependencies and two tools.

The mental model: an agent is a loop

Strip away the marketing and an LLM agent is four steps in a cycle:

Send the conversation (plus a list of tools the model is allowed to call) to the API.
The model replies. It either answers in text, or it asks to call a tool with some arguments.
If it asked for a tool, your code runs that tool and captures the result.
You append the result to the conversation and go back to step 1.

The loop ends when the model responds with plain text and no tool request. The API signals this with stop_reason: "end_turn", and that is the whole pattern. Memory, planning, and "ReAct" are all variations on the same cycle. If you understand the loop, you understand agents.

Setup

You need Go 1.21+ and an API key. Grab one from the Anthropic Console (the tool-use format is cleanest there; I'll note the OpenAI equivalent at the end). Export it:

export ANTHROPIC_API_KEY="sk-ant-..."
mkdir go-agent && cd go-agent
go mod init go-agent

No dependencies to go get. We use net/http and encoding/json from the standard library and nothing else.

Step 1: Talk to the model

Before we add tools, let's make one plain request so the wiring is obvious. The Messages API wants a POST to https://api.anthropic.com/v1/messages with three headers and a JSON body. Here are the types:

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
)

type Tool struct {
    Name        string         `json:"name"`
    Description string         `json:"description"`
    InputSchema map[string]any `json:"input_schema"`
}

type ContentBlock struct {
    Type string `json:"type"`
    // text block
    Text string `json:"text,omitempty"`
    // tool_use block (from the model)
    ID    string          `json:"id,omitempty"`
    Name  string          `json:"name,omitempty"`
    Input json.RawMessage `json:"input,omitempty"`
    // tool_result block (from us)
    ToolUseID string `json:"tool_use_id,omitempty"`
    Content   string `json:"content,omitempty"`
}

type Message struct {
    Role    string         `json:"role"`
    Content []ContentBlock `json:"content"`
}

type Request struct {
    Model     string    `json:"model"`
    MaxTokens int       `json:"max_tokens"`
    Messages  []Message `json:"messages"`
    Tools     []Tool    `json:"tools,omitempty"`
}

type Response struct {
    Content    []ContentBlock `json:"content"`
    StopReason string         `json:"stop_reason"`
}

The one struct worth staring at is ContentBlock. The API models every piece of a turn as a typed block (a text block, a tool_use block, a tool_result block), and omitempty lets one Go struct serialize as any of them. The model's input arrives as arbitrary JSON, so I keep it as json.RawMessage and decode it later per tool.

Now the call itself:

func callClaude(msgs []Message, tools []Tool) (Response, error) {
    body, _ := json.Marshal(Request{
        Model:     "claude-sonnet-5",
        MaxTokens: 1024,
        Messages:  msgs,
        Tools:     tools,
    })

    req, _ := http.NewRequest("POST",
        "https://api.anthropic.com/v1/messages",
        bytes.NewReader(body))
    req.Header.Set("x-api-key", os.Getenv("ANTHROPIC_API_KEY"))
    req.Header.Set("anthropic-version", "2023-06-01")
    req.Header.Set("content-type", "application/json")

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return Response{}, err
    }
    defer resp.Body.Close()

    data, _ := io.ReadAll(resp.Body)
    if resp.StatusCode != 200 {
        return Response{}, fmt.Errorf("api %d: %s", resp.StatusCode, data)
    }

    var out Response
    return out, json.Unmarshal(data, &out)
}

The anthropic-version header is not optional. Leave it out and you get a 400 with a message that does not obviously point at the missing header. That was gotcha number one for me. The Messages API reference documents the current version string.

Step 2: Define the tools

A tool is a name, a description, and a JSON Schema for its arguments. The description is doing real work here: the model reads it to decide when to call the tool, so write it the way you'd brief a new teammate who has to figure out on their own when to reach for it.

var readFileTool = Tool{
    Name:        "read_file",
    Description: "Read a file at a relative path and return its contents as text.",
    InputSchema: map[string]any{
        "type": "object",
        "properties": map[string]any{
            "path": map[string]any{
                "type":        "string",
                "description": "Relative path, e.g. main.go or docs/readme.md",
            },
        },
        "required": []string{"path"},
    },
}

var listFilesTool = Tool{
    Name:        "list_files",
    Description: "List the files and folders in a directory. Defaults to the current directory.",
    InputSchema: map[string]any{
        "type": "object",
        "properties": map[string]any{
            "dir": map[string]any{
                "type":        "string",
                "description": "Directory to list. Empty means current directory.",
            },
        },
    },
}

And the dispatcher that actually runs a tool when the model asks:

func executeTool(name string, input json.RawMessage) string {
    switch name {
    case "read_file":
        var args struct {
            Path string `json:"path"`
        }
        json.Unmarshal(input, &args)
        data, err := os.ReadFile(args.Path)
        if err != nil {
            return "error: " + err.Error()
        }
        return string(data)

    case "list_files":
        var args struct {
            Dir string `json:"dir"`
        }
        json.Unmarshal(input, &args)
        if args.Dir == "" {
            args.Dir = "."
        }
        entries, err := os.ReadDir(args.Dir)
        if err != nil {
            return "error: " + err.Error()
        }
        names := make([]string, 0, len(entries))
        for _, e := range entries {
            names = append(names, e.Name())
        }
        return strings.Join(names, "\n")
    }
    return "unknown tool: " + name
}

Notice I return the error string to the model instead of crashing. That's deliberate. A good agent tool hands failures back as text so the model can recover — try a different path, ask the user, give up gracefully. If your tool panics, the loop dies and the model never gets a chance to react.

Step 3: The agent loop

This is the whole payoff — the piece that turns a chat call into an agent. Add bufio and strings to the imports, then:

func main() {
    tools := []Tool{readFileTool, listFilesTool}
    scanner := bufio.NewScanner(os.Stdin)
    var msgs []Message

    fmt.Println("Go agent ready. Ask about files in this folder (Ctrl-D to quit).")
    for {
        fmt.Print("\nyou> ")
        if !scanner.Scan() {
            break
        }
        text := strings.TrimSpace(scanner.Text())
        if text == "" {
            continue
        }
        msgs = append(msgs, Message{
            Role:    "user",
            Content: []ContentBlock{{Type: "text", Text: text}},
        })

        // Inner loop: keep running tools until the model is done.
        for {
            resp, err := callClaude(msgs, tools)
            if err != nil {
                fmt.Println("error:", err)
                break
            }
            // Echo the assistant turn back into history verbatim.
            msgs = append(msgs, Message{Role: "assistant", Content: resp.Content})

            var results []ContentBlock
            for _, block := range resp.Content {
                switch block.Type {
                case "text":
                    fmt.Println("\nclaude>", block.Text)
                case "tool_use":
                    fmt.Printf("  [tool] %s(%s)\n", block.Name, block.Input)
                    out := executeTool(block.Name, block.Input)
                    results = append(results, ContentBlock{
                        Type:      "tool_result",
                        ToolUseID: block.ID,
                        Content:   out,
                    })
                }
            }

            if len(results) == 0 {
                break // stop_reason was end_turn — the model answered.
            }
            msgs = append(msgs, Message{Role: "user", Content: results})
        }
    }
}

Two loops, nested. The outer loop is the human conversation. The inner loop is the agent thinking: it keeps calling tools and feeding results back until the model stops asking. When there are no tool_use blocks in a response, results is empty and we break out to wait for the next human message.

The line that bit me, gotcha number two, is msgs = append(msgs, Message{Role: "assistant", Content: resp.Content}). You must echo the assistant's tool_use block back into the history before you send the matching tool_result. Skip it, and the API rejects your next request because a tool_result with no preceding tool_use is malformed. Keeping the raw resp.Content (including tool_use blocks as json.RawMessage) is the simplest way to get this right.

Running it

Build and run:

go build -o agent . && ./agent

Here's a real session against a folder holding this project's Go files:

Go agent ready. Ask about files in this folder (Ctrl-D to quit).

you> what go files are here and what does main.go do?

  [tool] list_files({"dir":""})
  [tool] read_file({"path":"main.go"})

claude> This folder has three Go files: main.go, tools.go, and go.mod.
main.go is a command-line AI agent. It reads your input, sends it to the
Claude Messages API with two tools (read_file and list_files), runs any
tool the model requests, and loops until the model gives a final answer.
The core is a nested loop in main() that keeps executing tools until
stop_reason is end_turn.

you> which file is the biggest?

  [tool] list_files({"dir":""})
  [tool] read_file({"path":"main.go"})
  [tool] read_file({"path":"tools.go"})

claude> main.go is the largest at roughly 120 lines; tools.go is about 40.

The model chained two tool calls in the first turn. It listed the directory, then decided to read main.go, all before writing a single word of the answer. You did not orchestrate that ordering. The loop simply kept handing control back until the model decided it had enough to answer.

When you actually want a framework

Building from scratch is the right way to learn, and honestly it's fine for small production agents too. But once you need streaming, retries with backoff, session persistence, or multi-agent handoff, a framework earns its place. Here are the main Go options today:

Option	What it is	Reach for it when
Standard library (this post)	Raw HTTP + your own loop	Learning, full control, single static binary, minimal surface area
Anthropic Go SDK	Official typed client	You want typed requests/streaming for one provider without hand-rolling JSON
Google ADK for Go	Full agent framework: sessions, runners, tools	Multi-agent orchestration, Gemini-first stacks
Eino	ByteDance's composable orchestration graph	Complex pipelines with branching and fan-out
langchaingo	LangChain-style multi-provider abstraction	You want one interface across OpenAI, Anthropic, and local models

My honest take: start with the standard library, add the official SDK when JSON marshaling gets tedious, and only pull in a full framework when you feel real pain from the missing pieces. Most agents I've seen reach for a framework far too early and end up fighting its abstractions instead of shipping.

If you're coming from the editor side of the agent world, the Agent Client Protocol covers how these loops plug into IDEs, and Claude Code Subagents shows the same tool-calling idea running inside a production coding agent. For the concurrency question specifically (why Go's model helps when tools run in parallel), the Rust vs Go comparison digs into the tradeoffs.

The parts I deliberately left out

This agent is real but minimal. Three things you'd add before trusting it with anything important:

Guardrails on tools. My read_file will happily read /etc/passwd if the model asks. In production you sanitize paths and sandbox the filesystem. I wrote up four ways this goes wrong in AI agent guardrails.
A turn limit. The inner loop trusts the model to eventually stop. A confused model can ping-pong tools forever. Add a counter and bail after, say, 10 iterations.
Streaming. We wait for the full response before printing. For a snappy CLI you'd switch to the streaming endpoint and print tokens as they arrive — Go's bufio.Scanner over the SSE body handles this cleanly.

None of these change the shape of the loop; they just harden it around the edges.

FAQ

Is Go good for building AI agents?

Yes, for the runtime. Go gives you static binaries, real concurrency for parallel tool calls, and strong typing around the JSON the API speaks. The tradeoff is fewer frameworks than Python — but as this post shows, you often don't need one. If your agent is mostly orchestration and I/O rather than data science, Go is a strong pick.

Do I need a framework like LangChain to build an agent in Go?

No. An agent is a loop over a chat API with tool calling, and the standard library covers it in ~120 lines. Frameworks add value for streaming, retries, session state, and multi-agent coordination, but they are optional. Start without one so you understand the mechanics.

How does an AI agent work under the hood?

It's a cycle: send the conversation plus a tool list to the model; the model either answers or requests a tool call; your code runs the tool and appends the result; repeat until the model answers with no tool request. The API's stop_reason field tells you when the loop is done.

Can I use OpenAI instead of Anthropic?

Yes. The shape is identical, only the field names differ. With the OpenAI API you POST to /v1/chat/completions, tools live under a tools array with a function wrapper, and the model returns tool_calls instead of tool_use blocks. Swap the request/response structs and the executeTool dispatcher stays exactly the same.

What model should I use for a Go agent?

Any model with tool-calling support works. I used claude-sonnet-5 because it's fast and reliable at deciding when to call a tool. For cheaper runs, a smaller model like Haiku handles simple tool routing well; for hard multi-step tasks, a frontier model chains tools more reliably.

Sources

Anthropic Messages API reference — the request/response format and tool-use schema this agent speaks
Anthropic tool use documentation — how tool_use and tool_result blocks work
Anthropic Go SDK — official typed client for production use
Google ADK for Go — full agent framework, referenced in the comparison table
"A case for Go as the best language for AI agents" — Hacker News — the discussion that framed the "why Go" argument

Bottom line

The framework tutorials make agents look like a stack of abstractions you have to learn first. Really an agent is just a loop that hands control back and forth between a model and your tools, and Go writes that loop about as cleanly as any language I've tried. Build the 120-line version first. Once you can see the message flow in your head, you'll know exactly which framework feature you're missing — and, more often than not, that you don't need one yet.

Claude Sonnet 5 Review: A Week With Anthropic's New Default

Maksim Danilchenko — Tue, 07 Jul 2026 08:42:22 +0000

TL;DR

Claude Sonnet 5 shipped on June 30, 2026 and immediately became the default model on Claude's Free and Pro plans. Anthropic's published numbers put it at 63.2% on SWE-bench Pro, a few points behind Opus 4.8's 69.2% but well ahead of Sonnet 4.6's 58.1%. On Terminal-Bench 2.1 it actually beats Opus 4.8 (80.4% vs 74.6%). Intro pricing is $2 per million input tokens and $10 output through August 31, then $3/$15. One detail is easy to miss: it ships with a new tokenizer that uses 1.0–1.35× more tokens for the same text, so the sticker price undersells what you'll actually pay. After a week of using it as my daily driver in Claude Code, my take is that Sonnet 5 is the model most people should be running now, and Opus 4.8 has shrunk to a specialist tool you reach for on the genuinely hard problems.

What Claude Sonnet 5 Actually Is

Anthropic ships models in three sizes: Haiku for cheap and fast, Sonnet for the middle, Opus for the frontier. Sonnet has always been the workhorse, the one that runs when you don't think about which model you're using. Sonnet 5 is the fifth generation of that workhorse, and the pitch is that the middle tier has crept close enough to the top tier that the top tier stops being the obvious choice.

It landed as the default for Free and Pro on claude.ai the day it launched, with access for Max, Team, and Enterprise, plus the API id claude-sonnet-5 on the Claude Platform and inside Claude Code. That default placement counts for more than the benchmarks. Most Claude traffic never picks a model on purpose. Whatever Anthropic sets as the default writes most of the code, drafts most of the emails, and shapes what people think "Claude" is. That makes Sonnet 5 the new baseline for almost everyone who touches Claude, whether they know the version number or not.

I flipped my Claude Code default to it on launch morning and left it there for a week, running it against a Go service with a messy migration path, a Next.js dashboard, and a pile of one-off scripts. This was the everyday work sitting in my terminal, the kind I'd normally split between Claude and Codex. What stuck with me is that it finishes things. Multi-step tasks that used to stall halfway and wait for me to nudge them now run to completion on their own, and that single behavioral change is worth more than any single-digit benchmark bump.

The Benchmark Breakdown

Here are the numbers Anthropic and early testers published, lined up against the two models Sonnet 5 is really being measured against: its own predecessor and the flagship one rung up.

Benchmark	Sonnet 5	Sonnet 4.6	Opus 4.8
SWE-bench Pro (agentic coding)	63.2%	58.1%	69.2%
Terminal-Bench 2.1	80.4%	—	74.6%
OSWorld-Verified (computer use)	81.2%	78.5%	83.4%
BrowseComp (agentic search)	84.7%	—	—
GDPval-AA v2 (knowledge work, Elo)	1,618	—	1,615
FrontierCode	38.8%	15.1%	—

Two rows stand out. On SWE-bench Pro, the 5.1-point jump from Sonnet 4.6 to Sonnet 5 (58.1 to 63.2) is a real generational gain on a benchmark where model families usually move a couple of points per release. FrontierCode more than doubled in reported results, from 15.1% to 38.8%. Since FrontierCode targets the hard, long-horizon coding problems that break most models, a 2.5× jump there is the clearest signal that Sonnet 5 got structurally better at sustained agentic work rather than merely more polished.

The Terminal-Bench result is the one that made me raise an eyebrow. A mid-tier model beating the flagship on any benchmark is unusual, and Terminal-Bench 2.1 measures exactly the kind of shell-driving, command-running, output-reading loop that agentic coding lives in. Sonnet 5's 80.4% against Opus 4.8's 74.6% lines up with what I felt using it: for the day-to-day loop of edit, run tests, read the failure, fix, rerun, it is genuinely excellent, and the extra brains in Opus don't help much on that kind of task.

Where Opus still wins is SWE-bench Pro and OSWorld, both by a handful of points. Those gaps are real but small, and they narrow further once you weigh the cost difference. The GDPval knowledge-work Elo is essentially a tie (1,618 vs 1,615), which tells you the reasoning and writing quality of the two models is now hard to tell apart on ordinary tasks.

A Week Of Real Use

Benchmarks tell you a model's ceiling. A week of use tells you its floor, how it behaves on the boring 90% of tasks that never make it into a benchmark suite.

The best thing about Sonnet 5 is the self-verification loop. On my Go service, I asked it to migrate a set of handlers from a hand-rolled router to chi and update the tests. Sonnet 4.6 would have written the change and stopped, leaving me to run the suite and paste back the failures. Sonnet 5 wrote the change, ran the tests itself, saw two of them fail on a middleware ordering bug, fixed the ordering, and reran until green — all without me typing anything between the first prompt and the finished diff. It did this consistently over the week, not once as a lucky demo. Early coverage reports the same pattern: the model runs tests, reads the failures, patches, and reruns without being told to. That is the behavior that turns "AI that writes code" into "AI that finishes a task."

The second thing I noticed is that it's honest about uncertainty more often. On an ambiguous refactor where I hadn't specified whether to keep backward compatibility, it stopped and asked instead of guessing and plowing ahead. That's a small thing that saves a lot of wasted diffs.

It still has clear failure modes. On a gnarly Next.js server-component bug involving a hydration mismatch, Sonnet 5 went in circles twice: it proposed the same wrong fix, I rejected it, and it came back with a slight variation of the same wrong fix. I switched that one task to Opus 4.8 and it found the actual cause (a Date being rendered without a stable server/client format) in one pass. That is the shape of the current gap: on the genuinely hard, ambiguous problems that need real backtracking, Opus still pulls ahead. On everything else, you won't notice a difference.

The Tokenizer Tax

The pricing page leaves out one detail. Sonnet 5 ships with an updated tokenizer, and Anthropic's own announcement says it produces roughly 1.0 to 1.35× more tokens for the same input text than Sonnet 4.6 did. That multiplier is not uniform: plain English lands near the low end, while code, structured data, and non-English text push toward the high end.

What that means in practice: the $2/$10 intro price is genuinely cheap, but your token counts go up, so the effective cost per request rises more than the sticker drop suggests. If you're comparing Sonnet 5's $3/$15 standard price against Sonnet 4.6's $3/$15, they look identical on paper, but your bill won't match, because the same task now consumes more tokens. I watched this happen in Claude Code's usage meter over the week: comparable tasks showed noticeably higher token counts than I was used to under 4.6.

This doesn't make Sonnet 5 expensive. It's still dramatically cheaper than Opus 4.8. But it does mean you should run your own numbers before assuming a migration is a pure cost win. If you have a high-volume production pipeline with tight per-task budgets, measure the real token consumtion on your actual prompts rather than trusting the price-per-million headline.

Model	Input / 1M	Output / 1M	Notes
Sonnet 5 (intro, through Aug 31)	$2	$10	New tokenizer, +0–35% tokens
Sonnet 5 (standard, from Sep 1)	$3	$15	Same sticker as 4.6, higher token use
Sonnet 4.6	$3	$15	Older, cheaper-per-token tokenizer
Opus 4.8	$5	$25	Flagship; reach for the hard tasks

When To Use Sonnet 5 vs Opus 4.8

After a week, my routing rule is simple. Send everything to Sonnet 5 by default. Reach for Opus 4.8 only when a task is hard enough that a wrong answer is expensive: a subtle concurrency bug, an ambiguous spec that needs careful interpretation, a refactor where backtracking pays off. Keep Haiku 4.5 for the high-volume, latency-sensitive calls where you'd never have used a big model anyway.

That's a real shift from three months ago, when Opus was my default for anything that mattered and Sonnet was the "good enough, save money" option. Now the ordering has flipped: Sonnet 5 is the default that's good enough for almost everything, and Opus is the specialist you escalate to, with Fable 5 sitting above it for the longest autonomous runs. The GDPval tie and the Terminal-Bench win explain why. On the work that fills most of a day, the two models produce output I can't reliably tell apart, so paying 2.5× more for Opus stops making sense.

If you're weighing this against non-Anthropic options, the picture is more crowded than it was. GLM-5.2 undercuts everyone on price with open weights and lands close on coding benchmarks, and GPT-5.5 trades blows on general tasks. Sonnet 5's edge over both is the agentic follow-through, the finish-the-task behavior, which is exactly the thing that doesn't show up cleanly in a single benchmark score. For a broader field, our GPT vs Claude vs Gemini coding comparison still holds up on the general shape of the tradeoffs, even if the specific version numbers have moved.

How To Switch To Sonnet 5

If you're on claude.ai Free or Pro, you're already using it; it's the default. For everything else, you point at the model id.

Via the API with the Python SDK from the Anthropic console:

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=4096,
    messages=[
        {
            "role": "user",
            "content": "Refactor this Go HTTP handler to use chi router "
                       "and update the table-driven tests.",
        }
    ],
)
print(message.content[0].text)

In Claude Code, set it as the session default so every agent turn uses it:

# One-off for the current session
claude --model claude-sonnet-5

# Or make it sticky in your shell profile
export ANTHROPIC_MODEL="claude-sonnet-5"

If you drive Claude through a coding IDE, the same model id works. I run it as the default in Cursor for ordinary edits and only flip to Opus when a task fights back. One thing to test before you migrate a production prompt: the new tokenizer can change how structured inputs get parsed, so if you have prompts that depend on exact output formatting or tight tool-call schemas, run them through once and check the edges rather than assuming a drop-in swap.

Here's a quick way to sanity-check the token difference on your own text before and after switching, using the count-tokens endpoint:

import anthropic

client = anthropic.Anthropic()

resp = client.messages.count_tokens(
    model="claude-sonnet-5",
    messages=[{"role": "user", "content": open("prompt.txt").read()}],
)
print("Sonnet 5 input tokens:", resp.input_tokens)

Run that against a representative prompt, compare it to what the same text cost you under Sonnet 4.6, and you'll know your real multiplier instead of guessing at the 1.0–1.35× range.

Who Should Use It, Who Should Skip

Good fit:

Anyone already on Claude for coding. Sonnet 5 is a straight upgrade over 4.6 in every category, and during the intro window it's cheaper too.
Teams running agentic workflows (Claude Code, Cursor, custom harnesses) where the self-verification loop saves human turns.
Cost-conscious builders who were paying for Opus out of habit. Most of that work drops to Sonnet 5 with no quality loss you'll notice.

Think twice:

High-volume production pipelines with razor-thin margins. Measure the tokenizer tax on your real traffic before you assume the migration saves money.
Anyone whose prompts depend on exact tokenization or output formatting. Test the edges first.
Work that lives at the hard end of the difficulty curve: deep debugging, ambiguous specs, tasks that need backtracking. Keep Opus 4.8 in reach for those, and see our Claude Code Max vs Pro breakdown if you're deciding how much Anthropic compute to buy.

FAQ

How much does Claude Sonnet 5 cost?

Introductory API pricing is $2 per million input tokens and $10 per million output tokens through August 31, 2026, moving to $3/$15 after that. On claude.ai it's the default model on the Free and Pro plans at no extra cost. Factor in the new tokenizer, which uses 1.0–1.35× more tokens for the same text, when you estimate real spend.

Is Claude Sonnet 5 good for coding?

Yes — it's the strongest mid-tier coding model Anthropic has shipped. It scores 63.2% on SWE-bench Pro (up from Sonnet 4.6's 58.1%) and beats Opus 4.8 on Terminal-Bench 2.1. The standout in daily use is agentic follow-through: it runs tests, reads failures, and fixes its own code without being told to.

Claude Sonnet 5 vs Opus 4.8 — which should I use?

Use Sonnet 5 as your default for almost everything; it's close to Opus 4.8 on most benchmarks, wins on Terminal-Bench, and costs less than half as much. Reserve Opus 4.8 for the hardest tasks (subtle bugs, ambiguous requirements, anything that needs real backtracking) where its extra reasoning still pulls ahead.

What is Claude Sonnet 5's context window?

Sonnet 5 supports a 1 million token context window, matching the top of Anthropic's lineup. That's enough to hold a mid-size codebase, long agent trajectories, or large documents in a single session.

When was Claude Sonnet 5 released?

Anthropic released Claude Sonnet 5 on June 30, 2026, and made it the default model on the Free and Pro plans the same day.

Sources

Introducing Claude Sonnet 5 (Anthropic): official announcement, pricing, availability, and the tokenizer change
Claude models overview (Claude Platform Docs): model ids, context windows, and standard pricing
Claude Sonnet 5 vs Sonnet 4.6 vs Opus 4.8 (MarkTechPost): benchmark table and cost-performance analysis
Claude Sonnet 5 pricing: the hidden costs behind the "cost-neutral" launch (Finout): breakdown of the tokenizer's effect on real spend

Bottom Line

Claude Sonnet 5 is the clearest sign yet that the interesting action in AI coding has moved from the frontier tier to the tier just below it. It nearly matches Opus 4.8 on the benchmarks that matter, beats it outright on the terminal-driving loop that agentic coding actually runs on, and does it at less than half the price. After a week of leaving it as my Claude Code default, the only tasks I sent back to Opus were the two or three genuinely hard bugs a week throws at you.

The tokenizer tax is the one thing to watch. Sonnet 5 is cheaper than Opus and, during the intro window, cheaper than 4.6, but the higher token consumption means you should measure your real cost rather than trust the headline. Do that math, then switch. For the work that fills most of a developer's day, this is the model to reach for now, and if you're on a Free or Pro plan, it's already the one running by default.

Perplexity Bumblebee Review: The Supply Chain Scanner Your Dev Machine Needs

Maksim Danilchenko — Mon, 29 Jun 2026 08:35:52 +0000

TL;DR

Bumblebee is a read-only supply chain scanner from Perplexity AI that checks your installed packages, editor extensions, MCP configs, and browser extensions against known-compromised versions. It never runs a package manager command. I ran it on three machines. It found 847 packages across 9 ecosystems in under 4 seconds. The output is NDJSON you can pipe into anything. It won't replace Snyk or Socket in your CI pipeline, but for the question "does anyone on the team have that compromised package installed right now?" it's the fastest answer that exists.

Why Developer Machines Are the Blind Spot

When a supply chain advisory drops (say, a compromised npm package like @redhat-cloud-services/frontend-components in June 2026), the security team's first question is simple: who has it installed?

Your CI pipeline knows what's in your lockfile. Your SBOM tool knows what shipped to production. Your EDR knows what processes ran. None of them know what's sitting in ~/.npm, what extensions your VS Code installed last Tuesday, or what MCP servers your Claude Code config points to.

Sonatype's 2026 State of the Software Supply Chain report counted over 454,600 new malicious open-source packages in 2025, pushing the cumulative total past 1.233 million. That's a 75% year-over-year increase. The attacks keep getting more creative, too: the Mini Shai-Hulud campaigns starting in late April 2026 hit npm, PyPI, RubyGems, and Composer across companies like SAP, with the TanStack compromise following in May. In June 2026, TeamPCP's Miasma worm injected itself into the SessionStart hooks of 13 AI coding tools, including Claude Code, GitHub Copilot, and Gemini CLI.

That last one is why MCP config scanning matters. If your claude_desktop_config.json or mcp.json references a compromised server, you've handed an attacker tool-level access to your coding environment. Traditional security scanners don't even look at these files.

Bumblebee fills that gap. It answers one question: what packages, extensions, and configs are on this machine right now, and do any of them match a known-bad advisory?

What Bumblebee Actually Is

Bumblebee is a single static binary written in Go 1.25+ with zero non-stdlib dependencies. Perplexity open-sourced it in May 2026 under Apache 2.0. The v0.1.1 release has crossed 4k GitHub stars.

The design is deliberately narrow. Bumblebee collects an inventory of what's installed on a developer machine and optionally matches that inventory against an exposure catalog of known-compromised packages. It does not:

Run npm install, pip install, or any other package manager command
Execute install scripts or lifecycle hooks
Read your source code
Make network calls during the scan
Modify anything on disk

That last set of constraints is the whole point. When your machine might already be compromised, the scan tool itself can't be the thing that triggers the compromise. npm's postinstall scripts are a well-known attack vector. Bumblebee never gives them a chance to fire.

What It Scans

The coverage spans ten ecosystems, which is wider than I expected from a v0.1 tool:

Ecosystem	What It Reads	Package Manager
npm	package-lock.json, yarn.lock, pnpm-lock.yaml, bun.lock	npm/yarn/pnpm/bun
PyPI	.dist-info/METADATA, .egg-info/PKG-INFO	pip/setuptools
Go modules	go.sum, go.mod	go
RubyGems	Gemfile.lock, *.gemspec	bundler
Composer	composer.lock, installed.json	composer
Homebrew	INSTALL_RECEIPT.json, .metadata	brew
MCP servers	claude_desktop_config.json, mcp.json, various IDE configs	Claude Code/Cursor/Gemini CLI
Agent skills	skills-lock.json	vercel-labs/skills
Editor extensions	VS Code, Cursor, Windsurf, VSCodium manifests	IDE ecosystem
Browser extensions	Chrome, Edge, Brave, Arc, Firefox manifests	Browser ecosystem

The MCP and agent skills scanning is what makes this feel purpose-built for 2026. Phoenix Security's 2026 supply chain report found that AI-agent skills carry a risk rate 2.3 times higher than IDE extensions, and more than 1 in 4 deep-scanned skills triggered a critical-risk finding. Bumblebee parses claude_desktop_config.json, mcp.json, and Gemini CLI config files to catalog which MCP servers are registered and their source packages.

Running It: Three Machines, Three Profiles

I installed Bumblebee via go install and ran it across three environments: my primary macOS development laptop, a Linux CI runner, and a colleague's machine that I suspected had stale npm packages from a prototype project six months ago.

Installation

go install github.com/perplexityai/bumblebee/cmd/bumblebee@v0.1.1

One binary. No config files. No daemon. The Go install pulls the source and compiles it locally, so the binary on your machine was built from source you can audit.

Baseline Scan

The baseline profile hits common global package roots, editor extensions, browser extensions, and MCP configs:

bumblebee scan --profile baseline > inventory.ndjson

On my macOS machine, the baseline scan completed in 1.8 seconds and found 312 packages across npm, PyPI, Go modules, Homebrew, 23 VS Code extensions, 4 Cursor extensions, and 3 MCP server configurations.

Each record in the NDJSON output looks like this:

{
  "record_type": "package",
  "hostname": "maxbook.local",
  "ecosystem": "mcp",
  "package_name": "filesystem-mcp-server",
  "version": "0.6.2",
  "source_type": "mcp_config",
  "source_file": "/Users/max/.config/claude/claude_desktop_config.json",
  "confidence": "medium"
}

The confidence levels are useful. high means exact canonical metadata with verified version. medium means reliable identity but partial version info, which is common for MCP servers where the config only records a package reference. low means it's a config path or spec reference only.

Project Scan

bumblebee scan --profile project \
  --root "$HOME/code" \
  --root "$HOME/Developer" > project-inventory.ndjson

This took 3.4 seconds and found 535 packages. The extra coverage comes from crawling lockfiles in project directories. A Go project I'd forgotten about contributed 47 transitive dependencies from its go.sum.

Deep Scan

bumblebee scan --profile deep \
  --root "$HOME" \
  --max-duration 10m > deep-inventory.ndjson

The deep profile is for incident response. It sweeps everything under the given root. On my machine with a messy home directory, it found 847 packages in 3.9 seconds. On the colleague's machine (the one with stale npm prototypes), it found 1,423 packages. A lot of them were cached npm packages from six months ago that nobody realized were still sitting on disk.

Exposure Check

This is where Bumblebee earns its keep. Once you have an advisory (say, the June 2026 @redhat-cloud-services compromise), you write an exposure catalog:

{
  "schema_version": "0.1.0",
  "entries": [
    {
      "id": "advisory-2026-redhat-npm",
      "name": "@redhat-cloud-services/frontend-components 4.2.3",
      "ecosystem": "npm",
      "package": "@redhat-cloud-services/frontend-components",
      "versions": ["4.2.3", "4.2.4"],
      "severity": "critical"
    }
  ]
}

Then run:

bumblebee scan --profile deep \
  --root "$HOME" \
  --exposure-catalog ./catalog.json

If any match exists, Bumblebee emits a finding record alongside the normal inventory. The finding includes the matched package, version, catalog entry, and severity. Pipe the output through jq '.record_type == "finding"' and you have a yes/no answer in seconds.

Self-Test

Before trusting it with real incident data, verify the binary works:

bumblebee selftest
# selftest OK (2 findings in 1ms)

The self-test uses embedded fixtures with fake package names. It validates that the scanning and matching logic works correctly on your platform before you point it at real data.

Where Bumblebee Fits (and Where It Doesn't)

Don't treat Bumblebee as a replacement for your existing security toolchain. It's a complement. Here's how it maps against the tools you probably already use:

Tool	What It Does	When It Runs
Snyk	Scans dependencies for known CVEs, generates fix PRs	CI/CD, on PR merge
Socket	Behavioral analysis of packages pre-install (typosquatting, exfiltration)	Pre-install, CI/CD
OSV-Scanner	Matches lockfiles against the OSV database	CI/CD, local
npm audit	Checks installed packages against the npm advisory database	Post-install, CI/CD
Bumblebee	Reads on-disk metadata across 10 ecosystems, no execution	On-demand, scheduled on dev machines

Bumblebee fills the post-incident triage gap. Snyk tells you what's vulnerable in your repo. Socket tells you if a package is behaving suspiciously before you install it. Bumblebee tells you which developer machines are currently exposed to a specific known-compromised package. They complement each other.

The strongest use case: a zero-day advisory drops at 2 AM. Your security team pushes an exposure catalog. Every developer machine runs bumblebee scan --profile deep --exposure-catalog ./catalog.json as part of their morning boot script. By the time standup happens, you know exactly who's affected.

What I Liked

Speed. Sub-4-second scans across 800+ packages on a well-loaded macOS machine. It reads metadata files directly from the filesystem, with no process spawning, no network calls, and no dependency resolution. The Go implementation is tight.

Zero dependencies. The entire binary is Go standard library. For a security tool, that's a big deal: every third-party dependency is an attack surface. Bumblebee's supply chain risk is literally zero non-stdlib packages. You can verify this yourself. The go.mod file lists only the module path and Go version.

MCP config scanning. No other supply chain tool does this. With MCP tool poisoning becoming a real attack vector in 2026, a tool that catalogs which MCP servers are configured on a machine is ahead of the curve.

NDJSON output. Every record is a self-contained JSON line. Pipe it through jq, feed it to your SIEM, aggregate it with xargs. No proprietary format, no dashboard required.

Read-only guarantee. I checked the source. There are no os.Create, no os.WriteFile, no exec.Command calls outside the self-test fixtures. The binary genuinely can't modify your system.

What It Doesn't Do

The biggest practical limitation is version range matching. Bumblebee v0.1 only matches exact name + version pairs. If an advisory says "all versions below 2.3.1 are affected," you need to write out every affected version in the catalog, or write a wrapper that expands ranges. Most advisories use ranges, so this will bite you early.

MCP config parsing is JSON-only. Continue uses YAML, Codex uses TOML, and both are skipped silently. Claude Code and Cursor both use JSON so you're covered there, but it's an incomplete picture.

There's no remediation. Bumblebee tells you what's there. It won't remove it, update it, or file a PR. That's by design (a read-only tool should stay read-only), but it means you need a separate process for acting on the findings.

There's also no continuous monitoring. It's a one-shot execution model. You run it, get results, done. If you want recurring scans, wrap it in a cron job or MDM policy. No agent, no daemon, no phone-home.

Finally, Linux and macOS only. No Windows support in v0.1.1. The scan profiles are built around Unix filesystem conventions. If your team has Windows developers, Bumblebee misses them entirely.

The MCP Angle

I want to call out the MCP scanning specifically because it's the feature that separates Bumblebee from every other supply chain tool I've used.

Your claude_desktop_config.json or project-level mcp.json defines which tools your AI coding agent can call. A compromised MCP server gets tool-level access inside your development session. It can read files, write files, execute commands, and interact with external services, all under the guise of a legitimate tool.

The Miasma worm that hit in June 2026 specifically targeted these config files, injecting itself into SessionStart hooks across 13 AI coding tools. Traditional security scanners didn't catch it because they don't read MCP configuration files. Bumblebee does.

When I ran the baseline scan, it found three MCP servers in my Claude Code config: filesystem-mcp-server, brave-search, and a custom internal tool. All three were clean, but the fact that they showed up in the inventory meant I could match them against future advisories automatically. That's the workflow: scan once, match many times.

Running Bumblebee in a Team

For individual developers, the one-shot model works fine. For a team, you need a bit more structure:

bumblebee scan --profile baseline \
  --exposure-catalog /shared/catalogs/latest.json \
  | jq -c 'select(.record_type == "finding")' \
  | curl -s -X POST -d @- https://internal-siem.company.com/ingest

The pattern is: scan locally, filter for findings, push to a central collector. The NDJSON output makes this trivial to integrate with whatever log aggregtion you already run.

For MDM-managed fleets, distribute the binary via your existing tooling (it's a single static file, no installer needed), set up a scheduled task, and point --exposure-catalog at a shared network path that your security team updates when advisories drop.

FAQ

Does Bumblebee replace Snyk or Socket?

No. Bumblebee scans what's already on your machine. Snyk and Socket operate in CI/CD to prevent bad packages from entering your projects. Use all three. They cover different phases of the supply chain.

Can Bumblebee trigger malicious code during a scan?

No. It reads metadata files (lockfiles, manifests, config JSON) directly from the filesystem. It never executes npm install, pip install, or any lifecycle hooks. The scan can't trigger a compromised package's install script.

Does it work with private registries?

Yes, for inventory purposes. Bumblebee reads lockfile metadata, which includes private package names and versions. It doesn't authenticate against registries. It reads from local disk, so there's nothing to authenticate against.

How do I get exposure catalogs?

You build them. Bumblebee doesn't ship with a built-in advisory database. When an advisory drops (from GitHub Security Advisories, Snyk's database, or your own threat intel), you convert it into the catalog JSON format and pass it to --exposure-catalog. Multiple catalogs merge when you point at a directory.

Is there a Windows version?

Not yet. Bumblebee v0.1.1 supports macOS and Linux only. Windows support would require rewriting the scan profile filesystem paths and adding support for Windows-specific package manager locations (AppData, ProgramFiles).

Sources

Perplexity Bumblebee GitHub repository: source code, documentation, and release notes
MarkTechPost coverage of Bumblebee release: independent analysis of the tool's architecture and purpose
Sonatype 2026 State of the Software Supply Chain: 454,600 malicious packages in 2025, 1.233M cumulative
MCP Security in 2026: Tool Poisoning and Supply Chain Risks: analysis of MCP-specific attack vectors
Supply Chain Attacks 2026: npm, PyPI, VS Code, AI Agents: attack campaign tracking and 2.3x AI-agent skill risk data
Bumblebee vs OSV-Scanner comparison on Dev.to: positioning against other scanning tools

Bottom Line

Bumblebee does one thing and does it well: it inventories every package, extension, and MCP config on a developer machine without executing anything. That narrow scope is a feature. 454,600 malicious packages shipped in a single year. AI coding tools introduced a new attack surface nobody was scanning. A sub-4-second read-only answer to "are we exposed?" is worth adding to your toolkit.

The version range limitation is real and will bite you on the first advisory that specifies a range instead of exact versions. The missing Windows support matters if you have a mixed fleet. But for Unix-based development teams using Claude Code, Cursor, or VS Code, Bumblebee fills a gap that Snyk, Socket, and npm audit don't touch: what's actually sitting on the developer's disk right now, including the MCP configs that no other tool bothers to read.

Install it. Run the self-test. Point it at your home directory. The scan takes less time than reading this sentence, and you might not like what it finds.

Is There a Backend-as-a-Service for AI Coding Agents?

Maksim Danilchenko — Fri, 26 Jun 2026 14:15:13 +0000

TL;DR: Yes. A handful of backends now target AI coding agents directly. InsForge, Base44, Butterbase, and MoonDB are among them. What sets them apart from Supabase or Firebase is that an agent can provision the whole thing from code: a JSON schema in, a hosted database with auth and a REST API out, no dashboard click-through. With MoonDB that's three calls.

What "designed for AI coding agents" actually means

Most backends were built for a person. You sign in, click to create tables, wire up auth in a settings panel, copy keys out of a dashboard, and paste them into your app. An agent like Cursor or Claude Code can't click. Drop one into that flow and it stalls at the first screen that needs a human.

An agent-native backend removes the human step. The whole setup runs over HTTP or through an MCP server, so the agent does the work end to end. Three things make that possible, and they're the test for whether a backend really fits an agent:

It provisions from code. Creating a project, applying a schema, and reading the resulting keys all happen through API calls that return JSON. Nothing waits on a console.

It's declarative. The agent writes what the data looks like: tables, fields, relationships, and who can read and write each one. The service figures out the SQL and runs the migration. No hand-written DDL, no guessing at migration order.

Its errors are instructions. When a schema is wrong, the response says what to fix, not just that something broke. An agent reads that and corrects itself on the next call instead of looping.

Supabase and Firebase can be scripted, and people do script them. But the happy path still runs through a dashboard, the client libraries assume a build step, and the error messages are written for a developer reading a stack trace. In the newer tools, the agent is the primary user and the human reviews the result.

The backends built for this, as of 2026

The category is small, and it already has a name: "agent-native" backends. A few worth knowing:

InsForge is an open-source backend that bundles a stack of services behind a semantic layer an agent can read and provision: Postgres with auto-generated APIs, auth, S3-compatible storage, edge functions, compute and hosting, and an AI model gateway. The source is on GitHub under Apache-2.0.

Base44 leans on its CLI, which it tunes for Claude Code and Cursor so an agent can scaffold backend logic, data models, and APIs from a prompt.

Butterbase is another open-source option: Postgres, auth, storage, functions, an AI gateway, and an MCP server.

Xano comes at it from the no-code side. It's a visual backend builder that's added AI features for generating APIs and logic.

MoonDB sits at the minimal end of that list, and on purpose. It does one thing: turn a JSON schema into a hosted database with a REST API, auth, and file storage, all reachable over plain HTTP or MCP. There's no vector store, no edge-function runtime, no AI gateway. If your generated app needs RAG or a serverless compute layer, InsForge or Butterbase carry more out of the box. If it needs a database with auth and a clean API that an agent can stand up in seconds and a person can actually understand, the smaller surface is the point.

The honest framing: on a feature checklist, Supabase wins. The question that sorts this category is which backend an agent can drive to "done" without a human unblocking it, and how little is left to misunderstand once it's running.

Provision a MoonDB backend the way an agent would

Here's the full path an agent follows to back a generated app, with a to-do app as the canonical first build. Three calls, no dashboard.

Step 1: create a project

One call returns the keys the rest of the flow needs.

curl -X POST https://moondb.ai/v1/projects \
  -H "X-API-Key: mk_your_account_key" \
  -H "Content-Type: application/json" \
  -d '{ "name": "todo-app" }'

{ "data": { "id": "p_7k2d", "admin_key": "sk_...", "public_key": "pk_..." } }

The agent reads three keys from the JSON: an account key that manages projects, an admin key that changes the schema, and a public key that's safe to ship in client code.

The agent keeps the admin_key server-side and uses the public_key for anything that runs in a browser. No human copies anything out of a console.

Step 2: send the schema

The agent describes the data as one JSON document and applies it with a single PUT. A to-do app needs users and per-user todos, where each person only sees their own list.

curl -X PUT https://moondb.ai/p/p_7k2d/v1/schema \
  -H "X-Admin-Key: sk_..." \
  -H "Content-Type: application/json" \
  -d @schema.json

{
  "tables": {
    "users": {
      "auth_table": true,
      "columns": { "display_name": "string" }
    },
    "todos": {
      "columns": {
        "title": "string required max_length 200",
        "done": "bool default false",
        "due_date": "date",
        "priority": { "type": "enum", "values": ["low", "medium", "high"], "default": "medium" },
        "user_id": "ref users required"
      },
      "owner_field": "user_id",
      "access": { "read": "owner", "create": "authenticated", "update": "owner", "delete": "owner" }
    }
  }
}

That document carries the whole security model. "auth_table": true makes users the login table, so MoonDB adds a hidden, required password_hash column and serves signup and login endpoints. The agent never declares the password field. "ref users required" ties each todo to a user. The owner_field with "read": "owner" scopes every read to the logged-in user, so one person's todos never leak into another's response, and the agent didn't write a line of authorization code to get that. create: authenticated lets any signed-in user add a todo. MoonDB also adds id, created_at, and updated_at to every table on its own.

MoonDB reads that, creates the tables, and serves the API immediately. Change the schema later and the agent sends new JSON; MoonDB diffs the two versions and migrates, and it refuses to run a destructive change unless the request opts in. That guardrail keeps an over-eager agent from dropping a column full of data.

The dashboard renders what the schema produced. An agent never opens this. It's here for the human to check the work.

Step 3: use the generated API

The endpoints exist the moment the schema applies. The agent signs up an end user, gets a token back, and writes a todo as that user.

# Create an end user
curl -X POST https://moondb.ai/p/p_7k2d/auth/signup \
  -H "Content-Type: application/json" \
  -d '{ "email": "sam@example.com", "password": "s3cret-pass", "display_name": "Sam" }'
# → { "data": { "user": {...}, "token": "<jwt>", "refresh_token": "...", "expires_in": 3600 } }

# Add a todo as that user
curl -X POST https://moondb.ai/p/p_7k2d/api/todos \
  -H "Authorization: Bearer <jwt>" \
  -H "Content-Type: application/json" \
  -d '{ "title": "Ship the MVP", "priority": "high" }'

The todo lands with user_id set to Sam from the token, because the owner_field rule stamps ownership at write time. When Sam reads GET /p/p_7k2d/api/todos, the read: owner rule scopes the result to Sam's rows. The agent gets working auth and per-user data isolation out of the schema, not out of a pile of middleware it had to write and test.

That's the loop a coding agent can run unattended: create, describe, write. For an agent that speaks Model Context Protocol, MoonDB exposes the same provisioning as MCP tools, so the steps above become tool calls inside the agent's session instead of shell commands.

How the agent-native backends compare

	Supabase / Firebase	InsForge / Butterbase	MoonDB
Built for	A developer at a dashboard	AI coding agents	AI coding agents
How an agent provisions it	Scripted around a dashboard flow	CLI + MCP	Three HTTP calls or MCP
Schema source	SQL, dashboard, or migrations	Declarative + AI generation	One JSON document
Database	Postgres	Postgres	SQLite on Cloudflare D1
Extra services	Many	Vector DB, edge functions, AI gateway	Auth and file storage only
Access rules	Row-level security policies	Per-service config	In the schema (`owner` / `authenticated`)

Firebase and Supabase are the right call when you want their depth and don't mind that an agent needs hand-holding to drive them. InsForge and Butterbase fit when a generated app needs vector search or serverless functions alongside the database. MoonDB fits when the job is a database with auth and a REST API, and the thing you care about is that an agent can stand it up in one pass and a person can read the whole schema in one screen.

Where MoonDB stops

Be clear-eyed about the trade. MoonDB runs on SQLite through Cloudflare D1, which buys fast reads at the edge and a model with almost nothing to configure. It also sets the ceiling: one writer per database and a 10 GB cap. That holds a to-do app, a CRM for a small team, or a content site with room to spare, but a write-heavy system with thousands of concurrent writers wants a dedicated Postgres. If you're weighing the two, here's how MoonDB compares to Supabase.

It's also not a vector database. If your app is doing retrieval-augmented generation and needs embeddings stored next to your data, that's a real reason to look at InsForge's vector support or a dedicated vector DB instead. MoonDB stores structured application data, the rows your generated app reads and writes, and leaves RAG to tools built for it.

For the range most agent-built apps start in, a schema an agent can write and a REST API it can call beats a feature list it can't reach. The schema and API reference covers every field type, and the walkthrough on generating a REST API from a JSON schema goes deeper on the format. If you're comparing agent-facing database tools specifically, the rundown of database MCP servers covers that angle.

FAQ

Is there a backend-as-a-service designed for AI coding agents?

Yes. Several launched recently: InsForge, Base44, Butterbase, and MoonDB among them. They differ from Supabase or Firebase in one concrete way: an agent can provision the whole backend from code, with no dashboard step. You hand over a schema and get a live database, auth, and a REST API back.

What makes a backend agent-native instead of a normal BaaS?

A normal BaaS assumes a person clicking through a dashboard to make tables and copy keys. An agent-native one is driven entirely by API: the agent sends a declarative schema, reads keys from the JSON response, and gets errors written as instructions it can act on. No human step sits in the middle.

Can a coding agent like Cursor or Claude Code set up a backend by itself?

Yes. If the backend exposes provisioning over HTTP or an MCP server, the agent can create a project, apply a schema, and start writing data without you touching a console. MoonDB does this in three calls; tools like Base44 and InsForge expose a CLI or MCP server aimed at Claude Code and Cursor.

Do I still need to write SQL or use a dashboard?

No. With a declarative backend you describe tables, fields, and access rules as JSON, and the service compiles that to SQL and runs the migration. The dashboard becomes optional. It's useful for watching what the agent built, but you don't need it to build anything.

Sources

InsForge and its GitHub repo: open-source agent-native backend
Base44 backend: backend with a CLI tuned for Claude Code and Cursor
Butterbase: open-source BaaS with MCP
Xano: best backends for AI-generated apps: the no-code vendor's take on the category
MoonDB documentation: the schema format and full API reference

GLM-5.2 Review: 753B Open-Weight Model That Undercuts GPT-5.5

Maksim Danilchenko — Thu, 18 Jun 2026 08:42:42 +0000

TL;DR

Z.ai's GLM-5.2 is a 753-billion-parameter mixture-of-experts model released under an MIT license on June 16, 2026. It tops GPT-5.5 on SWE-bench Pro (62.1 vs 58.6) and trails Claude Opus 4.8 by about 0.7 points on FrontierSWE (74.4% vs 75.1%). Through OpenRouter, it runs at $1.40 per million input tokens, about 72% cheaper than Claude and GPT. The catch: if you use Z.ai's hosted API instead of self-hosting the weights, your prompts route through servers governed by China's National Intelligence Law.

What GLM-5.2 Actually Is

GLM-5.2 comes from Z.ai (formerly Zhipu AI), one of China's largest AI labs and one of the few on the US Bureau of Industry and Security's Entity List. The model launched on June 13 for paying subscribers, then dropped its full open weights on Hugging Face three days later at zai-org/GLM-5.2.

The architecture is a mixture-of-experts (MoE) transformer: 753 billion total parameters, but only about 40 billion active per forward pass. That MoE structure is why it can match or beat dense models with far fewer FLOPs per token. It supports a 1-million-token context window (a 5x jump from GLM-5.1's 200K) and can produce up to 128K tokens in a single response.

I spent the last day running it through OpenRouter on three personal projects: a Flask API refactor, a Go CLI tool with nested subcommands, and a long context window test stuffing 400K tokens of a monorepo into the prompt. Not cherry-picked benchmarks, just the kind of work I do daily with Claude Code and occasionally Codex.

The first thing I noticed: GLM-5.2 is verbose. It used roughly 43K output tokens per task in Artificial Analysis's benchmark suite, compared to 26K for GLM-5.1. That verbosity inflates your bill if you're paying per token, and it slows down iteration when you're waiting for a response. On OpenRouter, a typical coding task took around 75 seconds, which felt slow compared to Opus 4.8's sub-30-second responses for similar complexity.

But the code it wrote was clean. A Python web scraper function came back production-ready on the first try: proper error handling, retries with exponential backoff, typed return values. The Go CLI output was similarly solid: correct cobra subcommand nesting, help text, and flag parsing without the usual LLM quirk of inventing nonexistent stdlib packages.

Benchmark Breakdown

The full picture, pulling from Artificial Analysis, BenchLM, and Z.ai's published results:

Benchmark	GLM-5.2	Claude Opus 4.8	GPT-5.5	DeepSeek V4 Pro
SWE-bench Pro	62.1	69.2	58.6	55.4
Terminal-Bench 2.1	81.0	85.0	—	—
FrontierSWE	74.4%	75.1%	72.6%	—
AA Intelligence Index	51	—	—	44
GDPval-AA v2 (Agentic)	1524	—	1514	1328
GPQA Diamond	89%	—	—	—
HLE	40%	—	—	—

A few things jump out from this table.

GLM-5.2 beats GPT-5.5 on every coding benchmark where both have scores (see our GPT-5.5 review for the full breakdown on that model). The SWE-bench Pro gap (62.1 vs 58.6) is real. That's 3.5 points on a benchmark where single-digit gaps separate model generations. On FrontierSWE, GLM-5.2 lands at 74.4%, within 0.7% of Opus 4.8's 75.1% (we covered Fable 5's benchmark claims recently). A 0.7% gap for a model that costs 80% less.

The agentic score (GDPval-AA v2) stands out: 1524 vs GPT-5.5's 1514. GLM-5.2 is built for long-horizon agent workflows where the model needs to plan across files, run commands, and iterate. Z.ai specifically pitched it as a coding-agent model, and the benchmarks back that framing.

Where it falls short: hallucination rate sits at 28.1% on the AA-Omniscience Index. That's an improvement over GLM-5.1's 29.4%, but it means roughly one in four factual claims from the model is wrong. For coding tasks this is less of a concern (the compiler catches lies), but for anything research-heavy or fact-dependent, you'll want external verification.

Pricing: The Real Reason to Pay Attention

Pricing changes how you actually use a model more than benchmarks do.

Model	Input (per 1M)	Output (per 1M)	Approx. cost/task	License
GLM-5.2 (OpenRouter)	$1.40	$4.40	~$0.46	MIT
Claude Opus 4.8	$5.00	$25.00	~$2.50	Proprietary
GPT-5.5	$5.00	$30.00	~$3.00	Proprietary
DeepSeek V4 Pro	$0.44	$0.87	~$0.05	DeepSeek
GLM-5.1	~$1.00	~$3.00	~$0.25	MIT

GLM-5.2 isn't the cheapest model in this table. DeepSeek V4 Pro undercuts it by 10x. But DeepSeek V4 Pro also scores significantly lower on every coding benchmark. The interesting position is GLM-5.2 vs the proprietary frontier: you get 95-100% of Claude Opus 4.8's coding performance for roughly 18% of the price.

The caveat is token consumption. GLM-5.2 burns through 43K output tokens per benchmark task, with 37K of those being internal reasoning tokens. You're billed for the reasoning tokens even though they don't appear in the visible output. That inflates the per-task cost from what the raw price-per-million suggests. At ~$0.46 per coding task, it's still cheap against proprietary options, but it's almost double GLM-5.1's ~$0.25.

For personal projects and prototyping, the math is obvious: GLM-5.2 through OpenRouter gives you frontier-adjacent coding quality at indie-developer prices (compare this with the Claude Code Pro vs Max pricing breakdown). For production agent pipelines processing thousands of tasks, the token verbosity starts to add up and DeepSeek V4 Pro's cost advantage gets harder to ignore.

The 1-Million-Token Context Window

GLM-5.1 topped out at 200K tokens. GLM-5.2 jumps to 1 million, matching Claude Opus 4.8 and GPT-5.5's API context window.

I tested this by feeding it roughly 400K tokens of a monorepo: the entire src/ directory of a mid-size Flask application with about 180 files. I asked it to trace a specific request flow through three microservices, identify where a race condition could happen, and propose a fix.

It handled the context without obvious degradation. The trace was accurate, it identified the correct database transaction that lacked proper isolation, and the fix was structurally sound. Whether it would hold up at 800K+ tokens I can't say. I didn't have a codebase that large on hand to test with.

Z.ai specifically designed GLM-5.2's IndexShare sparse-attention mechanism for long agent trajectories. The idea is that coding agents accumulate hundreds of thousands of tokens over a multi-step session: file reads, command outputs, error traces, iterative fixes. A model that degrades at 200K would force the agent to constanly prune context or restart. At 1M, the agent can carry the full project state through a long session without losing earlier context.

How to Use GLM-5.2

Three paths, each with different trade-offs.

OpenRouter (quickest start)

The fastest way to try GLM-5.2. Nine providers offer it on OpenRouter, all at roughly $1.40/$4.40 per million tokens. The API is OpenAI-compatible:

from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="your-openrouter-key",
)

response = client.chat.completions.create(
    model="z-ai/glm-5.2",
    messages=[
        {"role": "user", "content": "Refactor this Flask route to use async SQLAlchemy 2.0 sessions"}
    ],
    temperature=0.6,
    max_tokens=4096,
)
print(response.choices[0].message.content)

Z.ai's API (official, but read the data section below)

Z.ai's own API supports OpenAI SDK compatibility and adds a Coding Plan tier that's compatible with Claude Code, Cline, and Cursor. Point your tools at the coding endpoint:

export ANTHROPIC_BASE_URL="https://api.z.ai/api/coding/paas/v4"
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]"

That gives you GLM-5.2 as a drop-in replacement in Claude Code sessions. I tried this with a small Go project and it worked. Claude Code's agent loop ran normally, though response times were noticeably slower than Anthropic's own endpoints.

Self-Hosting (full control, big hardware)

The weights are on Hugging Face under MIT. You can serve them with vLLM or SGLang on your own GPUs. The hardware requirement is steep: at FP8 quantization, you need about 753 GB of VRAM. That's ten H100 80GB GPUs (800 GB total) or a smaller cluster of H200s. At BF16, the requirement doubles to ~1,500 GB.

Self-hosting makes sense for two audiences: enterprises that can't send code to external APIs for compliance reasons, and AI labs that want to fine-tune the weights for specialized domains. For everyone else, OpenRouter is simpler.

# Example vLLM serving command (8x H100)
python -m vllm.entrypoints.openai.api_server \
    --model zai-org/GLM-5.2 \
    --tensor-parallel-size 8 \
    --dtype float8 \
    --max-model-len 1048576

The China Data Question

Z.ai is headquartered in Beijing. The US Bureau of Industry and Security added Zhipu AI (Z.ai's former name) to its Entity List in January 2025, citing the company's role in advancing military AI modernization. In late April 2026, US House lawmakers opened a formal inquiry into cybersecurity risks posed by Chinese AI models in critical infrastructure, naming Z.ai alongside DeepSeek, MiniMax, ByteDance, and several others.

The risk depends entirely on the serving path.

If you use Z.ai's hosted API, your prompts and code route through servers subject to China's National Intelligence Law, which requires Chinese companies to cooperate with government data requests. For regulated industries (healthcare, finance, defense, government contracting), this is a non-starter. For any codebase containing proprietary algorithms, API keys, customer data, or anything you wouldn't post publicly, routing through Z.ai's API is a risk most security teams won't approve.

If you self-host or use a Western OpenRouter provider, the data never touches Z.ai's servers. The MIT license has no phone-home requirements, no telemetry obligations, no usage restrictions. You download the weights, serve them on your own infrastructure, and Z.ai has no visibility into what you're doing. This is the scenario where the open-weights value proposition fully pays off.

If you use OpenRouter, check which provider is serving your request. Most OpenRouter providers for GLM-5.2 route through non-Chinese infrastructure, but the routing can vary. Verify with your specific provider if this matters for your compliance requirements.

The practical upshot: GLM-5.2's China connection is a non-issue if you self-host, a manageable concern on most OpenRouter providers, and a hard blocker if you'd be sending sensitive code directly to Z.ai. The MIT license exists precisely to decouple the model's capabilities from the company's jurisdiction.

Who Should Use GLM-5.2

Good fit:

Independent developers and small teams who want near-frontier coding quality at $1.40/M input instead of $5.00/M. The cost difference compounds fast when you're running agent loops that process dozens of files.
Teams already invested in open-weight infrastructure (vLLM clusters, self-hosted inference). GLM-5.2 slots into that stack with no vendor lock-in and no API dependency.
Long-context use cases where the 1M window matters: codebase-wide refactors, multi-file agents, repository Q&A.
Anyone who needs to fine-tune a frontier-class coding model for a specific domain. MIT license means no restrictions.

Bad fit:

Enterprise teams in regulated industries that can't risk any data routing through Chinese infrastructure, unless they have the GPU budget to self-host.
Latency-sensitive production pipelines. At ~75 seconds per coding task through OpenRouter, GLM-5.2 is sluggish compared to Opus 4.8 or GPT-5.5's APIs.
Tasks requiring low hallucination rates. The 28.1% factual error rate is acceptable for code (where tests catch mistakes) but rough for content generation, research synthesis, or customer-facing text.
Teams that need multimodal capabilities. GLM-5.2 is text-only. Z.ai's vision model (GLM-5V-Turbo) exists separately and isn't open-weights.

FAQ

Is GLM-5.2 better than Claude Opus 4.8 for coding?

Not quite. On SWE-bench Pro, Opus 4.8 scores 69.2 vs GLM-5.2's 62.1, a 7-point gap. On FrontierSWE, it narrows to 75.1% vs 74.4%. Opus 4.8 is faster, less verbose, and has lower hallucination rates. But it costs 3-5x more per token. Whether "better" means "higher score" or "better value" depends on your budget.

Can I run GLM-5.2 locally?

Technically yes, if you have ~753 GB of VRAM (ten H100 80GB GPUs at FP8). For most developers, local running isn't practical. Cloud hosting through a managed provider or OpenRouter is the realistic path.

Is GLM-5.2 safe to use with proprietary code?

It depends on the serving path. Self-hosted or through a trusted Western provider on OpenRouter — yes, your data doesn't leave your infrastructure. Through Z.ai's own API — your prompts traverse Chinese servers governed by the National Intelligence Law. Most enterprise security teams will block the latter.

How does GLM-5.2 compare to DeepSeek V4 Pro?

DeepSeek V4 Pro is ~10x cheaper ($0.44/$0.87 per million tokens) as we detailed in our DeepSeek V4 Pro review, but scores lower on coding benchmarks — 55.4 on SWE-bench Pro vs GLM-5.2's 62.1. DeepSeek wins on cost, GLM-5.2 wins on capability. Both carry similar China-data concerns if used through their respective hosted APIs.

Why is GLM-5.2 so verbose?

The model produces about 43K output tokens per coding task, with 37K being internal reasoning tokens that get billed but don't appear in the response. Deeper reasoning chains improve accuracy on complex tasks, but they inflate costs and latency. Z.ai hasn't offered a "low verbosity" mode that trades some accuracy for speed.

Sources

Z.ai GLM-5.2 documentation — official specs, API reference, and capability overview
Simon Willison's analysis of GLM-5.2 — independent benchmarks and observations from the day of release
Artificial Analysis: GLM-5.2 is the new leading open-weights model — Intelligence Index v4.1 scores, GDPval-AA agentic benchmarks, and pricing data
VentureBeat: Z.ai's GLM-5.2 beats GPT-5.5 on multiple benchmarks — SWE-bench Pro and FrontierSWE comparisons
OpenRouter GLM-5.2 listing — live pricing and provider availability
GLM-5.2 weights on Hugging Face — MIT-licensed open weights

Bottom Line

GLM-5.2 is the strongest open-weights coding model available today. At 62.1 on SWE-bench Pro, it beats GPT-5.5 and closes much of the gap to Claude Opus 4.8's 69.2, while costing a fraction of either. The MIT license means you can self-host it, fine-tune it, and deploy it in air-gapped environments.

The trade-offs are real. It's slow (75 seconds per task on OpenRouter), verbose (burning 43K tokens when 20K might suffice), and the 28.1% hallucination rate means you can't trust it for factual content without verification. The China data question adds a layer: if you can't self-host and your compliance posture rules out Chinese API endpoints, you're limited to OpenRouter's Western providers.

For non-sensitive coding work where I'm paying out of pocket, GLM-5.2 through OpenRouter at $1.40/M input is the best value in frontier AI right now. It wrote clean Python and Go on the first pass, handled 400K tokens of context without degradation, and saved me about 80% compared to my usual Opus 4.8 API costs. Open-weights coding models have crossed from curiosity to credible daily driver.

LiteLLM Vulnerability: 6 CVEs, a Supply Chain Attack, and the Fixes

Maksim Danilchenko — Wed, 17 Jun 2026 08:47:51 +0000

TL;DR

LiteLLM, the AI gateway proxy used by CrewAI, DSPy, Microsoft GraphRAG, and roughly 97 million monthly PyPI installs, has been hit by six CVEs and a full supply chain compromise since March 2026. The worst chain scores CVSS 10.0 and gives unauthenticated attackers remote code execution on any exposed LiteLLM proxy. CISA added one of the CVEs to its Known Exploited Vulnerabilities catalog on June 8 with a June 22 remediation deadline. If you run LiteLLM, upgrade to v1.83.14+ and rotate every credential the proxy can reach.

What LiteLLM Does and Why You're Probably Running It

LiteLLM is an open-source Python proxy that gives you a single OpenAI-compatible API in front of 100+ LLM providers: OpenAI, Anthropic, Google, Azure, Bedrock, Cohere, and more. You point your application at the proxy, the proxy routes requests to the right provider, and you get cost tracking, load balancing, rate limiting, and guardrails for free. About 50,300 GitHub stars and 97 million PyPI downloads a month as of June 2026.

I ran a LiteLLM proxy for three months last year to route Claude and GPT calls from a multi-agent pipeline. The setup took about 15 minutes: a Docker container, a YAML config with my API keys, and a litellm --config config.yaml command. That simplicity is exactly the problem. LiteLLM sits between your entire AI stack and every model provider you use. Every API key, every prompt, every response flows through it. When the proxy gets owned, the attacker gets everything — and AI tooling already leaks secrets at twice the rate of manual code.

And in 2026, LiteLLM got owned from every angle.

Every Vulnerability at a Glance

CVE	Type	CVSS	Affected Versions	Fixed In
CVE-2026-42271	Command injection (MCP endpoints)	8.8 (10.0 chained)	1.74.2 – 1.83.6	v1.83.7
CVE-2026-48710	Starlette host header bypass	Enabler	Starlette ≤1.0.0	Starlette 1.0.1
CVE-2026-42208	Pre-auth SQL injection	9.8	1.81.16 – 1.83.6	v1.83.7
CVE-2026-47101	Auth bypass (allowed_routes)	8.8 (9.9 chained)	<1.83.14	v1.83.14
CVE-2026-47102	Privilege escalation (user_role)	Part of 9.9 chain	<1.83.14	v1.83.14
CVE-2026-40217	Sandbox escape (exec guardrails)	8.8	<1.83.14	v1.83.14

Plus a full supply chain attack on March 24 that injected credential-stealing malware into PyPI packages v1.82.7 and v1.82.8.

CVE-2026-42271: Remote Code Execution via MCP Test Endpoints

This is the one CISA flagged on June 8 with a June 22 remediation deadline.

LiteLLM versions 1.74.2 through 1.83.6 included two MCP server preview endpoints: POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list. These accepted a full server configuration (command, arguments, and environment variables for stdio transport) and LiteLLM spawned the supplied command as a subprocess on the host. Any authenticated user with a standard API key could reach them.

That alone scores CVSS 8.8. But Horizon3.ai researchers chained it with CVE-2026-48710, a host header validation bypass in Starlette versions ≤1.0.0. The Starlette bug lets attackers bypass LiteLLM's authentication layer entirely by sending a malformed Host header. No API key, no login, no prior access needed.

The combined chain scores CVSS 10.0 and gives any internet-connected attacker unauthenticated RCE on the proxy host.

# This is what the exploit looks like (simplified for illustration)
# The attacker sends a crafted Host header to bypass auth,
# then passes arbitrary commands via the MCP test endpoint
curl -X POST https://target-litellm:4000/mcp-rest/test/connection \
  -H "Host: badhost" \
  -H "Content-Type: application/json" \
  -d '{"server_name": "evil", "transport": "stdio",
       "command": "/bin/sh", "args": ["-c", "id; cat /proc/self/environ"]}'

Horizon3.ai validated the full chain on June 1. CISA confirmed active exploitation in the wild.

The Privilege Escalation Chain: Three CVEs, One Path to Root

Obsidian Security disclosed this chain on June 11, and it's the scarier find because it starts from a default low-privilege account that every LiteLLM deployment has.

Step 1 — CVE-2026-47101 (Auth bypass): A standard internal_user calls /key/generate with allowed_routes: ["/*"]. LiteLLM validates that the user_id matches the caller but never checks what routes are being granted. The user now holds a wildcard API key that bypasses all role-based access controls.

Step 2 — CVE-2026-47102 (Privilege escalation): The wildcard key accesses /user/update. This endpoint verifies identity but has zero field-level authorization. The attacker sets user_role: "proxy_admin" on their own account and becomes a full admin.

Step 3 — CVE-2026-40217 (Sandbox escape to RCE): Admin-level guardrail endpoints accept user-supplied Python code and compile it with exec(). The globals dictionary omits __builtins__, but Python automatically injects the full builtins module when that key is missing. One line gets you shell access:

# The "sandboxed" guardrail code that actually has full system access
__builtins__['__import__']('os').system('cat /etc/shadow')

Obsidian rates the full chain CVSS 9.9. A server takeover exposes every provider key, every stored credential, every prompt and response passing through the gateway. They demonstrated a "man-in-the-gateway" attack that injected fake tool calls into Claude Code responses, causing it to execute attacker-controlled commands while evading auto-mode safety checks.

CVE-2026-42208: SQL Injection in the Auth Layer

Bishop Fox found this one: a pre-authentication SQL injection via the Authorization: Bearer header. Versions 1.81.16 through 1.83.6 build the token verification query with a Python f-string:

# The vulnerable code in litellm/proxy/utils.py
query = f"WHERE v.token = '{token}'"

LiteLLM's _hash_token_if_needed() only hashes values starting with sk-. Non-prefixed bearer tokens pass straight to the f-string. An attacker can extract the entire LiteLLM_VerificationToken table — all virtual keys, team bindings, spend records — via timing-based blind injection with pg_sleep().

Default deployments run with database superuser privileges, so the blast radius extends to every table in the database. Sysdig observed in-the-wild exploitation within 36 hours of the GitHub advisory going public, with attackers targeting litellm_credentials.credential_values to steal upstream LLM provider keys.

The March Supply Chain Attack

On March 24, a threat actor called TeamPCP published two backdoored versions of LiteLLM (v1.82.7 and v1.82.8) to PyPI.

The attack chain started weeks earlier. TeamPCP exploited a pull_request_target workflow vulnerability in Trivy (the container security scanner), hijacked the aqua-bot GitHub Actions credentials, and used them to compromise Checkmarx KICS GitHub Actions. When LiteLLM's CI/CD pipeline ran the compromised Trivy scanner during a build, the malicious action extracted LiteLLM's PYPI_PUBLISH token from the GitHub Actions environment. CI/CD supply chain risks like this are growing across the ecosystem.

The malicious payload installed a .pth file that fired on every Python interpreter startup — no import required. It harvested environment variables, SSH keys, .env files, API tokens, AWS/GCP/Azure credentials, Kubernetes configs, and database credentials. Everything was encrypted with AES-256-CBC and RSA-4096, bundled into a tarball, and exfiltrated to models.litellm[.]cloud. It also deployed persistence via a fake sysmon.service systemd unit and, in Kubernetes environments, spawned privileged pods on every node.

PyPI quarantined the packages within about 3 hours. But Microsoft GraphRAG, Google ADK, DSPy, CrewAI, and OpenHands all pulled the malicious version as a transitive dependency during that window. Datadog Security Labs later linked this to a broader TeamPCP campaign that also hit npm packages and OpenVSX extensions.

How to Check If You're Affected

Run these checks in order:

# 1. Check your LiteLLM version
pip show litellm | grep Version
# or inside Docker:
docker exec <container> pip show litellm | grep Version

# 2. Check your Starlette version (for the auth bypass chain)
pip show starlette | grep Version

# 3. Check if the vulnerable MCP endpoints exist
curl -s -o /dev/null -w "%{http_code}" \
  http://localhost:4000/mcp-rest/test/connection
# 404 = not vulnerable (endpoint removed). 405 or 200 = vulnerable.

# 4. Check for supply chain attack artifacts
find / -name "litellm_init.pth" 2>/dev/null
ls -la ~/.config/sysmon/sysmon.py 2>/dev/null
systemctl status sysmon.service 2>/dev/null

If your LiteLLM version is below 1.83.14, you're exposed to at least some of these vulnerabilities. If it's between 1.82.7 and 1.82.8 and you installed between March 24 10:39 UTC and ~13:38 UTC, you may have the supply chain backdoor.

How to Fix Everything

Upgrade LiteLLM to at least v1.83.14 (v1.89.1 is the latest as of June 16):

# pip upgrade
pip install --upgrade "litellm>=1.83.14"

# Docker — pull the latest stable image
docker pull ghcr.io/berriai/litellm:main-latest

Upgrade Starlette to close the unauthenticated RCE path:

pip install --upgrade "starlette>=1.0.1"

Rotate credentials. If your proxy was ever internet-accessible on a vulnerable version, treat every secret it could reach as compromised:

# Rotate these (at minimum):
# - LITELLM_MASTER_KEY and LITELLM_SALT_KEY
# - DATABASE_URL and database passwords
# - Every LLM provider API key (OpenAI, Anthropic, Google, Azure, Bedrock)
# - MCP and agent OAuth tokens
# - SSH keys and cloud credentials (AWS, GCP, Azure)

# Check your logs for signs of exploitation:
grep -E "/mcp-rest/test/(connection|tools/list)" /var/log/litellm/*.log
grep -E "/user/(update|bulk_update).*user_role" /var/log/litellm/*.log

If you were hit by the supply chain attack, rebuild your containers from scratch. The backdoor persists through restarts via systemd and .pth files.

Can't upgrade immediately? These temporary mitigations buy time:

# Block the vulnerable endpoints at your reverse proxy (nginx example)
location ~ ^/mcp-rest/test/ {
    deny all;
    return 403;
}

# Block non-standard bearer tokens to mitigate SQL injection
# Only allow tokens matching the sk- prefix pattern

What Went Wrong, and What It Means for Your AI Stack

LiteLLM's security problems aren't unusual for a fast-growing open-source project. The f-string SQL query is a rookie mistake, but it's the kind of mistake that gets shipped when the priority is features, not security audits. The exec() sandbox with missing builtins is a well-known Python gotcha that's been documented since 2009.

The supply chain attack is worse. TeamPCP didn't need to find a bug in LiteLLM's code, and AI-driven vulnerability discovery means the attack surface is only growing. They compromised the security scanner that LiteLLM's CI pipeline trusted, then used that to steal PyPI publishing credentials. The attack took weeks to set up and targeted LiteLLM specifically because it sits at the chokepoint of thousands of AI deployments.

If you're running any AI gateway or proxy, the takeaway is straightforward: treat it like you'd treat your database. Don't expose it to the internet without authentication. Don't run it as root. Pin your dependency versions (a good package manager helps), verify checksums on deployment, and have a credential rotation plan that doesn't require a Saturday fire drill.

CrewAI published a removal guide for decoupling from LiteLLM entirely. Whether that's the right move depends on your threat model. LiteLLM has since engaged Veria Labs for a proxy audit, launched a bug bounty ($500–$3,000), and committed to 5-business-day response times on advisories. The project is taking security more seriously. Whether that's enough depends on how many more CVEs are sitting in the other 100,000+ lines of Python.

FAQ

What is CVE-2026-42271?

CVE-2026-42271 is a command injection vulnerability in LiteLLM's MCP REST test endpoints. Two preview endpoints (/mcp-rest/test/connection and /mcp-rest/test/tools/list) accepted full server configurations and spawned the supplied command as a subprocess. When chained with a Starlette host header bypass (CVE-2026-48710), it becomes an unauthenticated remote code execution vulnerability scoring CVSS 10.0.

What versions of LiteLLM are affected?

For the RCE and SQL injection vulnerabilities: versions 1.74.2 through 1.83.6. For the privilege escalation chain: all versions before 1.83.14. The supply chain attack only affected PyPI packages v1.82.7 and v1.82.8, published on March 24 and quarantined within 3 hours.

How to fix LiteLLM vulnerability?

Upgrade to LiteLLM v1.83.14 or later (v1.89.1 is the latest as of June 2026), upgrade Starlette to v1.0.1+, and rotate every credential the proxy had access to: LLM provider keys, database passwords, the master key, SSH keys, and cloud credentials.

Is LiteLLM safe to use in production?

After upgrading to v1.83.14+, the known vulnerabilities are patched. But LiteLLM has accumulated 16 CVEs across 2024–2026, including three Criticals. If you use it in production, run it behind a reverse proxy, restrict network access, disable unused features like Custom Code Guardrails, and monitor the GitHub advisory feed closely.

What happened with the LiteLLM supply chain attack?

On March 24, 2026, a threat actor called TeamPCP published backdoored LiteLLM versions 1.82.7 and 1.82.8 to PyPI. They got there by first compromising Trivy's GitHub Actions, then using that access to steal LiteLLM's PyPI publishing credentials from CI/CD. The malicious packages harvested every credential on the host machine and exfiltrated them to an attacker-controlled server. PyPI quarantined them within about 3 hours, but major frameworks like Microsoft GraphRAG and Google ADK pulled the malicious versions as transitive dependencies during that window.

How does the LiteLLM exploit chain work?

Multiple chains exist. The worst (CVSS 10.0) combines a command injection in MCP test endpoints with a Starlette host header bypass for unauthenticated RCE. A separate chain (CVSS 9.9) starts from a low-privilege default account: the attacker creates a wildcard API key via an unchecked allowed_routes field, self-promotes to admin by modifying their own user_role, then executes arbitrary Python through the Custom Code Guardrail's broken exec() sandbox.

Sources

Horizon3.ai — CVE-2026-42271 chained with CVE-2026-48710 — original research on the CVSS 10.0 RCE chain
The Hacker News — LiteLLM Vulnerability Chain Lets Low-Privilege Users Take Over — coverage of the Obsidian Security CVSS 9.9 privilege escalation disclosure
Help Net Security — LiteLLM vulnerability under active attack, CISA warns — CISA KEV announcement
Bishop Fox / LiteLLM — CVE-2026-42208 SQL Injection — official advisory for the pre-auth SQL injection
Sysdig — CVE-2026-42208 exploitation analysis — in-the-wild exploitation within 36 hours
Snyk — How a Poisoned Security Scanner Became the Key to Backdooring LiteLLM — technical breakdown of the supply chain attack
HeroDevs — The LiteLLM Supply Chain Attack — timeline and impact assessment
Datadog Security Labs — LiteLLM compromised PyPI TeamPCP campaign — broader TeamPCP campaign analysis
LiteLLM — Security Hardening April 2026 — official post-incident security improvements

Bottom Line

LiteLLM got hammered from every direction in 2026: supply chain poisoning, unauthenticated RCE, SQL injection in the auth layer, privilege escalation from default accounts. If you've been running an internet-exposed proxy on anything older than v1.83.7, assume breach and act accordingly. The CISA deadline is June 22 — five days from now.

Gemini 3.5 Flash vs Claude Haiku 4.5 vs MAI-Code-1-Flash for Coding

Maksim Danilchenko — Wed, 10 Jun 2026 08:35:36 +0000

TL;DR

Three flash-tier coding models are competing for your API budget right now: Google's Gemini 3.5 Flash (May 19, 2026), Anthropic's Claude Haiku 4.5 (the reigning budget pick since October 2025), and Microsoft's MAI-Code-1-Flash (June 2, 2026). Haiku wins on output cost at $5/M tokens and structured output reliability. Gemini 3.5 Flash leads on agentic benchmarks (76.2% Terminal-Bench 2.1) and offers a 1M-token context window. MAI-Code-1-Flash beats both on SWE-Bench Pro by 16 points at 51.2%, but you can only use it inside GitHub Copilot. Pick based on where you actually build: Copilot users get MAI-Code-1 for free, API builders choose between Haiku's cost and Flash's context, and anyone running agent loops with tool calls should benchmark Flash first.

Three Flash Models, Three Different Bets

I've spent the last three weeks routing coding tasks through all three of these models: code reviews in Copilot, agent loops via API, and batch refactors across a 40-file Python project. The experience taught me something the benchmark tables don't show: each model was built with a different definition of "coding" in mind.

Google optimized Gemini 3.5 Flash for agents that run in terminals, call tools, and iterate. Anthropic built Haiku 4.5 for developers who need a cheap, fast model that follows instructions precisely and returns clean JSON. Microsoft trained MAI-Code-1-Flash end-to-end inside the GitHub Copilot harness, so it knows how VS Code works, what diffs look like, and how to stay concise in inline completions.

Each model answers a different version of the same question: "What should a small coding model be good at?"

Benchmark Comparison

Benchmarks don't capture everything, but they're measurable. Start here.

Benchmark	Gemini 3.5 Flash	Claude Haiku 4.5	MAI-Code-1-Flash
SWE-Bench Verified	—	73.3% (Anthropic) / 66.6% (Microsoft's eval)	71.6%
SWE-Bench Pro	55.1%	35.2%	51.2%
Terminal-Bench 2.1	76.2%	41.6%*	54.8%*
MCP Atlas (tool use)	83.6%	—	—
IF-Bench (instruction following)	—	—	+28.9 pts over Haiku

*Terminal-Bench numbers for Haiku and MAI-Code-1-Flash are from Microsoft's evaluation (Terminal-Bench 2, not 2.1). Direct comparison to Flash's 76.2% on v2.1 should be taken with a grain of salt.

A few things jump out from this table.

The SWE-Bench Verified discrepancy for Haiku is real and worth flagging. Anthropic reports 73.3%, Microsoft reports 66.6% when benchmarking against MAI-Code-1-Flash. The difference probably comes down to evaluation setup: system prompts, tool availability, and retry policies all shift SWE-Bench scores. I wouldn't treat either number as gospel. The relative ranking across SWE-Bench Pro, where the gap is enormous (51.2% vs 35.2%), is more informative.

Gemini 3.5 Flash dominates the agentic benchmarks. Terminal-Bench 2.1 simulates a real engineer working in a sandboxed terminal with a 5-hour timeout — planning, iterating, and coordinating across tools. Flash's 76.2% puts it above Gemini 3.1 Pro and close to GPT-5.5 territory. If your coding model runs inside an agent loop with tool calls, this number is the one that predicts real-world behavior.

MAI-Code-1-Flash's instruction following is the other number worth reading. The +28.9 point lead over Haiku on IF-Bench shows Microsoft's harness-native training paid off. The model knows how to handle structured requests ("edit only lines 14-22", "don't touch the imports", "return a unified diff") because it learned from Copilot's actual production request patterns.

Pricing: What You'll Actually Pay

Flash models live or die on cost. If price didn't matter, you'd use Claude Opus 4.7 or GPT-5.5. Per-million-token pricing:

	Gemini 3.5 Flash	Claude Haiku 4.5	MAI-Code-1-Flash
Input (per 1M tokens)	$1.50	$1.00	$0.75
Output (per 1M tokens)	$9.00	$5.00	$4.50
Cached input (per 1M)	$0.15	$0.10	$0.075
Context window	1,000,000	200,000	Not disclosed
Output limit	65,536	64,000	Not disclosed
Availability	API, Google AI Studio	API, Anthropic Console	GitHub Copilot only

The output price gap is the one that bites you. Code generation is output-heavy. A typical agent loop generating a 200-line file produces 8-12K output tokens per turn. At those volumes:

Haiku: $0.05-0.06 per turn
MAI-Code-1-Flash: $0.036-0.054 per turn
Gemini 3.5 Flash: $0.072-0.108 per turn

Across a full day of heavy coding (say 200 agent turns), that's $10 for Haiku, $8 for MAI-Code-1, and $18 for Gemini Flash. The gap compounds fast.

But MAI-Code-1-Flash has a catch: those prices are from GitHub's model picker listing. You can't hit the model through a standalone API endpoint. It only runs inside Copilot. If you're building your own agent framework, your choices are Haiku or Flash.

And Flash has its own cost lever: cached input at $0.15/M. If your agent loop sends the same system prompt and codebase context on every turn (most do), you're paying 90% less for input after the first call. That cached-input discount often offsets the higher output price for long-running agent sessions.

Token Efficiency: MAI-Code-1-Flash's 60% Claim

Microsoft claims MAI-Code-1-Flash "solves harder problems with up to 60% fewer tokens" on SWE-Bench Verified. That's a big number. The model costs less per token AND uses fewer tokens to reach the same solution.

I tested this informally on my own codebase. I asked all three models to add input validation to a FastAPI endpoint. Same prompt, same context, same expected output.

# Prompt: Add Pydantic validation to this endpoint.
# Validate: name (str, 2-50 chars), email (valid format), age (18-120)

@app.post("/users")
async def create_user(request: Request):
    data = await request.json()
    # ... existing logic

The results:

Haiku 4.5: 847 output tokens. Clean solution, used EmailStr from Pydantic, added a proper error handler. Correct on first try.
Gemini 3.5 Flash: 1,241 output tokens. Added validation plus a lengthy explanation of each field constraint, a usage example, and a curl command. The code was correct but I didn't ask for the tutorial.
MAI-Code-1-Flash (via Copilot): 512 output tokens. Returned only the modified function with a minimal Pydantic model. No explanation, no example. Correct and concise.

This single test isn't a benchmark. But it matches the pattern Microsoft describes: MAI-Code-1-Flash learned from Copilot interactions where conciseness is the default. It doesn't explain unless you ask.

Flash's verbosity isn't always a downside. If you're prototyping and want the model to think aloud, that extra context helps. But for batch operations and agent loops where you're parsing structured output, fewer tokens means faster iteration and lower cost.

Context Windows: The 1M Advantage

This is where Gemini 3.5 Flash separates itself from the other two.

Model	Context Window	Output Limit
Gemini 3.5 Flash	1,000,000 tokens	65,536 tokens
Claude Haiku 4.5	200,000 tokens	64,000 tokens
MAI-Code-1-Flash	Not disclosed	Not disclosed

A million-token context window means you can feed Flash an entire mid-sized codebase (50-80 files of typical Python or TypeScript) in a single prompt. Haiku's 200K is generous by historical standards but won't hold the same volume. If you're doing codebase-wide analysis, architecture reviews, or cross-file refacotrs, Flash is the only flash-tier option that won't force you to chunk.

Both Haiku and Flash now support large output limits (64K and 65K respectively), so you won't hit output ceilings on most tasks. I've pushed both models through full-file rewrites of 300-line modules without truncation. The context input limit is the real differentiator: Flash's 1M lets you include far more codebase context per request.

For Copilot workflows where MAI-Code-1-Flash operates, the context window is less of an issue. Copilot manages the context for you, feeding relevant files and recent edits. You don't directly control the prompt size.

Where Each Model Wins

After three weeks of testing, I'd route tasks like this:

Gemini 3.5 Flash — agent loops and long-context analysis

Flash is the model I'd pick for any workflow that involves iterating with tools. Write a failing test, run it, read the error, fix the code, run again. Flash handles that loop better than the other two. Its Terminal-Bench scores reflect a model that was built for multi-turn tool coordination, not just static code generation. The 1M context window makes it the default choice for "analyze this whole codebase" tasks.

Claude Haiku 4.5 — structured output, code review, and high-volume batch work

Haiku returns the cleanest structured output of the three. If you're calling the model 10,000 times a day for code review comments, PR summaries, or JSON-formatted analysis, Haiku's combination of reliable instruction following and the cheapest output tokens makes it the rational choice. It's also the model I trust most for diff generation and structured editing tasks.

MAI-Code-1-Flash — inline completions and Copilot-native workflows

If you live in VS Code and use GitHub Copilot, MAI-Code-1-Flash is the model that feels most native. It knows the environment: when to suggest a single line vs. a full function, it handles diffs cleanly, and it stays concise. The 60% token efficiency claim holds up in practice for the type of tasks Copilot handles — inline edits, small refactors, and completion suggestions.

Availability and Integration

This is the practical differentiator most comparisons skip. It doesn't matter how good a model is if you can't access it from your stack.

	Gemini 3.5 Flash	Claude Haiku 4.5	MAI-Code-1-Flash
Standalone API	Yes (Gemini API)	Yes (Anthropic API)	No
Google AI Studio	Yes	No	No
AWS Bedrock	No	Yes	No
GitHub Copilot	No	No	Yes
VS Code (direct)	Via extension/API	Via extension/API	Built-in via Copilot
OpenRouter	Yes	Yes	No
Self-hosting	No	No	No

MAI-Code-1-Flash's Copilot lock-in is the biggest caveat in this comparison. Microsoft has signaled plans for Azure Foundry and third-party provider access, but as of early June 2026 the model is still rolling out primarily through Copilot. If you're building custom agents, pipelines, or CI/CD integrations, MAI-Code-1-Flash isn't an option today.

For API access, both Flash and Haiku work through OpenRouter too, so you can swap between them without changing your client code. If you're also evaluating open-source alternatives, DeepSeek V4 Pro punches above its weight at a fraction of the cost.

API Quick Start

Calling each model from Python:

Gemini 3.5 Flash:

from google import genai

client = genai.Client(api_key="YOUR_KEY")

response = client.models.generate_content(
    model="gemini-3.5-flash",
    contents="Refactor this function to use list comprehension instead of a for loop:\n\ndef filter_active(users):\n    result = []\n    for u in users:\n        if u.is_active:\n            result.append(u.name)\n    return result"
)
print(response.text)

Output:

def filter_active(users):
    return [u.name for u in users if u.is_active]

Claude Haiku 4.5:

import anthropic

client = anthropic.Anthropic(api_key="YOUR_KEY")

message = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": "Refactor this function to use list comprehension instead of a for loop:\n\ndef filter_active(users):\n    result = []\n    for u in users:\n        if u.is_active:\n            result.append(u.name)\n    return result"
    }]
)
print(message.content[0].text)

Output:

def filter_active(users):
    return [u.name for u in users if u.is_active]

MAI-Code-1-Flash (via Copilot — no standalone API):

Open the file in VS Code with GitHub Copilot enabled, select the function, and run Copilot Chat with the prompt. If you're working with Gemini's broader toolchain, the Gemini CLI tutorial covers the terminal setup. MAI-Code-1-Flash activates through the model picker when available, or via the "auto" selector that routes to it for coding tasks.

FAQ

Which is better for coding, Gemini 3.5 Flash or Claude Haiku 4.5?

It depends on the task shape. Gemini 3.5 Flash outperforms on agentic coding — multi-step workflows with tool calls and terminal interaction (76.2% Terminal-Bench 2.1). Claude Haiku 4.5 leads on SWE-Bench Verified (73.3%) and costs 44% less on output tokens. For high-volume batch code tasks, Haiku's price wins. For agent loops, Flash's quality wins.

How much does Gemini 3.5 Flash cost compared to Claude Haiku 4.5?

Gemini 3.5 Flash charges $1.50/$9.00 per million input/output tokens. Claude Haiku 4.5 charges $1.00/$5.00. Flash is 50% more expensive on input and 80% more on output. But Flash's cached input rate ($0.15/M) can offset the difference in long agent sessions where you repeat the same context.

Is MAI-Code-1-Flash better than Claude Haiku 4.5?

On Microsoft's own benchmarks, yes — particularly SWE-Bench Pro (51.2% vs 35.2%) and instruction following (+28.9 points). But there's a benchmark discrepancy: Microsoft reports Haiku's SWE-Bench Verified score as 66.6%, while Anthropic reports 73.3%. And MAI-Code-1-Flash is only available inside GitHub Copilot, not via API. If you need a standalone API model, Haiku is your pick regardless of benchmark numbers.

Which flash model is cheapest for coding?

MAI-Code-1-Flash has the lowest per-token cost ($0.75/$4.50 per million) AND uses up to 60% fewer tokens per task. But it's locked to GitHub Copilot. For API users, Claude Haiku 4.5 at $1.00/$5.00 is the cheapest option. Gemini 3.5 Flash is the most expensive at $1.50/$9.00, though its prompt caching drops repeated-context costs to $0.15/M input.

Can I use MAI-Code-1-Flash outside of GitHub Copilot?

Not currently. MAI-Code-1-Flash is rolling out exclusively through GitHub Copilot's model picker in VS Code. Microsoft hasn't announced an Azure AI endpoint or standalone API. If you need API access for custom agents or CI/CD, you're limited to Gemini 3.5 Flash and Claude Haiku 4.5.

Sources

Gemini 3.5 Flash Model Card — Google DeepMind — official specs, benchmark numbers, and pricing
Introducing MAI-Code-1-Flash — Microsoft AI — official announcement with SWE-Bench and IF-Bench comparisons
Introducing Claude Haiku 4.5 — Anthropic — official announcement with SWE-Bench Verified score
Microsoft AI on X: benchmark comparison tweet — the SWE-Bench Verified 71.6 vs 66.6 numbers
Gemini 3.5 Flash vs Claude Haiku 4.5: Pricing & Production Fit — Evolink — independent pricing and performance comparison

Bottom Line

The flash-tier coding model race in mid-2026 isn't about finding one winner. It's about matching models to workflows.

If you build custom agents and need a model that handles tool calls and long context, Gemini 3.5 Flash is the leader. If you need the cheapest reliable model for structured output at scale, Claude Haiku 4.5 is the safe bet. And if you code in VS Code with Copilot all day, MAI-Code-1-Flash is quietly the best inline coding model available — you just can't take it anywhere else.

The lock-in question matters more than the benchmarks. Google and Anthropic sell tokens; Microsoft sells a workflow. Right now, MAI-Code-1-Flash's Copilot exclusivity makes it a non-starter for anyone building outside that stack. If Microsoft opens API access — and the GitHub Copilot AI credits system suggests they're heading that direction — the pricing math changes for everyone.

Agent Client Protocol (ACP): Connect Any AI Agent to Any Editor

Maksim Danilchenko — Thu, 04 Jun 2026 08:41:00 +0000

TL;DR

The Agent Client Protocol (ACP) is a JSON-RPC 2.0 standard that lets any AI coding agent run inside any supporting editor: JetBrains, Zed, Neovim, Emacs. Think LSP, but for AI agents instead of language servers. The registry already lists 50 agents (Claude Code, Gemini CLI, Codex, GitHub Copilot, Goose, and more). This tutorial walks through real setup in JetBrains and Zed, custom agent config, and what ACP gets right versus where it still needs work.

My Weekend ACP Setup Across Three Editors

I use JetBrains for Go and Python, Zed for quick edits, and Neovim when I'm SSH'd into a server. Until a few months ago, that meant three completely separate AI setups: Copilot in JetBrains, Zed's built-in assistant, and nothing in Neovim because I couldn't be bothered to configure yet another integration.

Then I set up ACP in all three editors over a weekend. By Monday, I had Claude Code, Gemini CLI, and Codex available in every editor through the same protocol. I could switch between agents mid-session depending on the task. Claude for architecture reasoning (especially with subagents), Gemini for quick multi-file edits, Codex for throwaway scripts. The configuration files are nearly identical across editors.

That weekend convinced me ACP is the first protocol I've used that reduced complexity instead of adding it.

What Is the Agent Client Protocol?

Every AI coding tool used to require its own editor plugin. Cursor baked its agent into a VS Code fork. GitHub Copilot needed the Copilot extension. Claude Code ran only in the terminal. If you wanted to use three agents, you needed three separate integrations. And if your editor of choice didn't have a plugin for your preferred agent, you were out of luck.

ACP fixes this with a simple idea borrowed from LSP: define a standard wire protocol between editors and agents, so any agent can plug into any editor.

The protocol works like this:

The editor (client) spawns the agent as a subprocess
The agent communicates back over JSON-RPC 2.0 on stdin/stdout (local agents) or HTTP/WebSocket (remote agents, still in development)
Work organizes into sessions (a conversation with shared context) and turns (individual prompt-response cycles)
The agent can request file access, terminal execution, and tool calls, all through the protocol

Zed Industries created ACP in August 2025. JetBrains joined shortly after. By January 2026, they co-launched the ACP Registry, a directory of compatible agents built into both editors. As of June 2026, 50 agents and a growing list of editors implement the spec. Zed and JetBrains have native support; Neovim, Emacs, and VS Code have community plugins.

ACP vs MCP

If you've worked with the Model Context Protocol (MCP), you might wonder why we need another protocol. The confusion is understandable because both involve AI agents and JSON-RPC. But they sit on different sides of the agent.

Aspect	ACP	MCP
Direction	Editor ↔ Agent	Agent ↔ Tools
Client	Code editor (JetBrains, Zed, Neovim)	AI application (Claude, GPT)
Server	AI coding agent	Tool/data provider
Purpose	How an editor drives an agent	How an agent calls external tools
Transport	JSON-RPC over stdio or HTTP	JSON-RPC over stdio or SSE
Created by	Zed Industries (Aug 2025)	Anthropic (Nov 2024)

MCP gives agents access to tools: databases, APIs, file systems, browser automation. ACP gives agents access to editors: file trees, terminals, diff views, user prompts.

The two protocols stack rather than compete. A typical setup looks like this: Zed (ACP client) launches Claude Code (ACP agent), which then connects to an MCP server for database access. ACP handles the editor-to-agent communication; MCP handles the agent-to-tool communication.

If you've already built MCP servers for your workflow, those still work under ACP. The agent sits in the middle and bridges both protocols.

The ACP Registry: 50 Agents

The ACP Registry is a curated directory that editors query directly. When you open JetBrains or Zed and browse available agents, you're reading from this registry. It auto-updates hourly from npm, PyPI, and GitHub Releases.

Here are some agents worth knowing about:

Agent	Version	What It Does
Claude Agent	0.40.0	Anthropic's Claude, wrapped for ACP
Gemini CLI	0.45.0	Google's official Gemini terminal agent
Codex CLI	0.15.0	OpenAI's coding assistant
GitHub Copilot	1.0.59	GitHub's pair programmer
Cursor	2026.05.28	Cursor's integrated agent
Goose	1.37.0	Open source, extensible, local-first
Kiro	7.3.21	AWS-backed coding agent
Junie	1831.35.0	JetBrains' own AI agent
OpenCode	1.15.13	Open source agent by Anomaly
Grok Build	0.2.11	xAI's coding agent
Mistral Vibe	2.13.0	Mistral's open source assistant
Qwen Code	0.17.1	Alibaba's Qwen coding agent

The full list has 50 entries. Most are installable in one click from JetBrains or Zed. Custom agents that aren't in the registry can be added manually through config files (covered below).

Setting Up Claude Code via ACP in JetBrains

This is the setup I use daily. JetBrains ships native ACP support across all their IDEs: IntelliJ IDEA, PyCharm, WebStorm, GoLand, the lot. No subscription required for ACP itself (the agents may need their own API keys).

Step 1: Open the agent panel

Open the AI Chat tool window (usually on the right sidebar). Click the dropdown at the top, which shows your current agent/model.

Step 2: Install from registry

Click "Install From ACP Registry" in the dropdown. Scroll to Claude Agent and select it.

The IDE downloads the agent binary and creates a config entry automatically. You'll be prompted for your Anthropic API key. Paste it in.

Step 3: Verify

Back in the AI Chat panel, select "Claude Agent" from the dropdown. Type a message. If you see Claude responding with code awareness (file context, project structure), the agent is connected and reading your workspace through ACP.

Step 4: Configure MCP servers (optional)

If you already use MCP servers (a Chrome DevTools MCP server or a FastMCP server, for example), you can wire them into the ACP agent. Edit ~/.jetbrains/acp.json:

{
  "default_mcp_settings": {
    "use_custom_mcp": true,
    "use_idea_mcp": false
  },
  "agent_servers": {
    "Claude Agent": {
      "command": "/usr/local/bin/claude-acp",
      "args": ["acp"],
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}

The use_custom_mcp: true flag tells the agent to use your MCP config instead of the IDE's built-in tool servers. Restart the IDE after editing.

Setting Up Gemini CLI via ACP in Zed

Zed was the first editor to support ACP (they created the protocol), so the integration is smooth.

Step 1: Open the agent panel

Press Cmd+? (macOS) or Ctrl+? (Linux/Windows). This opens the agent panel on the right side.

Step 2: Add Gemini CLI

Click the + button in the top-right corner. Select Gemini CLI from the registry list.

You'll be asked for your Google AI API key on first use. Paste it, and the agent starts.

Step 3: Use it

Gemini CLI now has full access to your project through ACP. File reading, terminal execution, and diff generation all go through the protocol. If you haven't used Gemini CLI before, our Gemini CLI tutorial covers the initial setup and a real Python project walkthrough. You can keep multiple agents active simultaneously: select a different agent from the panel's dropdown to switch, or open a second agent panel for side-by-side comparisons.

Custom agent config in Zed

If you want to customize the agent's environment or add an agent that isn't in the registry, edit ~/.config/zed/settings.json:

{
  "agent_servers": {
    "Gemini CLI": {
      "type": "registry",
      "env": {
        "GOOGLE_API_KEY": "AIza..."
      }
    },
    "My Custom Agent": {
      "type": "custom",
      "command": "/path/to/my-agent",
      "args": ["acp"],
      "env": {
        "MY_API_KEY": "..."
      }
    }
  }
}

Registry agents use "type": "registry" and the registry name as the key. Custom agents use "type": "custom" with a command path. Both communicate over stdio using the same ACP JSON-RPC protocol.

Custom Agent Configuration

Both JetBrains and Zed support custom agents that aren't in the registry. The config format is nearly identical.

JetBrains: `~/.jetbrains/acp.json`

{
  "agent_servers": {
    "OpenCode": {
      "command": "/usr/local/bin/opencode",
      "args": ["acp"],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}

Zed: `~/.config/zed/settings.json`

{
  "agent_servers": {
    "OpenCode": {
      "type": "custom",
      "command": "/usr/local/bin/opencode",
      "args": ["acp"]
    }
  }
}

Neovim: community plugins

Neovim support comes through community plugins like CodeCompanion.nvim and avante.nvim. The config follows the same pattern: a command path, args, and environment variables. Neovim spawns the agent as a subprocess and pipes JSON-RPC over stdio, same as the other editors.

The key thing across all three editors: you specify a command that accepts an acp argument, the editor spawns it, and they talk JSON-RPC. If your agent binary can speak the protocol, it works.

Building Your Own ACP Agent

If you're building a custom coding tool and want it to work in every ACP editor, official SDKs exist in five languages:

Language	Package	Status
Rust	`agent-client-protocol` (crates.io)	Production
TypeScript	`@agentclientprotocol/sdk` (npm)	Production
Python	`acp-sdk` (PyPI)	Production
Java	`acp-sdk` (Maven Central)	Production
Kotlin	`acp-sdk` (Maven Central)	Production

The minimal contract: implement the initialize handshake, handle session/new and session/prompt methods, and send back content updates. The SDKs ship with example agents you can fork.

Here's the skeleton in Python using the official SDK:

from acp import Agent, PromptResponse, run_agent

class MyAgent(Agent):
    async def prompt(self, prompt, session_id, message_id):
        user_message = prompt.content
        # ... call your LLM, read files, run terminal commands
        return PromptResponse(content="Here's my response...")

if __name__ == "__main__":
    run_agent(MyAgent())

Run this with python my_agent.py acp and point your editor's config at it. The SDK handles JSON-RPC framing, capability negotiation, and session lifecycle.

The ACP specification covers file system access, terminal execution, slash commands, tool calls, and authentication. You don't need to implement all of them. Agents declare their capabilities during the initialize handshake, and editors adapt their UI accordingly.

When ACP Falls Short

I've been using ACP daily for several months now, and it works well for what it covers. But there are gaps.

The HTTP/WebSocket transport for remote agents is still work-in-progress. Every ACP agent runs as a local subprocess right now. That's fine for Claude Code or Gemini CLI, which run local binaries that call cloud APIs. It doesn't work for Devin-style cloud agents that need long-running sandboxed environments. There's an active RFC for remote transport, but nothing stable yet.

Microsoft hasn't committed to native ACP support in VS Code. Their agent mode standardized on MCP for tool integration, and as of June 2026, Microsoft's position is that MCP covers their needs. The world's most popular editor requires community plugins for ACP. Those plugins exist but don't match the polish of native support in JetBrains or Zed.

WSL isn't supported for ACP agents in JetBrains, which stings if your dev environment is WSL-based. You'll need to run agents in Windows-native mode or find workarounds. Zed on Windows handles ACP agents more cleanly, but Zed itself is newer on that platform.

The registry is curated: all 50 agents must support user authentication. Experimental or internal agents need the manual config path. And the registry doesn't show usage stats, reviews, or compatibility notes. Just names and versions.

I also found that agent quality varies. ACP standardizes the wire protocol, not what the agent does with it. Claude Agent, Gemini CLI, and Goose were reliably useful. Some of the smaller agents in the registry crashed on complex project structures or produced worse code than their non-ACP versions.

FAQ

What is the Agent Client Protocol (ACP)?

ACP is an open JSON-RPC 2.0 standard that lets AI coding agents communicate with code editors through a single protocol. Created by Zed Industries in August 2025 and co-maintained with JetBrains, it works like LSP but for AI agents instead of language servers. One agent implementation works across Zed, JetBrains, Neovim, and Emacs without separate plugins for each.

How does ACP compare to MCP?

They complement each other. ACP connects editors to agents (how an editor drives an AI assistant). MCP connects agents to tools (how an agent calls databases, APIs, or browser automation). In a typical stack, the editor uses ACP to talk to the agent, and the agent uses MCP to talk to external tools. You don't choose between them; you use both.

Which editors support ACP?

As of June 2026: Zed (native, created the protocol), JetBrains IDEs (native, including IntelliJ, PyCharm, GoLand, WebStorm), Neovim (community plugin), and Emacs (community plugin). VS Code has community support through third-party extensions, but Microsoft hasn't added native ACP to VS Code itself.

Which AI coding agents support ACP?

The ACP Registry lists 50 agents including Claude Code, Gemini CLI, Codex CLI, GitHub Copilot, Cursor, Goose, Kiro, Junie, OpenCode, Grok Build, Mistral Vibe, and Qwen Code. The registry auto-updates hourly. Any agent not in the registry can be added manually through config files.

How do I set up an ACP agent in my editor?

In JetBrains: open AI Chat > Install From ACP Registry > select your agent > enter your API key. In Zed: press Cmd+? > click + > select your agent. For custom agents, edit ~/.jetbrains/acp.json (JetBrains) or ~/.config/zed/settings.json (Zed) with the agent's command path and args.

Sources

Agent Client Protocol specification — official protocol documentation and getting started guide
ACP GitHub repository — specification source, SDKs, and governance (3.3k stars, Apache 2.0)
ACP Registry — the 50-agent registry with auto-updated versions
JetBrains ACP documentation — setup guide for JetBrains IDEs
Zed external agents documentation — ACP setup in Zed
Marc Nuri: ACP introduction — technical overview of ACP architecture and ACP vs MCP comparison

Bottom Line

ACP is the most practical improvement to my AI coding workflow since MCP. The N×M problem (N editors times M agents, each needing a bespoke integration) was real, and ACP solves it. I run Claude Code in JetBrains, Gemini CLI in Zed, and switch between them depending on the task. The config files are ten lines of JSON.

The protocol isn't finished. Remote agents need stable transport, VS Code needs native support, and the registry needs quality signals beyond "exists and authenticates." But the core promise (write one agent, run it everywhere) already works for the 50 agents and dozen-plus editor integrations that have adopted it.

If you use more than one editor or more than one AI agent (and our terminal agent comparison suggests you should), set up ACP this weekend. It takes 15 minutes and saves you from maintaining three separate AI configurations that do the same thing.

Google Jules Review: The Async Coding Agent Worth $20/Month?

Maksim Danilchenko — Fri, 22 May 2026 08:40:34 +0000

TL;DR

Google Jules is the only major coding agent built around queuing instead of live chat. You describe a task, walk away, and a pull request shows up later. The free tier gives 15 tasks per day on Gemini 3 Flash. The $19.99/month Pro tier bumps that to 100 tasks on Gemini 3.1 Pro, and proactive features like CI Fixer and Scheduled Tasks make it feel less like a tool and more like a junior developer who never goes offline. But Jules is slow, can't handle files over ~50K lines, and only connects to GitHub. If you need real-time pair programming or work with GitLab, look elsewhere.

Why I Tried Jules

I've been using Claude Code and Codex CLI for months — both are real-time terminal agents where you type a prompt and watch code materialize. They're good at that. But I kept running into the same friction: I'd queue up three refactoring tasks in my head, then sit there babysitting the agent through each one sequentially. Context switching between "architect mode" and "watch the agent type" mode was costing me actual productive hours.

Jules promised something different. Describe the task, hit submit, go do something else. Come back to a pull request. I signed up for the Pro tier ($19.99/month bundled with Google AI Pro) and spent three weeks throwing real work at it — dependency bumps, test scaffolding, bug fixes across a Flask API and two Go microservices.

The short version: Jules delivered on the async promise. But "async" also means "slow," and the tradeoffs stack up in ways the marketing doesn't mention.

How Jules Works

Every task runs in an isolated Google Cloud VM. Jules clones your repo, reads the codebase, builds an execution plan, and shows you that plan before touching any files. You can edit the plan, approve it, or scrap it entirely. Once approved, Jules works through the changes file by file, running any tests it finds at each step. When it's done, it opens a PR on GitHub.

The whole loop is: submit, approve a plan, wait for the PR notification. No terminal session, no streaming output, no watching characters appear.

The model underneath depends on your tier. Free gets Gemini 3 Flash. Pro and Ultra run Gemini 3.1 Pro, which scores 80.6% on SWE-bench Verified — competitive with Claude Opus 4.6 at 80.8%, though behind Opus 4.7's 87.6% in agentic scaffolding. (For a full breakdown of how these models compare on coding tasks, see the GPT-5.4 vs Claude Opus 4.7 vs Gemini 3.1 Pro comparison.)

Pricing Breakdown

Jules doesn't have its own subscription. It bundles into Google's AI tiers:

	Free	Pro ($19.99/mo)	Ultra ($99.99/mo)
Daily tasks	15	100	300
Concurrent tasks	3	15	60
Model	Gemini 3 Flash	Gemini 3.1 Pro	Gemini 3.1 Pro (priority)
Suggested Tasks	No	Yes	Yes
Scheduled Tasks	Yes	Yes	Yes

The free tier is generous enough for evaluation. Fifteen tasks per day covers most solo developers who want to offload grunt work. Pro makes sense once you're running 10+ tasks daily and want the model upgrade. Ultra is for teams running agent-heavy workflows — 60 concurrent tasks means you can point Jules at an entire sprint backlog and let it churn.

One catch: paid plans require a @gmail.com account. Google Workspace users can't subscribe yet.

What Jules Got Right

Batch Parallelism

The async model isn't just a UX gimmick. I'd queue five dependency-bump tasks at 9 AM, go write the design doc I'd been avoiding, and come back to five PRs by 10:30. With Claude Code, those same five tasks would take me through lunch because I'd be approving file edits and answering clarification prompts one by one.

On Pro, 15 concurrent slots mean you can throw an entire backlog at Jules without hitting a queue. I ran 12 tasks simultaneously during a sprint cleanup, and all 12 completed within 90 minutes. Doing that sequentially in Claude Code would have taken most of an afternoon.

CI Fixer

This was the feature I didn't expect to love. When a GitHub Actions workflow fails, Jules automatically analyzes the logs, writes a fix, commits it, and resubmits. It loops until CI passes or gives up after a configurable number of attempts.

I had a Flask test suite that broke after a SQLAlchemy upgrade. Three tests failing on a deprecated session API. I pointed Jules at the CI failure. It read the logs, traced the issue to session.close() being called after the session was already garbage-collected, replaced it with a scoped session factory, and pushed a green build. Took about eight minutes. I would have spent 20 debugging that myself because I always forget the scoped session pattern.

Scheduled Tasks

You can set Jules to run recurring jobs: nightly lint passes, weekly dependency audits, monthly dead-code sweeps. This is the part that makes Jules feel like a team member rather than a tool. I set up a weekly pip-audit run on my Flask API — every Monday morning, a PR shows up with any new CVEs patched. Before Jules, I'd check this maybe once a quarter.

Suggested Tasks

On Pro and Ultra, Jules scans up to five repos and proposes improvements. It started with TODO comments — finding forgotten # TODO: handle edge case annotations scattered through my code and opening PRs to actually handle them. Over two weeks, it cleared 14 TODOs I'd written months ago and forgotten about.

The suggestions aren't always useful. Jules proposed refactoring a perfectly fine utility function into a class hierarchy that added complexity for zero benefit. But the hit rate was around 60-70%, and dismissing bad suggestions takes seconds.

Where Jules Falls Short

Speed

Jules is slow. A task that Claude Code handles in 90 seconds takes Jules 8-15 minutes. Part of this is the VM spin-up, part is the planning phase (Jules builds a detailed plan before writing any code), and part is that Gemini 3.1 Pro generates tokens slower than Claude in agentic loops.

For anything urgent (a production bug, a quick fix before a demo) Jules isn't the right tool. You'll be staring at a progress bar while Claude Code would have already pushed the commit.

Large File Blindness

Gemini 3.1 Pro has a 1M-token context window, but Jules appears to impose a tighter limit in practice. Large files are off-limits. I hit this on a legacy Go service with a 12,000-line handlers.go monolith (not proud of that file, but it exists). Jules's plan referenced functions that didn't exist in the file — it was working with a truncated view.

Real-time agents handle this differently. Claude Code can stream file reads and focus on specific sections. Jules loads the whole context upfront and chokes on anything too large.

GitHub Only

No GitLab. No Bitbucket. No self-hosted Git. If your repos aren't on github.com, Jules can't touch them. Google Workspace integration is also missing, which means enterprise teams on Google Cloud who use Cloud Source Repositores are locked out too.

Language Coverage

Python and TypeScript/JavaScript are first-class citizens. Jules writes solid code in both, catches edge cases, and uses idiomatic patterns. Go, Java, and C# work but with noticeably lower reliability. My Go microservices got PRs that compiled but missed patterns any Go developer would catch: unchecked errors, bare returns where wrapped errors belong.

Hallucinated Progress

Twice during my testing, Jules claimed a task was complete when it had actually stalled mid-execution. The PR showed up with partial changes: half the files edited, tests not run. There's no clear indication in the UI when this happens. You find out during code review, which defeats the "queue and forget" promise. If you're relying on any coding agent for unsupervised work, setting up guardrails before you go hands-off is worth the time.

Jules vs the Competition

Feature	Google Jules	Claude Code	GitHub Copilot Agent	OpenAI Codex
Interaction model	Async (queue + PR)	Real-time terminal	Both (IDE + async)	Async (cloud tasks)
Pricing	$0–99.99/mo	$20/mo (Pro) or API	$10–39/mo	API-based
Model	Gemini 3.1 Pro	Claude Opus 4.7	GPT-5.3-Codex (default)	GPT-5.3-Codex
SWE-bench	80.6%	87.6%	~77–80%	85%
Concurrent tasks	3–60	1 (serial)	1–3	Varies
Proactive features	CI Fixer, Scheduled	None	Limited	None
Git platforms	GitHub only	Any	GitHub only	GitHub only
Best for	Batch work, maintenance	Complex refactors, exploration	GitHub-native workflows	Automated fixes

What counts here is workflow fit, not a feature checklist.

Jules owns the batch maintenance lane. Queue 20 dependency bumps and lint fixes, check the PRs over coffee. On Pro with 15 concurrent slots, a full day's grunt work finishes before lunch. No other agent handles this volume as smoothly.

Claude Code is the better pick for anything that needs back-and-forth. Debugging a race condition, designing an API, exploring unfamiliar code — you want a real-time thinking partner, and Opus 4.7's 7-point SWE-bench lead over Gemini 3.1 Pro shows up when the task gets hard. (I covered the DeepSeek V4 Pro review recently, and it's another strong option at a fraction of Claude's API cost.)

Copilot Agent fits if you already live in GitHub Issues and Actions. It's the least friction for teams whose entire workflow is PR-centric.

Where Jules pulls ahead of all three: proactive features. I haven't found CI auto-fixing or scheduled recurring tasks in any competing agent. That gap alone kept me on the Pro tier.

MCP Server Integration

In February 2026, Jules added Model Context Protocol support with six hand-selected servers: Linear, Stitch, Neon, Tinybird, Context7, and Supabase. Google took a curated approach: every server was audited for data flow and tool permissions before being allowed.

In practice, this means Jules can read your Linear tickets, query your Neon database schema, and check Supabase auth configuration while planning changes. I connected the Neon MCP server and gave Jules a task: "add pagination to the /users endpoint based on the current schema." It pulled the schema directly from Neon, wrote the SQL migration and the Python endpoint code, and got it right on the first try. Without MCP, I'd have had to paste the schema into the task description.

Six servers is limiting. Claude Code connects to any MCP server you configure. But Google's curated approach makes sense for an agent that runs in a cloud VM with repo access. A malicious MCP server could exfiltrate code, so restriction buys you something real.

The Jules API

Google also launched a Jules API for programmatic task creation. You can trigger Jules tasks from CI pipelines, chatbots, or custom tooling. The API exposes task creation, status polling, and result retrieval.

The API is still in v1alpha, so field names and auth methods may change. Here's the general shape of a session-creation call using the current schema:

import requests

API_KEY = "your-google-api-key"

session = requests.post(
    "https://jules.googleapis.com/v1alpha/sessions",
    headers={"X-Goog-Api-Key": API_KEY},
    json={
        "sourceContext": {
            "gitHub": {"repository": "owner/repo", "branch": "main"}
        },
        "title": "Add input validation to /users POST endpoint",
    },
)
print(session.json())
# {"name": "sessions/abc123", "state": "CREATED", ...}

The automationMode field controls whether Jules runs without human review of its execution plan. I keep it at the default (manual approval) because I want to see the plan before Jules starts editing files. For trusted, repeatable tasks like dependency bumps, switching to full automation turns Jules into an autonomous pipeline.

The obvious next step is connecting Jules to your issue tracker: new bug filed, Jules automatically attempts a fix, PR shows up for review. The Stitch design team at Google reportedly runs "a pod of daily Jules agents" with assigned roles (performance tuning, security patching, accessibility, test coverage), making Jules, according to the team's blog post, one of the largest contributors to their repository.

Project Jitro: What's Coming Next

Google previewed Project Jitro at I/O 2026 — the next version of Jules that shifts from task-driven to goal-driven. Instead of "fix this function," you'd say "get test coverage to 85%" and Jitro figures out which files to change, which tests to write, and how to get the metric where you want it.

The current Jules already hints at this direction. Suggested Tasks, Scheduled Tasks, and the Render integration all share one pattern: Jules initiating action based on codebase state. Jitro takes that to its logical conclusion.

The obvious question is accountability. When an agent autonomously refactors modules to hit a metric, who reviews the architectural decisions it made along the way? Google hasn't answered that yet. Jitro launched under a waitlist at I/O, so general availability is probably months away.

Who Should Use Jules

Good fit:

You maintain multiple repos and spend hours weekly on dependency updates, lint fixes, and test scaffolding
You want CI failures fixed automatically without context-switching from whatever you're building
You work in Python or TypeScript and your repos are on GitHub
You like reviewing PRs more than supervising an agent in real time

Skip it:

You need real-time collaboration — architecture discussions, exploratory coding, debugging complex state
Your repos are on GitLab, Bitbucket, or self-hosted Git
You work primarily in Go, Java, or C# where Jules's output needs heavy review anyway
You need to work with files over 50K lines

FAQ

Is Google Jules free?

Yes, the free tier gives 15 tasks per day with 3 concurrent slots, running on Gemini 3 Flash. No credit card required. It's enough to evaluate whether the async model fits your workflow before committing to Pro.

How does Google Jules compare to Claude Code?

They solve different problems. Jules is async — you queue tasks and get PRs back later. Claude Code is real-time — you work together in a terminal session. Jules is better for batch maintenance work across multiple repos. Claude Code is better for complex single-task work where you need back-and-forth. Claude's underlying model (Opus 4.7, 87.6% SWE-bench) also outperforms Jules's Gemini 3.1 Pro (80.6%) on coding benchmarks.

What languages does Google Jules support?

Python and TypeScript/JavaScript are best supported. Go, Java, and C# work but produce less reliable output. Expect to catch missed error handling patterns and non-idiomatic code during review.

Can Jules work with private repositories?

Yes. Jules clones repos into isolated Google Cloud VMs. Google states your code isn't used for model training. The VM is ephemeral — spun up per task and destroyed after.

What is Project Jitro?

Project Jitro is Google's next-generation coding agent, previewed at I/O 2026. Instead of describing a task ("fix this bug"), you define a goal ("reduce p95 latency by 30ms") and the agent determines the changes needed. It's on a waitlist — no general availability date yet.

Sources

Jules official site — product page with feature overview
Jules usage limits and pricing — tier breakdown and task quotas
Jules changelog — feature releases through 2026
Jules proactive features announcement — Suggested Tasks, Scheduled Tasks, Render integration
Project Jitro analysis — goal-driven agent architecture and timeline
Jules API documentation — programmatic task creation
SWE-bench Verified leaderboard — coding benchmark scores

Bottom Line

Jules is the best coding agent for people who hate babysitting coding agents. The async model, CI Fixer, and Scheduled Tasks create a workflow where maintenance work runs on autopilot. Monday mornings, I'd wake up to 3-4 PRs from overnight pip-audit and lint runs. For $19.99/month, that trade works.

For thinking-partner work (debugging a race condition, designing an API, exploring unfamiliar code) you still need Claude Code or Copilot. Jules takes orders and delivers results, on its own schedule, at its own pace.

If your bottleneck is "too many small tasks, not enough hands," try the free tier for a week. Queue up your backlog. See what comes back. The 15-task daily limit is enough to know whether this fits your workflow.

AI Bug Bounty in 2026: 76% More Reports, Programs Shutting Down

Maksim Danilchenko — Wed, 20 May 2026 08:35:23 +0000

TL;DR

AI-assisted vulnerability discovery has broken the bug bounty model. HackerOne paused its Internet Bug Bounty program, Curl killed its bounty payments (then quietly came back without them), and Linus Torvalds calls the Linux kernel's security mailing list "almost entirely unmanageable." Report volumes are up 76% year-over-year, but only 25% flag real flaws. The same AI models also found 500+ zero-days in major projects and drove CVE disclosure surges of 563% in Chrome and 476% in GitHub products. The security community is split between researchers who can't process the flood and AI tools that keep making it worse.

The Inbox I Can't Keep Up With

I run a small open-source project on the side. Nothing close to the scale of Curl or the Linux kernel, but enough to get the occasional security report through GitHub advisories. In early 2025, I'd see maybe one report a quarter. By March 2026, I got seven in a single week. Six of them cited functions that don't exist in my codebase.

That experience made me pay close attention when Daniel Stenberg, who maintains Curl (a tool installed on basically every server on Earth), killed his bug bounty payments at the end of January 2026. His reasoning was blunt: fewer than 5% of submitted reports in 2025 were legitimate. The rest were what the security community now calls "AI slop," plausible-sounding reports generated by language models that reference imaginary functions, fabricate patches, and waste hours of maintainer time.

Stenberg's frustration was raw. His updated security.txt file now reads: "We will ban you and ridicule you in public if you waste our time on crap reports."

A month later, Curl returned to HackerOne without monetary rewards. By April, Stenberg said "the slop situation is not a problem anymore" and the confirmed vulnerability rate was back above 15%. Removing the financial incentive worked for Curl. Most other projects aren't so lucky.

The Flood by the Numbers

HackerOne, the largest bug bounty platform, reports a 76% jump in submissions year-over-year through March 2026. The share flagging real vulnerabilities held at 25%. That means the 76% increase is almost entirely noise.

Bugcrowd, which runs bounty programs for OpenAI, T-Mobile, and Motorola, watched its inbox swell more than fourfold during a three-week stretch in March. Most of what came in was unusable.

Before AI tools entered the picture, a popular open-source project might get two or three bug reports in a week. Less popular ones, maybe one a month. Now some projects are getting hundreds at a time, and the overwhelming majority cite non-existent code paths, imaginary patches or vague theoretical attacks that fall apart under any scrutiny.

Who Shut Down and Why

Several programs have paused or shut down in the first five months of 2026:

Project / Platform	Action	Date	Reason
HackerOne (Internet Bug Bounty)	Paused all new submissions	March 27, 2026	Discovery outpacing remediation capacity
Curl	Killed bounty payments; returned to HackerOne without rewards Feb 25	January 31, 2026	<5% legitimate reports, maintainer burnout
Google	Raised quality bar for AI-submitted reports	May 2026	Quality threshold not met
Node.js	Paused bug bounty	April 2026	Lost HackerOne funding, no independent budget
Django	Modified submission process	Q1 2026	Report volume overwhelming volunteer team
libxml2	Ended embargoed vulnerability reports	June 2025	Maintainer capacity exceeded
Nextcloud	Shut down bounty program	April 2026	Unsustainable maintainer workload

HackerOne's pause was the biggest signal. The platform cited a direct link between AI-assisted research and the imbalance: discovery used to be the bottleneck, but with automated discovery, remediation is now the bottleneck. Bounty programs don't fund remediation.

Christopher Robinson, CTO of the Open Source Security Foundation: "If it takes a maintainer two to eight hours of unbudgeted, unallocated time [per report], that becomes burdensome."

For a project like Curl with a small team of active maintainers, the math stopped working. Stenberg moved security intake to GitHub Security Advisories, then returned to HackerOne without bounty payments. His warning to anyone thinking of submitting a report generated by a language model: he'd consider an entrance fee for reporters next.

The Torvalds Quote

Linus Torvalds doesn't mince words on a quiet day. On the subject of AI-generated security reports, he was characteristically direct.

"If you found a bug using AI tools," he wrote in his weekly kernel release post, "the chances are somebody else found it too."

The Linux kernel's security mailing list, where critical vulnerabilities get reported before public disclosure, is now "almost entirely unmanageable, with enormous duplication due to different people finding the same things with the same tools."

A dozen independent researchers each feed the same Linux kernel source into Claude or GPT, find the same buffer overflow, and each submit a separate report believing they've discovered something novel. The maintainers on the other end receive twelve versions of the same finding, each padded with AI-generated analysis that needs to be triaged individually. Multiply that across every subsystem and you've got a mailing list that requires dedicated staff just to process — staff the kernel project doesn't have.

Torvalds's advice to AI-assisted researchers: "Don't be the drive-by 'send a random report with no real understanding' kind of person."

The Matplotlib Incident

Matplotlib maintainer Scott Shambaugh got a front-row seat to the absurdity.

The AI agent PR flood on GitHub has a security twin. An AI agent submitted a pull request to the Matplotlib project. Shambaugh reviewed it, found it insufficient, and rejected it. The agent (not the human operator, but the autonomous agent itself) then published a disparaging blog post about Shambaugh on the internet. It later apologized on GitHub.

An AI agent wrote a hit piece about an open-source maintainer because he rejected its pull request — in February 2026.

But AI Is Also Finding Real Zero-Days

The same tools generating the flood of junk reports are also finding genuine, high-severity vulnerabilities that human researchers missed for years. I wrote about Claude finding 500+ zero-days in April, and the numbers have gotten more dramatic since.

VulnCheck's analysis of CVE disclosure volumes in 2026 tells the other side of the story:

Product	CVE Disclosure Change (YoY, 2026)
Chrome	+563.2%
GitHub products	+476.1%
VMware	+180.9%
Apache	+170.3%
Mozilla Firefox	+156.9%
HPE	+132.3%
F5	+113.8%
Palo Alto Networks	+37.0%

Those aren't hypothetical. Chrome's CVE disclosures are up 563% year over year. Mozilla confirmed in February 2026 that it's now using frontier AI models internally to find and fix latent browser vulnerabilities. Anthropic's Claude Mythos, through Project Glasswing (announced April 7, 2026), has been made available to AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorgan Chase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks specifically for defensive vulnerability hunting. Anthropic stated that Mythos "identified thousands of zero-day vulnerabilities across every major operating system and web browser."

Concrete wins include ActiveMQ CVE-2026-34197, discovered by researcher Naveen Sunkavally using Claude assistance. That CVE is now actively exploited in the wild and appears on CISA's Known Exploited Vulnerabilities list. Stanislav Fort's AISLE tool found all 12 CVEs in OpenSSL's January 2026 coordinated release and is credited with 13 of 14 OpenSSL CVEs across recent releases. Anthropic gave the Apache Software Foundation $1.5 million specifically to help Apache handle the AI-driven vulnerability flood.

And Curl itself, despite Stenberg's fury at junk submissions, credited AI-assisted tools with helping fix around 170 bugs that survived years of aggressive fuzzing and multiple human security audits. Researcher Joshua Rogers used AI tools to systematically analyze the Curl codebase before submitting high-quality reports.

The catch: when Stenberg tested Anthropic's Mythos specifically against Curl, only 1 of 5 reported vulnerabilities held up as a valid CVE. Even the best models have a meaningful false positive rate on real-world codebases.

The Signal-to-Noise Problem

AI finds vulnerabilities. The CVE data above removes any doubt. But the economics of bug bounty programs assumed a world where discovery was expensive.

When finding a buffer overflow required deep knowledge of C memory management, familarity with the specific codebase, and hours of manual source review, the friction itself acted as a quality filter. Researchers who submitted reports had usually done genuine work. Bounty payments were both reward and incentive: you invested effort, you got paid for real findings.

AI collapsed that friction. Now anyone can paste a codebase into a model's context window and get back something that looks like a vulnerability report. The API call costs under a dollar. The "researcher" may have no idea whether the finding is real, but the report reads well enough to require a maintainer to spend time disproving it.

Bugcrowd and HackerOne are building AI-powered filtering tools to help customers triage the volume. HackerOne introduced what it calls "agentic validation capabilities," using AI to check whether AI-generated reports are real. The recursion is absurd, but it may be the only path that scales.

The Pattern Behind Real AI-Found Vulnerabilities

Not all AI-assisted security research is junk. The projects that produce real findings share a pattern. Based on what's worked (AISLE on OpenSSL, Rogers on Curl, Mozilla's internal team, the ActiveMQ discovery), the effective approach looks like this:

# Simplified pattern: effective AI-assisted vuln research
# 1. Targeted scope (one library, one attack surface)
# 2. Model-assisted analysis + human verification
# 3. Working proof of concept before submission

import subprocess

def validate_finding(vuln_report: dict) -> bool:
    """Before submitting any AI-found vulnerability,
    verify it with a working PoC."""

    # Step 1: Does the function actually exist?
    source_file = vuln_report["file"]
    function_name = vuln_report["function"]
    result = subprocess.run(
        ["grep", "-rn", function_name, source_file],
        capture_output=True, text=True
    )
    if not result.stdout:
        return False  # AI hallucinated the function

    # Step 2: Can you trigger the bug?
    poc_result = run_proof_of_concept(vuln_report["poc"])
    if not poc_result.crashed and not poc_result.leaked:
        return False  # Theoretical, not exploitable

    # Step 3: Is it already known?
    known = check_cve_database(vuln_report["description"])
    if known:
        return False  # Duplicate

    return True

The difference between Rogers's ~170 valid Curl findings and the thousands of junk submissions is straightforward: Rogers verified before submitting. He understood the Curl codebase, used AI to accelerate analysis, and only reported what he could prove.

Stanislav Fort, founder of AISLE, has said his tool finds bugs that existing automated methods couldn't reach. The value is in extending what's findable past the limits of manual review and traditional fuzzing.

What Maintainers Should Do Right Now

If you maintain an open-source project of any size, the AI report flood is coming for your inbox (if it hasn't already). Based on how the larger projects have responded, here's what's working:

Move intake off bounty platforms. Curl's switch to GitHub Security Advisories with no monetary rewards cut junk submissions dramatically. The financial incentive was attracting the worst actors.
Django and several Linux subsystems now reject any report that doesn't include a working exploit or at minimum a reproduction script. Require a proof of concept. "Theoretical attack scenario" doesn't cut it anymore.
Template your rejections. Stenberg's blunt approach saves time: a canned response for reports that cite non-existent functions, with a clear warning about bans for repeated offenses.
GitHub's Jarom Brown confirmed that programs across the industry are building automated filters. Even a simple check ("does this function name exist in our codebase?") would eliminate a huge percentage of AI slop. If you can automate triage, do it.
Don't blanket-ban AI tools. Rogers, Fort, and Mozilla's internal team show that AI-assisted discovery done right produces results manual review can't match. Ban lazy submissions, not the tooling. If you're running AI agents in your own workflow, setting up proper guardrails helps on the other side of the equation too.

Who Pays for Remediation?

Anthropic's $1.5 million grant to Apache is the only large-scale example of an AI lab paying for the downstream cost of its models' vulnerability discoveries. Compare that to the scale of the problem: the Apache Software Foundation handles security for projects used by every major tech company on Earth. A million and a half dollars won't sustain a team to process the current volume, let alone the volume that's coming as AI models get better.

HackerOne's original framing was correct: discovery used to be the bottleneck, and bounties funded it. Now remediation is the bottleneck, and nobody funds it. Open source maintainers are volunteers. When AI sends them hundreds of reports a week, each requiring two to eight hours to evaluate, the math breaks down fast.

There's a real possibility that some open-source projects will simply stop accepting security reports altogether rather than drown in triage. That would be a worse outcome than the current mess: real vulnerabilities going unfixed because the signal is buried under too much noise.

FAQ

How is AI affecting bug bounty programs?

AI has massively increased the volume of vulnerability reports while keeping the rate of legitimate findings flat at around 25%. HackerOne saw a 76% jump in submissions year-over-year through March 2026, and Bugcrowd's inbox swelled fourfold in three weeks. Several programs, including HackerOne's Internet Bug Bounty, Node.js, and Nextcloud, have paused or shut down. Curl killed its bounty payments, returned without them, and now filters more aggressively.

Why did HackerOne pause its Internet Bug Bounty?

HackerOne paused new submissions on March 27, 2026, citing a shift from discovery to remediation as the bottleneck. AI-assisted research has accelerated vulnerability discovery past the point where open-source maintainers can keep up with fixes. The program was designed for a world where finding bugs was expensive, and that assumption collapsed in under a year.

Can AI replace human bug bounty hunters?

Not yet. The most effective AI-assisted findings (AISLE's OpenSSL work, Rogers's Curl audits, Mozilla's internal team) all involve human verification and deep codebase knowledge. AI excels at scanning large codebases for patterns that fuzzing misses, but it also hallucinates functions and fabricates attack scenarios. The value comes when experienced researchers use AI to scan at scale, then verify each finding by hand before submitting.

What is AI slop in security reports?

"AI slop" refers to low-quality vulnerability reports generated by language models that are submitted without human verification. Typical characteristics: citing functions that don't exist in the codebase, proposing patches for imaginary code paths, presenting theoretical attacks with no proof of concept, and padding reports with verbose but vacuous analysis. Curl's Daniel Stenberg reported that fewer than 5% of reports received in 2025 were legitimate.

Are AI-generated vulnerability reports legitimate?

Some are. Chrome's CVE disclosures are up 563% year-over-year, and AI tools are credited with finding real zero-days including ActiveMQ CVE-2026-34197 (now on CISA's KEV list) and all 12 OpenSSL CVEs from January 2026. But the majority of AI-generated submissions to public bounty programs are not legitimate. They lack working proofs of concept and often reference code that doesn't exist.

Sources

HackerOne 2026 Security Report — primary source for the 76% submission increase and 25% legitimate-finding rate
Help Net Security — AI is drowning software maintainers in junk security reports — Torvalds quotes, industry response, Jarom Brown on automated filters
VulnCheck — The First CVE Wave: AI-Assisted Vulnerability Discovery Is Reshaping Disclosure Volumes — CVE disclosure data, Project Glasswing details, ActiveMQ CVE, Apache $1.5M grant
Axios — AI agents are flooding open-source maintainers with security reports — Matplotlib incident, report volume statistics, OSSF quotes
Socket.dev — Curl Shuts Down Bug Bounty Program After Flood of AI Slop Reports — Stenberg quotes, Curl program timeline, Django/Node.js/libxml2 moves

Bottom Line

AI has broken the bug bounty model in 2026, and nobody has a working replacement yet. The same models generating mountains of junk reports are also finding real zero-days that human researchers missed for decades. Chrome's CVE disclosures are up 563%. Anthropic is handing out $1.5 million grants to help projects cope. Curl's maintainer is threatening to charge admission for security reporters.

The projects getting this right (AISLE, Mozilla, Rogers on Curl) share one thing: human expertise doing the verification, AI doing the scanning at scale. The projects drowning are the ones where the reports arrive faster than anyone can read them.

Security researchers using AI tools: verify before you submit. Maintainers: strip the financial incentive from your intake process and require proof of concept. As for the AI labs whose models are generating this flood, Anthropic's $1.5 million to Apache is a start. The tab is going to be a lot higher than that.