DEV Community: Cursuri AI

The Batch API Halves Your Claude Bill — Unless Your Prompt Cache Was Already Doing the Work

galian — Fri, 02 Oct 2026 11:38:58 +0000

The Message Batches API has the simplest pitch in Anthropic's pricing table: send work that doesn't need an answer right now, get the results back later (most batches finish within an hour, the hard limit is 24), and pay half of every token. For plenty of workloads it is the easiest cost cut you will ever ship.

For one very common workload it isn't a cut at all. Picture a job that sends a large shared prefix (instructions, a label taxonomy, few-shot examples, a policy document), a small per-item payload, and asks for a short answer. If you already run that job synchronously with prompt caching, most of your bill is cache reads. Batch halves those too. But inside a batch, cache hits are best-effort, and Anthropic's documentation puts typical hit rates anywhere between 30% and 98%. Every miss is billed as a cache write, which costs 25 times a cache read on Opus 5.5. Move that job to batch at the wrong hit rate and it costs several times what it cost synchronously.

What follows applies the cost-versus-reliability reasoning from the production LLM integration course at Cursuri-AI.ro to this one API: the arithmetic on current list prices, a function that makes the call, the documented recipe for raising batch hit rates, and the operational details that decide whether you lose data (idempotent custom_ids, unordered results, the four result types, re-submitting only what failed, the 24-hour and 29-day clocks).

Prompt caching itself (prefix matching, breakpoints, silent invalidators) is assumed knowledge here.

What "50% off" actually applies to

The discount covers all usage: input, output and, because the caching multipliers stack with the Batch API discount, cache writes and cache reads as well. For the three current models, in USD per million tokens, synchronous → batch:

Model	Input	5-min cache write	1-hour cache write	Cache read	Output
Claude Fable 5.1	$10 → $5	$12.50 → $6.25	$20 → $10	$0.25 → $0.125	$50 → $25
Claude Opus 5.5	$4 → $2	$5 → $2.50	$8 → $4	$0.20 → $0.10	$20 → $10
Claude Sonnet 5.5	$2 → $1	$2.50 → $1.25	$4 → $2	$0.20 → $0.10	$10 → $5

Two things in that table matter more than the headline.

First, the batch input price is far above the synchronous cache-read price: ten times on Opus 5.5 ($2 vs $0.20), five times on Sonnet 5.5, twenty times on Fable 5.1. Halving a token you were already reading from cache at a few percent of list price saves almost nothing in absolute terms. Halving it and then losing the cache hit costs you a great deal.

Second, the read multiplier differs across the lineup. Fable 5.1 reads cache at 0.025× its input price, Opus 5.5 at 0.05×, Sonnet 5.5 at the standard 0.1×. The cheaper the read, the less of your bill is left for the batch discount to work on, and the more a miss hurts by comparison.

A worked example: 10,000 classifications against a 50K prefix

Take a hypothetical but typical backfill on Claude Opus 5.5 (Anthropic's recommended starting model for most workloads):

Shared prefix: 50,000 tokens of instructions, label definitions and examples, identical on every request.
Per item: 500 tokens of fresh input (the ticket, the document, the record).
Output: 150 tokens. On Opus 5.5 adaptive thinking is always on, and thinking tokens are billed as output, so treat this as "measured output including thinking at low effort", not as the length of the JSON you get back.

Run synchronously with a steady request rate, the prefix is written once and then read on nearly every call. Per request: $0.010 for the cached prefix, $0.002 for the fresh input, $0.003 for the output, so $0.015. Here is the same job on batch, for all 10,000 items:

Setup (Opus 5.5, 10,000 requests)	Cost
Synchronous, prompt caching, ≈100% hits	$150
Batch, no prompt caching	$1,025
Batch, caching, 98% hit rate	$99
Batch, caching, 95% hit rate	$135
Batch, caching, 90% hit rate	$195
Batch, caching, 60% hit rate	$555
Batch, caching, 30% hit rate	$915

These figures use 5-minute writes for the misses. With 1-hour writes a miss costs more ($4 instead of $2.50 per million batch tokens), so 90% becomes $270 and 60% becomes $855.

Read the table from the bottom. Batch without caching costs almost seven times the cached synchronous job; batch with caching at the low end of the documented range, six times. Batch wins only near the top of that range, and the gap between winning and losing badly is a few points of hit rate you don't directly control. You may well land high, and the recipe below pushes you that way, but you won't know until you measure.

The break-even, for any workload

The comparison generalizes. Let P be the shared prefix in tokens, r the synchronous cache-read price, w the synchronous cache-write price for the TTL you use in the batch, and U the uncacheable part of each request, priced the same way (fresh tokens × input price + output tokens × output price). If your synchronous job already hits the cache on essentially every request, batch is cheaper only when its hit rate h satisfies:

h > (w - 2r - U/P) / (w - r)

For the example shape (50K prefix, 500 fresh tokens, 150 output tokens):

Model	Break-even hit rate, 5-min writes	Break-even hit rate, 1-hour writes
Claude Fable 5.1	95.9%	97.5%
Claude Opus 5.5	93.8%	96.2%
Claude Sonnet 5.5	89.1%	93.4%

The threshold moves fast with output size, because output is the part caching can't touch and batch always halves:

Opus 5.5, 50K prefix, 5-min writes	Break-even hit rate
150 output tokens	93.8%
1,000 output tokens	86.7%
4,000 output tokens	61.7%
8,000 output tokens	28.3%
above ~11,400 output tokens	none: batch always wins

And it moves the other way with prefix size: with 150 output tokens on Opus 5.5, a 2K prefix needs about 44%, a 5K prefix 75%, a 20K prefix about 91%, a 200K prefix about 95%.

Batch's advantage grows with everything that can't be cached (long answers, heavy thinking, large per-item inputs) and shrinks as the shared prefix grows. Short answers over a huge shared context are exactly where synchronous caching is hardest to beat. Long generations over a modest prompt are where batch is a clean 50% cut.

Here is the decision as code. Prices are synchronous list prices; the function applies the discount itself:

# Synchronous list prices, USD per million tokens (Anthropic pricing page, 2 Oct 2026)
PRICES = {
    "claude-fable-5-1":  {"input": 10.00, "write_5m": 12.50, "write_1h": 20.00, "read": 0.25, "output": 50.00},
    "claude-opus-5-5":   {"input": 4.00,  "write_5m": 5.00,  "write_1h": 8.00,  "read": 0.20, "output": 20.00},
    "claude-sonnet-5-5": {"input": 2.00,  "write_5m": 2.50,  "write_1h": 4.00,  "read": 0.20, "output": 10.00},
}
BATCH_DISCOUNT = 0.5  # applies to every column above, cache reads and writes included
MIN_CACHEABLE = 512   # tokens, for all three models above


def cost_per_request(model: str, prefix: int, fresh: int, output: int,
                     hit_rate: float, ttl: str = "5m", batch: bool = False) -> float:
    """Expected USD per request. A miss on a cache_control prefix is billed as a cache write."""
    p = PRICES[model]
    write = p["write_1h"] if ttl == "1h" else p["write_5m"]
    prefix_rate = hit_rate * p["read"] + (1 - hit_rate) * write
    usd = (prefix * prefix_rate + fresh * p["input"] + output * p["output"]) / 1_000_000
    return usd * BATCH_DISCOUNT if batch else usd


def breakeven_hit_rate(model: str, prefix: int, fresh: int, output: int,
                       sync_hit_rate: float = 1.0, ttl: str = "1h") -> float:
    """Lowest batch cache hit rate at which batch beats your current synchronous job."""
    p = PRICES[model]
    write = p["write_1h"] if ttl == "1h" else p["write_5m"]
    sync = cost_per_request(model, prefix, fresh, output, sync_hit_rate)
    uncacheable = (fresh * p["input"] + output * p["output"]) / 1_000_000
    needed_prefix_rate = (sync / BATCH_DISCOUNT - uncacheable) * 1_000_000 / prefix
    return (write - needed_prefix_rate) / (write - p["read"])


def should_batch(model: str, prefix: int, fresh: int, output: int, *,
                 sync_hit_rate: float = 1.0, measured_batch_hit_rate: float | None = None,
                 ttl: str = "1h", hard_deadline_under_24h: bool = False,
                 zdr_or_hipaa: bool = False) -> tuple[bool | None, str]:
    if zdr_or_hipaa:
        return False, "Batch is not ZDR- or HIPAA-eligible: inputs and outputs are stored up to 29 days."
    if hard_deadline_under_24h:
        return False, "A batch can end with requests expired, unprocessed, after 24 hours."
    if prefix < MIN_CACHEABLE:
        return True, "Nothing here can be cached, so batch is a flat 50% cut."
    need = breakeven_hit_rate(model, prefix, fresh, output, sync_hit_rate, ttl)
    if need <= 0:
        return True, "Uncacheable tokens dominate: batch wins even with zero cache hits."
    if measured_batch_hit_rate is None:
        return None, f"Run a pilot batch first: you need a cache hit rate above {need:.1%}."
    if measured_batch_hit_rate > need:
        return True, f"Measured {measured_batch_hit_rate:.1%} beats break-even {need:.1%}."
    return False, f"Measured {measured_batch_hit_rate:.1%} is below break-even {need:.1%}: stay synchronous."

sync_hit_rate is there because your synchronous job may not be the ideal baseline either. A job that fans out hundreds of parallel requests can't read a cache entry until the first response that wrote it has started, so it pays for many writes at the start; plug in what your logs show. And price isn't the only input. Batch traffic has its own rate limits and doesn't count against your Messages API limits, so a backfill on batch doesn't compete with production traffic. If your synchronous backfill is throttling user-facing requests, that can justify batch at break-even.

Raising the batch hit rate: the documented recipe

The prompt-caching docs describe a specific pattern for batches. Each step below closes one way a batch request can miss.

Put an explicit breakpoint at the end of the shared prefix. Don't rely on automatic caching here. It places the breakpoint on the last cacheable block, which in a batch is the per-item content: every request writes an entry no other request will ever read.

Make every request render the same prefix, byte for byte. Same tools, same system, and also the same thinking configuration, output_config.effort and output_config.format. Those settings are rendered into the prompt, so a request at a different effort or with a different schema has a different prefix, even when the text is identical.

Use the 1-hour TTL. The docs note that batch requests commonly take between 5 minutes and an hour to complete, which is longer than the default 5-minute cache lifetime. The 1-hour write costs more, and the break-even function accounts for it.

Prime the cache with a one-request batch. Submit a batch containing a single request that carries the shared prefix and the 1-hour cache block. When it ends, submit the rest. The primer can be one of your real items, so nothing is wasted.

Stay above the minimum. The minimum cacheable prompt length is 512 tokens on Fable 5.1, Opus 5.5 and Sonnet 5.5. Below it nothing is cached and no error is returned.

Stay in one workspace. Both batches and prompt caches are isolated per workspace on the Claude API, so a primer written from another workspace's key primes nothing for you.

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-opus-5-5"
# INSTRUCTIONS, TAXONOMY_AND_EXAMPLES and LABEL_SCHEMA are your own constants.

SHARED_SYSTEM = [
    {"type": "text", "text": INSTRUCTIONS},
    {
        "type": "text",
        "text": TAXONOMY_AND_EXAMPLES,                         # the large shared prefix
        "cache_control": {"type": "ephemeral", "ttl": "1h"},   # explicit breakpoint
    },
]
# Effort and output format are rendered into the prompt: keep them identical everywhere.
OUTPUT_CONFIG = {"effort": "low", "format": {"type": "json_schema", "schema": LABEL_SCHEMA}}


def params_for(item_text: str) -> dict:
    return {
        "model": MODEL,
        "max_tokens": 2048,            # thinking counts toward max_tokens
        "system": SHARED_SYSTEM,
        "output_config": OUTPUT_CONFIG,
        "messages": [{"role": "user", "content": item_text}],
    }


def submit_primed(items: list[tuple[str, str]]) -> tuple[str, str]:
    """items: (custom_id, text). One request writes the 1-hour cache, then the rest follow."""
    first_id, first_text = items[0]
    primer = client.messages.batches.create(
        requests=[{"custom_id": first_id, "params": params_for(first_text)}],
    )
    wait_for(primer.id)                # defined in the polling section below
    main = client.messages.batches.create(
        requests=[{"custom_id": cid, "params": params_for(text)} for cid, text in items[1:]],
    )
    return primer.id, main.id

Then measure. Every succeeded result carries the full usage object, so your real hit rate is one division away: cache_read_input_tokens / (cache_read_input_tokens + cache_creation_input_tokens), summed across results. That token-weighted ratio is the h the break-even formula wants. Run a pilot of a few thousand items, compute it, feed it to should_batch(), and only then move the full job.

Running it in production without losing results

The economics decide whether to batch. The operational details decide whether every item comes back exactly once. A batch job has more ways to lose data quietly than a synchronous loop: results arrive out of order, failures are reported per request inside a "successful" batch, and everything has an expiry date.

`custom_id` is your idempotency key

Each request carries a custom_id of 1 to 64 characters matching ^[a-zA-Z0-9_-]{1,64}$. Results can come back in any order, so the custom_id is the only reliable link between a result and its input. Never zip results against your input list.

The API requires uniqueness only within a batch. Nothing stops you from submitting the same item in two batches and paying for it twice. That makes the custom_id your job, and the useful version is deterministic: derive it from the record's identity and the prompt version, so the same item under the same prompt always maps to the same ID, and a new prompt produces new IDs. Then keep a ledger keyed by it. The same discipline applies to any idempotent data pipeline: the key, not the run, defines what has been processed.

import hashlib
import json
import sqlite3

PROMPT_VERSION = "labels-v3"   # bump on any change to model, prompt, schema or effort

db = sqlite3.connect("batch_ledger.db")
db.execute("""
    CREATE TABLE IF NOT EXISTS ledger (
        custom_id   TEXT PRIMARY KEY,
        record_id   TEXT NOT NULL,
        status      TEXT NOT NULL DEFAULT 'pending',  -- pending|submitting|submitted|succeeded|retry|failed
        batch_id    TEXT,
        attempts    INTEGER NOT NULL DEFAULT 0,
        error       TEXT,
        result_json TEXT
    )
""")


def make_custom_id(record_id: str, prompt_version: str = PROMPT_VERSION) -> str:
    digest = hashlib.sha256(f"{prompt_version}:{record_id}".encode()).hexdigest()
    return f"r_{digest[:40]}"          # 42 chars, only [a-z0-9_]


def enqueue(record_ids: list[str]) -> None:
    db.executemany(
        "INSERT OR IGNORE INTO ledger (custom_id, record_id) VALUES (?, ?)",
        [(make_custom_id(rid), rid) for rid in record_ids],
    )
    db.commit()

INSERT OR IGNORE makes enqueueing safe to re-run: items already in the ledger keep their status.

Chunk by count and by bytes

A single batch is capped at 100,000 requests or 256 MB, whichever comes first; an oversized request body can come back as a 413 request_too_large. There is also a per-organization cap on batch requests waiting in the processing queue, which depends on your usage tier (200,000 at the Start tier, 500,000 at Scale), so a large backfill gets fed in waves anyway.

There is a second reason to keep batches smaller than the cap. Results become available only when every request in the batch has finished, or after 24 hours. Ten batches of 10,000 give you results as each finishes. One batch of 100,000 gives you nothing until the slowest request is done.

MAX_REQUESTS = 10_000          # hard cap is 100,000; smaller batches return sooner
MAX_BYTES = 200_000_000        # stay well under the 256 MB cap


def chunks(requests: list[dict]):
    batch, size = [], 0
    for req in requests:
        n = len(json.dumps(req).encode())
        if batch and (len(batch) >= MAX_REQUESTS or size + n > MAX_BYTES):
            yield batch
            batch, size = [], 0
        batch.append(req)
        size += n
    if batch:
        yield batch

Validate one request synchronously first

Validation of each request's params happens asynchronously, and validation errors come back only when the whole batch has ended. A typo in a parameter name turns into 10,000 errored results an hour later. Send one request with exactly the batch shape to the ordinary Messages API before submitting. While you're at it, remember the three parameters a batch rejects: stream: true, speed (fast mode) and max_tokens: 0.

Submit with an intent marker

There is a crash window between "the batch was created" and "the ledger knows its ID". Mark the rows submitting before the call and submitted after it. If a restart finds rows stuck in submitting, check client.messages.batches.list() for recent batches before re-submitting. If you do end up submitting twice, the deterministic custom_id means the second result overwrites the first rather than creating a duplicate: it costs money, not correctness.

MAX_ATTEMPTS = 3


def submit(rows: list[tuple[str, str]], build_params) -> str:
    """rows: (custom_id, record_id). build_params(record_id) returns the Messages params."""
    ids = [(cid,) for cid, _ in rows]
    db.executemany("UPDATE ledger SET status='submitting' WHERE custom_id=?", ids)
    db.commit()
    batch = client.messages.batches.create(
        requests=[{"custom_id": cid, "params": build_params(rid)} for cid, rid in rows],
    )
    db.executemany(
        "UPDATE ledger SET status='submitted', batch_id=?, attempts=attempts+1 WHERE custom_id=?",
        [(batch.id, cid) for cid, _ in rows],
    )
    db.commit()
    return batch.id

Poll gently, and don't expect a progress bar

processing_status moves from in_progress to ended (via canceling if you cancel). The request_counts object looks like a progress meter but isn't one: succeeded, errored, canceled and expired all stay at zero until the entire batch has ended, and only processing is meaningful before then. Poll with backoff: the Batches API has its own requests-per-minute limit, and a tight polling loop spends it.

import random
import time

from anthropic.types.messages import MessageBatch


def wait_for(batch_id: str, max_sleep: float = 300.0) -> MessageBatch:
    delay = 30.0
    while True:
        batch = client.messages.batches.retrieve(batch_id)
        if batch.processing_status == "ended":
            return batch
        time.sleep(delay + random.uniform(0, 5))
        delay = min(delay * 1.5, max_sleep)

expires_at on the batch object tells you when the 24-hour clock runs out. Any request not processed by then comes back as expired. The docs warn that processing can slow down with demand and with your own volume, and that more requests may expire as a result. Treat expiry as a normal outcome your retry path handles, not as an incident.

Four result types, and "succeeded" isn't the same as usable

Each result has one of four types:

Result type	Meaning	Billed?	What to do
`succeeded`	A message was created	Yes	Check `stop_reason` before trusting it
`errored`	No message: invalid request or server-side error	No	Retry server errors; fix `invalid_request_error` first
`canceled`	You canceled before it was sent to the model	No	Re-submit if still needed
`expired`	The 24-hour limit hit before it was sent to the model	No	Re-submit, ideally in a smaller batch

The column that trips people up is the first one. A succeeded result can still carry stop_reason: "max_tokens" (truncated; remember that thinking counts toward the budget), "refusal", or "pause_turn" (a server-tool loop that paused and needs continuing). None of those is a usable answer. Route them to review, not into your table of answers.

Stream the results instead of downloading them whole (the SDK's results() iterator reads the .jsonl line by line) and write each one to durable storage as it arrives:

RETRYABLE = {"api_error", "overloaded_error", "rate_limit_error", "timeout_error"}
USABLE_STOPS = {"end_turn", "stop_sequence"}


def collect(batch_id: str) -> dict:
    stats = {"succeeded": 0, "errored": 0, "canceled": 0, "expired": 0, "read": 0, "write": 0}
    for n, line in enumerate(client.messages.batches.results(batch_id), start=1):
        outcome = line.result
        stats[outcome.type] += 1
        payload = None
        if outcome.type == "succeeded":
            msg = outcome.message
            stats["read"] += msg.usage.cache_read_input_tokens or 0
            stats["write"] += msg.usage.cache_creation_input_tokens or 0
            if msg.stop_reason in USABLE_STOPS:
                status, error, payload = "succeeded", None, msg.model_dump_json()
            else:
                status, error = "failed", f"stop_reason:{msg.stop_reason}"
        elif outcome.type == "errored":
            error = outcome.error.error.type
            status = "retry" if error in RETRYABLE else "failed"
        else:                                   # canceled or expired: never reached the model
            status, error = "retry", outcome.type
        db.execute(
            "UPDATE ledger SET status=?, error=?, result_json=? WHERE custom_id=?",
            (status, error, payload, line.custom_id),
        )
        if n % 500 == 0:
            db.commit()
    db.commit()
    total = stats["read"] + stats["write"]
    stats["hit_rate"] = stats["read"] / total if total else None
    return stats

stats["hit_rate"] is the measured value to feed back into should_batch().

Re-submit only what failed

With the ledger in place, the whole job is a loop: pick rows that are pending or retry and under the attempt cap, chunk them, submit, wait, collect, repeat until nothing is left. Re-using the same custom_ids in a new batch is fine, since uniqueness is only required within one batch, and it keeps every attempt for an item under a single key.

def run(build_params) -> None:
    while True:
        rows = db.execute(
            "SELECT custom_id, record_id FROM ledger "
            "WHERE status IN ('pending', 'retry') AND attempts < ?",
            (MAX_ATTEMPTS,),
        ).fetchall()
        if not rows:
            break
        batch_ids = [submit(chunk, build_params) for chunk in chunks(rows)]
        for batch_id in batch_ids:
            wait_for(batch_id)
            print(batch_id, collect(batch_id))

chunks() sizes ledger rows here, which is fine for small items; for large ones, size by the serialized params. Rows that end as failed (validation errors, truncations, refusals) or hit the attempt cap go to a person. Everything else converges on its own.

The clocks: 24 hours, 29 days

24 hours from creation is the processing limit. You can't modify a batch once it's submitted. To change it, cancel it (status goes to canceling, and requests already in flight may still complete) and submit a new one. A canceled batch still ends with partial results for whatever was processed.
29 days from created_at (not ended_at) is how long results stay downloadable. After that the batch object is still visible, with archived_at set, but its results are gone. That's why collect() writes to your own storage instead of treating the API as the system of record. Once results are stored, you can delete the batch with client.messages.batches.delete(batch_id); an in-progress batch has to be canceled first.

One more note from the docs: because of high throughput and concurrent processing, a batch can go slightly over your workspace's configured spend limit, so don't treat that limit as an exact cap for batch jobs.

Where batch is the right tool

Offline evals. An eval suite is the textbook batch workload. Nobody is waiting on the answer, it reruns on every prompt or model change, and it shouldn't use up the rate limits your product needs. Score the results from the ledger after the batch ends. If the suite itself is the weak part, start with a proper offline eval harness before optimizing what it costs to run.

Classification and extraction backfills. A new field to fill on a million old records, a taxonomy applied to an archive. Short outputs over a big shared prefix: this is the case to measure before you move.

Re-processing after a prompt change. Because PROMPT_VERSION is part of the custom_id, version 4 of the prompt creates new IDs next to version 3's. Run the new version as a batch, then diff the two result sets by record_id to see exactly which answers changed before switching production over.

Very long generations. With the output-300k-2026-03-24 beta header, batch requests on Opus 5.5, Sonnet 5.5 and several earlier models can set max_tokens up to 300,000, against 128K on the synchronous API. It's batch-only, available on the Claude API and Claude Platform on AWS, and a single generation that long can take over an hour, so plan around the 24-hour window.

Server tools at scale. Web search, web fetch and code execution work inside batches, and with no open connection to keep alive, the batch loop runs more iterations per turn before returning pause_turn.

And where it isn't:

Anything a user is waiting for, or that needs streaming (stream: true is rejected).
Fast mode (speed is rejected; fast mode tunes synchronous latency).
Cache pre-warming (max_tokens: 0 is rejected inside a batch).
Zero-data-retention or HIPAA workloads. Batch processing stores inputs and outputs for up to 29 days and isn't eligible for either arrangement. Under HIPAA readiness the API rejects it with a 400. Under ZDR it isn't blocked, but using it takes that data outside your ZDR arrangement.
Work with a hard deadline shorter than a day. "Most batches finish within an hour" describes the typical case. It isn't a guarantee.

The checklist

[ ] Compute the break-even hit rate for your actual prefix, fresh-input and output sizes, including thinking tokens in output
[ ] Pilot a few thousand items and measure the hit rate from usage before moving the full job
[ ] Explicit breakpoint at the end of the shared prefix, 1-hour TTL, a primer batch first
[ ] Identical system, tools, effort, thinking configuration and output format on every request
[ ] Deterministic custom_id from record ID plus prompt version; a ledger keyed by it
[ ] Chunk by count and bytes; smaller batches if you want results sooner
[ ] One synchronous dry run of the request shape before submitting
[ ] Check stop_reason on every succeeded result
[ ] Retry server errors, expired and canceled; fix invalid_request_error; cap attempts
[ ] Persist results as you stream them; they stop being downloadable 29 days after creation

FAQ

Does the batch discount apply to cached tokens?
Yes. Caching multipliers stack with the Batch API discount, so cache reads and writes are half price too. What batch can't promise is the hit: cache hits inside a batch are best-effort.

Are partial results visible while a batch is running?
No. Results become available when every request has finished, or after 24 hours, and the per-type counts stay at zero until then. If you need earlier partial results, submit several smaller batches.

Do batch requests count against Messages API rate limits?
No. The Batches API has its own limits, on HTTP requests per minute and on requests waiting in the processing queue, and batch usage doesn't affect your Messages API limits.

Want to go deeper?

A 50% discount looks like an easy call until you put it next to the caching bill you already have. Then it's a calculation, and sometimes the answer is to stay synchronous. Working out cost per request and per customer before choosing an architecture is a recurring thread in Build and Ship a Production AI SaaS, which follows an LLM product from unit economics to paying users, and it pairs well with the caching and batching material in the LLM integration track linked at the top.

Published by Cursuri-AI.ro, Eastern Europe's AI education platform, with hands-on courses on LLM integration, evaluation and AI engineering.

Prices, limits and behavior were checked against Anthropic's documentation on 2 October 2026: batch processing, pricing, prompt caching, rate limits and API and data retention. Prices and limits change, so re-check them before you commit a budget.

Claude Opus 5.5 Migration: Four Errors You'll See and Three You Won't

galian — Wed, 23 Sep 2026 14:37:27 +0000

Anthropic released Claude Opus 5.5 on 22 September 2026. It costs less than Opus 5 ($4/$20 per million input/output tokens, down from $5/$25), cache reads dropped from $0.50 to $0.20, and in Anthropic's own benchmark table it beats Claude Fable 5.1 on agentic coding at 40% of Fable's list price. Most teams running Opus 5 will move, and most will start by changing one string:

model = "claude-opus-5-5"  # was "claude-opus-5"

That line is the whole migration only if your integration never disabled thinking, never forced a tool, never used the older computer use tool, and never edits its own history. Four request shapes that Opus 5 accepted now come back as a 400. Those are the easy ones: they fail loudly, in staging, with a readable message. The three changes that don't throw are the ones that reach production — a default that moved, an agent UI that goes quiet, and an unattended agent that stops halfway.

Below: each exact error message (so you can grep your logs for it), the fix, and the three silent changes with the code that catches them. It is the model-upgrade drill we run in the production LLM integration track at Cursuri-AI.ro, cut down to what this release actually changes.

What changed, in one table

	Claude Opus 5	Claude Opus 5.5
Model ID	`claude-opus-5`	`claude-opus-5-5`
Input / output (per MTok)	$5 / $25	$4 / $20
Cache read (per MTok)	$0.50	$0.20 (0.05× input)
5-minute cache write (per MTok)	$6.25	$5
Batch API (per MTok)	$2.50 / $12.50	$2 / $10
Fast mode (Claude API only, research preview)	$10 / $50	$8 / $40
Default `effort`	`high`	`medium`
`thinking: {"type": "disabled"}`	Accepted at `high` or below	400 at every level
`tool_choice` of type `any` / `tool`	Accepted	400
`computer_20251124` on Claude API / Google Cloud	Accepted with beta header	400 (toolset only)
Text between tool calls	`text` blocks	`thinking` blocks, empty by default

The context window (1M tokens) and max output (128K; up to 300K on the Batch API with the output-300k-2026-03-24 beta header) are unchanged. The knowledge cutoff is June 2026, and Anthropic commits to not retiring the model before 22 September 2027. The numbers come from the Opus 5.5 model page, what's new in Opus 5.5 and the fast mode docs.

The four errors you'll see

1. `"thinking.type.disabled" is not supported for this model`

"thinking.type.disabled" is not supported for this model. Use "thinking.type.adaptive" and "output_config.effort" to control thinking behavior.

On Opus 5, thinking was on by default, but you could switch it off at high effort or below — and latency-sensitive integrations often did. On Opus 5.5 adaptive thinking is always on, and disabled returns a 400 at every effort level. (Manual {"type": "enabled", "budget_tokens": N} budgets were already rejected on Opus 5, with the same message naming "thinking.type.enabled".)

The replacement for "off" is a lower effort:

resp = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16_000,
    output_config={"effort": "low"},   # was: thinking={"type": "disabled"}
    messages=[{"role": "user", "content": prompt}],
)

The error is the easy part. The follow-on bug is in the code that reads the response. With thinking disabled, resp.content[0] was always your text. Now a response can begin with one or more thinking blocks, and at the default display: "omitted" their text is an empty string, so they don't even look like content when you print them. resp.content[0].text starts raising AttributeError on some turns and not others, depending on whether the model decided to think. Select blocks by type, always:

def text_of(resp) -> str:
    return "".join(b.text for b in resp.content if b.type == "text")

Two more consequences of "always on". Thinking tokens are billed as output tokens even when their text isn't returned to you, so effort now moves your bill whether or not you ever look at the reasoning. And max_tokens covers thinking plus the reply: a limit sized for Opus 5 with thinking off can now end replies early with stop_reason: "max_tokens".

2. `tool_choice: type "tool" and "any" are not supported for this model`

tool_choice: type "tool" and "any" are not supported for this model.

Forced tool use is gone. {"type": "any"} and {"type": "tool", "name": ...} are rejected; auto (the default) and none still work. The same validation runs on the token-counting endpoint, so if a cost estimator counts tokens for the same request shape, it breaks too — in a different service, usually owned by a different person.

Teams forced tools for two different jobs, and each job has its own replacement.

Job one: "give me JSON." Forcing a single extract tool was the classic way to get schema-shaped output. That job now belongs to structured outputs, which are generally available and need no beta header:

import json

resp = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=4_096,
    output_config={
        "effort": "low",
        "format": {
            "type": "json_schema",
            "schema": {
                "type": "object",
                "properties": {
                    "vendor": {"type": "string"},
                    "total_cents": {"type": "integer"},
                    "due_date": {"type": "string", "format": "date"},
                },
                "required": ["vendor", "total_cents", "due_date"],
                "additionalProperties": False,
            },
        },
    },
    messages=[{"role": "user", "content": f"Extract the invoice fields:\n\n{invoice_text}"}],
)
invoice = json.loads(text_of(resp))

The JSON arrives in a text block, which is one more reason text_of() exists. Check the schema limitations before you port: numeric bounds (minimum, maximum), string lengths and recursive schemas aren't supported, so keep validating those in code.

Job two: "call this tool now." Where you forced a specific tool in an agent step, the migration guide's answer is to keep tool_choice on auto, add strict: true so every call matches the tool's schema, and say in the prompt when the tool applies:

resp = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16_000,
    tools=[{**tool, "strict": True} for tool in TOOLS],
    tool_choice={"type": "auto"},
    messages=[{"role": "user", "content": "Look up order 4417 with get_order before you answer."}],
)
called_tool = any(b.type == "tool_use" for b in resp.content)

What you lose is the guarantee. Log called_tool as a metric: a rising share of turns where the model answered in text instead of calling the tool is a prompt problem you want to see on a dashboard, not in a support ticket.

3. `'claude-opus-5-5' does not support tool types: computer_20251124.`

'claude-opus-5-5' does not support tool types: computer_20251124.

The message goes on to list the tool types the model does accept. On the Claude API and Google Cloud, Opus 5.5 takes computer use only as the computer_toolset_20260801 toolset. Amazon Bedrock still accepts the old tool, which is a trap of its own: the same code passes on Bedrock and fails on the Claude API.

The request change is small. Drop the beta header; the toolset entry takes no name and no display size:

# Before (Opus 5)
client.beta.messages.create(
    model="claude-opus-5",
    max_tokens=4096,
    betas=["computer-use-2025-11-24"],
    tools=[{"type": "computer_20251124", "name": "computer",
            "display_width_px": 1024, "display_height_px": 768}],
    messages=messages,
)

# After (Opus 5.5)
client.messages.create(
    model="claude-opus-5-5",
    max_tokens=4096,
    tools=[{"type": "computer_toolset_20260801"}],
    messages=messages,
)

The loop change is bigger. With the toolset, the action is the tool_use block's name, not input.action; one turn can carry several of those blocks; and every result has to echo toolset_name. A dispatcher written as match block.input["action"] needs a rewrite, not a rename. The computer use migration notes list every change.

4. Invalid `signature` in `thinking` block

messages.{i}.content.{j}: Invalid `signature` in `thinking` block. The block is bound to a different conversation. Remove the block, or set `thinking.block_binding.prefix_mismatch_behavior` to "drop_block".

Opus 5.5 inherits Fable 5.1's binding: a thinking block stays valid only while the system prompt, the tools and every message before it are unchanged. For accounts created on or after 31 August 2026, editing any of those and replaying the block returns this 400; older accounts get the check only when they opt in. Rebuilding the system prompt per request, trimming old turns on the client, or injecting a reminder and deleting it on the next call all trigger it, and retrying the same request body won't clear it.

The fix is structural: keep the history append-only, change instructions with mid-conversation system messages instead of edits, and leave trimming to server-side compaction or context editing. It is the same discipline as keeping agent context append-only for caching reasons, except that on Opus 5.5 it is also a correctness rule.

What is new with Opus 5.5 is which models can read whose reasoning, and that part never throws:

Conversation moves…	Reasoning carried over?
Opus 5 (or earlier Opus, Sonnet, Haiku) → Opus 5.5	Yes
Opus 5.5 → Fable 5.1 or Mythos 5.1 (Claude API)	Yes
Opus 5.5 → any other model (Sonnet 5, Haiku 4.5, Opus 5, Opus 4.8…)	No, blocks silently dropped
Fable or Mythos → Opus 5.5	No, blocks silently dropped

A block the target model can't read is removed before the model sees it: the request succeeds, and the dropped blocks aren't billed. With the thinking-binding-controls-2026-08-01 beta header, the drop is reported in a top-level input_transformations array. Turn that on in any service that routes between models, or you will never know it happened.

The design consequence for routers: escalate up, don't hand down. Start a hard task on Opus 5.5 and escalate to Fable 5.1 when it stalls, and the escalated model keeps everything Opus 5.5 reasoned so far. Route the other way — Fable plans, Opus executes in the same conversation — and every hand-off starts cold. The same applies to refusal fallbacks and to mid-conversation downgrades that save money. If your system is built on routing and escalation between agents, this table belongs in its design doc.

The three changes you won't see

1. The default effort dropped a level, and the benchmarks didn't

A request that omits effort ran at high on Opus 5. On Opus 5.5 it runs at medium. No error, no warning, different behavior.

Anthropic's position is that this is fine: in its testing, Opus 5.5 at medium matches or beats Opus 5 at high on coding and knowledge-work evaluations, and on several coding evaluations low comes close at much lower cost. That may well hold for your workload. But read the announcement's fine print before you rely on it:

The benchmark table is run at max effort: "unless otherwise noted, all Claude Opus 5.5 results use adaptive thinking at max effort."
The cost claim, 40% less than Opus 5 on typical workloads, is "at default settings", which means medium.

Both statements are true. They describe different configurations. The model that scores 66.4% on Terminal-Bench 4.0 and the model that costs 40% less are two settings of the same weights, and you choose one per request.

Two more effort facts change the arithmetic. At any given level, Opus 5.5 tends to think more per turn than Opus 5, most of all at xhigh and max, so carrying your Opus 5 setting over means longer turns and more output tokens, not the same bill. And level names don't map across models: Opus 5.5's medium is not Opus 5's medium.

So don't carry settings over. Re-run the sweep. Each case is a prompt plus a check function, the same shape as any eval:

import time
from anthropic import Anthropic

client = Anthropic()
USD_IN, USD_OUT = 4 / 1_000_000, 20 / 1_000_000   # claude-opus-5-5 list prices

def run_case(case: dict, effort: str) -> dict:
    started = time.monotonic()
    with client.messages.stream(               # stream long turns
        model="claude-opus-5-5",
        max_tokens=64_000,
        output_config={"effort": effort},
        messages=[{"role": "user", "content": case["prompt"]}],
    ) as stream:
        resp = stream.get_final_message()
    return {
        "passed": case["check"](text_of(resp)),
        "seconds": time.monotonic() - started,
        "usd": resp.usage.input_tokens * USD_IN + resp.usage.output_tokens * USD_OUT,
    }

def sweep(cases: list[dict], levels=("low", "medium", "high", "xhigh")) -> None:
    for effort in levels:
        rows = [run_case(case, effort) for case in cases]
        n = len(rows)
        print(f"{effort:>6}  pass {sum(r['passed'] for r in rows) / n:5.0%}  "
              f"avg {sum(r['seconds'] for r in rows) / n:6.1f}s  "
              f"avg ${sum(r['usd'] for r in rows) / n:.4f}")

Pick the lowest level that holds your pass rate, set it explicitly in code (never rely on a default that has changed once and can change again), and leave max_tokens room for thinking; Anthropic reports that 128,000, the model's maximum, worked well for long agentic coding turns. If one conversation needs different levels at different points, use a per-message effort change (beta header mid-conversation-output-config-2026-07-01), because changing the top-level value between requests invalidates the prompt cache. The sweep above is deliberately tiny. The version that earns its keep is a real eval harness with your own traffic as the dataset.

2. Your agent's progress messages went quiet

On Opus 5, the short notes the model writes between tool calls ("Found the failing test, checking the auth module next") came back as text blocks. Plenty of agent UIs stream exactly those as a status line.

On Opus 5.5, as on Fable 5.1, those notes arrive as progress-update thinking blocks, at most one before each tool call. At the default display: "omitted" their text is an empty string. Nothing fails. Your UI just shows a spinner for three minutes where it used to narrate.

To get the notes back, request display: "updates" (beta), which keeps reasoning hidden and returns progress updates as text, and render any non-empty thinking delta:

with client.beta.messages.stream(
    model="claude-opus-5-5",
    max_tokens=64_000,
    thinking={"type": "adaptive", "display": "updates"},
    betas=["thinking-display-updates-2026-08-18"],
    tools=TOOLS,
    messages=messages,
) as stream:
    for event in stream:
        if (event.type == "content_block_delta"
                and event.delta.type == "thinking_delta"
                and event.delta.thinking):         # non-empty text = a progress update
            ui.status(event.delta.thinking)
    final = stream.get_final_message()

messages.append({"role": "assistant", "content": final.content})  # unchanged, see error 4

Three details. Send the blocks back unchanged with the rest of the assistant turn. Expect fewer updates at higher effort and in long tool chains, and a pause of several seconds before each one. And when a response stops on max_tokens right after a tool call, the last progress block reads This part of the response was interrupted before it finished.: show it rather than swallow it. display: "summarized" also returns the updates, but mixed with reasoning summaries you can't tell apart.

3. Your unattended agent stops halfway

On long, multi-part tasks Opus 5.5 reports progress readily, and some of those reports end the turn with text instead of a tool call: stop_reason: "end_turn", a tidy summary, and "next, I'll migrate the remaining two endpoints." A loop that treats end_turn as "task complete" exits right there. In an interactive product that is a feature. In a nightly job it is a half-done migration that looks finished in the logs.

Anthropic's prompting guide for Opus 5.5 says to treat a text-only end of turn as a report, not as proof of completion: keep the task's parts in a checklist (a to-do tool, a file), and if items remain open with no blocker stated, nudge — two or three times at most:

MAX_NUDGES = 3

def run_unattended(task: str, checklist) -> list:
    messages = [{"role": "user", "content": task}]
    nudges = 0
    while True:
        with client.messages.stream(
            model="claude-opus-5-5",
            max_tokens=128_000,
            output_config={"effort": "medium"},
            tools=TOOLS,
            messages=messages,
        ) as stream:
            resp = stream.get_final_message()
        messages.append({"role": "assistant", "content": resp.content})

        if resp.stop_reason == "tool_use":
            messages.append({"role": "user", "content": run_tools(resp.content)})
            continue

        open_items = checklist.open_items()        # your state, not the model's summary
        if resp.stop_reason == "end_turn" and open_items and nudges < MAX_NUDGES:
            nudges += 1
            messages.append({"role": "user", "content": (
                f"Your task list still has open items: {'; '.join(open_items)}. "
                "Continue with them. If one is blocked, say what is blocking it."
            )})
            continue

        return messages   # finished, refused, out of tokens, or stuck: a human looks next

The checklist is the important part: completion comes from your state, not from the tone of the model's last message. And cap the nudges. An agent that is genuinely stuck should stop and be reviewed, not loop politely forever.

One more non-error: refusals have new categories

Opus 5.5 runs a biology classifier in addition to the cybersecurity one, plus a new reasoning_extraction category. A decline is an HTTP 200 with stop_reason: "refusal" and a stop_details.category, so code that only checks the status code logs it as a success with an empty answer. Two practical notes:

Prompts that tell the model to write out its reasoning in the response, a common workaround from the thinking-disabled days, can be declined as reasoning_extraction. Remove those instructions, and read display: "summarized" thinking if you need the reasoning.
Server-side fallback (fallbacks: "default", beta header server-side-fallback-2026-07-01) retries a declined request on a recommended model, except for reasoning_extraction, which comes straight back to you. And a fallback to another model runs without Opus 5.5's reasoning; see the table under error 4.

A refusal that arrives before any output isn't billed, but it still counts against your rate limits.

The money, honestly

At identical token counts the list-price drop is real. One agent turn with a 60K-token cached prefix, 3K fresh input tokens and 2K output tokens:

	Opus 5	Opus 5.5
60K cache read	$0.030	$0.012
3K input	$0.015	$0.012
2K output	$0.050	$0.040
Per turn	$0.095	$0.064 (−33%)

Token counts won't stay identical, though, and they move in opposite directions: more thinking per turn at a given effort pushes output up, while a lower default effort and (according to Anthropic and its early testers) fewer steps per task push it down. The honest answer to "will my bill drop?" is "re-baseline and see", which the sweep above already does. Two other levers: the Batch API halves everything ($2/$10) for work that can wait, and fast mode ($8/$40, up to 2.5× more output tokens per second, Claude API only, research preview) is for when latency is worth paying double. Requests at different speeds don't share cached prefixes, so don't flip fast mode on and off within a conversation.

Opus 5.5 or Fable 5.1?

Vendor-reported results, at max effort, from Anthropic's announcement:

Benchmark	Opus 5.5	Fable 5.1	Opus 5
Terminal-Bench 4.0	66.4%	55.8%	52.3%
CursorBench 4.0	57.8%	51.8%	46.6%
GDPval-AA v2.1 (Elo)	1846	1735	1708
OSWorld 2.0 (partial)	81.8%	80.7%	74.0%

The same table puts GPT-6 Astra ahead of Opus 5.5 on AutomationBench (41.4% vs 40.0%) and Terminal-Bench-Science (64.6% vs 58.7%), a useful reminder that "the leading model" depends on which row you read.

At $4/$20 against Fable 5.1's $10/$50, the default move for most workloads is Opus 5.5, with Fable 5.1 kept as the escalation tier rather than retired. Thanks to the compatibility table above, that direction — Opus 5.5 first, Fable 5.1 on escalation — is also the one that doesn't throw away reasoning. One caveat for cache-heavy traffic: Fable 5.1's cache read is $0.25 per MTok, close to Opus 5.5's $0.20, so for workloads dominated by cached context the saving comes mostly from output and fresh input, and it is smaller than the list prices suggest.

The migration checklist

[ ] Model ID → claude-opus-5-5 (on Bedrock: anthropic.claude-opus-5-5)
[ ] Remove thinking: {"type": "disabled"}; set output_config.effort explicitly
[ ] Read content by block type, never content[0].text
[ ] Raise max_tokens to leave room for thinking
[ ] Replace forced tool_choice: structured outputs for JSON jobs, auto + strict: true + a prompt instruction for tool jobs, including in token-counting calls
[ ] Computer use on the Claude API or Google Cloud → computer_toolset_20260801, and rewrite the dispatcher
[ ] No edits to system, tools or earlier turns mid-conversation; report input_transformations wherever you route between models
[ ] display: "updates" if your UI shows progress between tool calls
[ ] Unattended loops: completion comes from a checklist, with capped nudges on a text-only end_turn
[ ] Handle stop_reason: "refusal"; remove "write out your reasoning" instructions
[ ] Re-run the effort sweep and re-baseline cost and latency before moving production traffic

If your team works in Claude Code, the bundled Claude API skill automates the mechanical half: /claude-api migrate this project to claude-opus-5-5 swaps model IDs, fixes the breaking parameters, calibrates effort and hands back a list of things to verify by hand. It can't know that your UI depends on text between tool calls, or that your nightly job trusts end_turn; those are the items only your tests will catch. Structured Claude Code workflows — plan, migrate, verify against a test suite — are what turn that checklist into a one-afternoon job.

FAQ

Is Claude Opus 5.5 cheaper than Opus 5?
On list price, yes: $4/$20 against $5/$25 per million tokens, and cache reads fall from $0.50 to $0.20. Anthropic says typical workloads cost about 40% less at default settings, but the default effort is also lower, so compare the two at the effort level you will actually run.

Can I still turn thinking off on Opus 5.5?
No. Thinking is always on, and disabled returns a 400. Use effort: "low" for latency-sensitive paths, and keep display: "omitted" (the default) if you only want thinking text out of your responses.

Will conversations started on Opus 5 continue on Opus 5.5?
Yes. Opus 5.5 reads thinking blocks from Opus 5 and earlier Opus, Sonnet and Haiku models, so in-flight conversations keep their reasoning. The reverse doesn't hold: hand an Opus 5.5 conversation back to Opus 5 and its reasoning is dropped.

Does anything differ on Bedrock, Google Cloud or Microsoft Foundry?
The thinking, tool-choice and history changes apply on every platform. Computer use is the exception: the old computer_20251124 tool is rejected on the Claude API and Google Cloud but keeps working on Amazon Bedrock; for Foundry, check the compatibility table in the computer use docs. Fast mode exists only on the Claude API.

How do I get guaranteed JSON now that I can't force a tool?
Use structured outputs: output_config.format with a JSON schema. They are generally available and return the JSON in a text block.

The bottom line

On paper, Opus 5.5 is an easy upgrade: cheaper, faster and, by Anthropic's numbers, stronger on agentic work. The four 400s make sure you fix the obvious parts, so spend your testing time on the other three. Set effort explicitly and sweep it again, render progress from thinking blocks, and take task completion from your own state rather than from the model's last sentence. None of those three will ever show up as an error in your logs.

I build and teach production AI systems at Cursuri-AI.ro, Eastern Europe's AI education platform — hands-on courses on LLM integration, agent architecture, evaluation, and shipping model upgrades without surprises.

The Model in Your Config Has an Expiry Date — a Runbook for LLM Deprecations

galian — Mon, 31 Aug 2026 22:05:44 +0000

There is a category of production incident that has no bug, no bad deploy, and no root cause inside your codebase. Your code is byte-identical to the version that worked yesterday. Your tests pass. Your dependencies are pinned. And a feature is returning 404s.

The model string in your config refers to something that no longer exists.

This is the strangest failure mode in the whole LLM stack, because it is the only one that arrives entirely on somebody else's schedule — and the only one you were told about, in writing, months in advance, in an email nobody on the on-call rotation read.

I teach AI engineering at Cursuri-AI.ro, and model retirement is the incident I see teams handle worst, not because it's hard, but because it doesn't feel like engineering work until the day it does. Here's the whole thing: what actually breaks, what the provider tells you and when, and the runbook that turns a surprise outage into a boring Tuesday ticket.

The vocabulary you're missing two-thirds of

Most teams have two mental states for a model: "it works" and "it's old." Anthropic's model deprecations page defines four, and the gap between them is where the incident lives:

Active — fully supported, recommended.
Legacy — no longer receiving updates, may be deprecated in future.
Deprecated — still functional, no longer recommended, has an assigned retirement date.
Retired — gone. Requests fail.

The important one is deprecated. A deprecated model works perfectly. Latency is normal, quality is normal, your dashboards are green. Nothing in the response payload tells you a clock is running. The only signal is a documentation page and an email to the account owner — who is usually in finance.

And "retired" means what it says. The model ID stops resolving, and the API answers the way it answers any unknown model ID: HTTP 404, not_found_error, whose documented cause is "invalid endpoint or model ID." Your retry logic will not help you, because 404 is not retryable. If your error handling catches one broad exception class and retries everything, you'll spend the outage generating three times the traffic and no useful log lines.

import anthropic

client = anthropic.Anthropic()

try:
    response = client.messages.create(
        model=settings.CHAT_MODEL,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
except anthropic.NotFoundError:
    # This is not a blip. This is a retired or misspelled model ID.
    # Page someone. Do not retry.
    raise
except anthropic.RateLimitError:
    ...
except anthropic.APIConnectionError:
    ...

That first except block is maybe ninety seconds of work, and it's the difference between an alert that says "the model is gone" and an alert that says "elevated error rate, service degraded."

The dates are published before the deprecation, and almost nobody reads them

Here is the part that genuinely surprises people, and it's the single most useful fact in this article.

Anthropic publishes tentative retirement dates for models that are currently active. Not deprecated — active. As of the end of August 2026, the model status table reads, in part:

Model	State	Earliest retirement
`claude-sonnet-4-5-20250929`	Active	Not sooner than September 29, 2026
`claude-haiku-4-5-20251001`	Active	Not sooner than October 15, 2026
`claude-opus-4-5-20251101`	Active	Not sooner than November 24, 2026
`claude-opus-4-6`	Active	Not sooner than February 5, 2027
`claude-opus-5`	Active	Not sooner than July 24, 2027
`claude-fable-5`	Active	Not sooner than June 9, 2027

Read the first row again. If you are running claude-sonnet-4-5-20250929 in production today, its published floor is roughly four weeks out. That is not a retirement announcement — "not sooner than" is a floor, not a date, and no deprecation has been announced for it. But it is a planning input you already have, for free, and it costs you nothing to put in a calendar.

The counterpart is the models that already went: claude-opus-4-1-20250805 was deprecated on June 5, 2026 and retired on August 5, 2026 — three weeks before this article. claude-opus-4-20250514 and claude-sonnet-4-20250514 were deprecated April 14, 2026 and retired June 15, 2026.

Do the arithmetic on those pairs: 61 days, and 62 days. Anthropic commits to "at least 60 days' notice before model retirement for publicly released models," and in practice the notice window is exactly that. Sixty days is enough time to run a migration. It is not enough time to build an eval suite, get budget approval, and negotiate with a team that owns the prompt. That work has to already exist when the email lands.

Aliases are a strategy, not a shield

Anthropic exposes two shapes of model ID: aliases like claude-opus-5 or claude-sonnet-4-6, and dated snapshots like claude-sonnet-4-5-20250929. Note that the current generation are complete as aliases — claude-opus-5 is the whole ID, and appending a date suffix to it produces a 404 as surely as a retired model does.

Neither shape saves you:

A dated snapshot is reproducible and will never change behavior under you — and it is precisely what appears in the retirement table, by name, with a date.
An alias may keep resolving across point releases, but the alias is scoped to a model family. When the family goes, the alias goes. And an alias that silently starts pointing somewhere new is its own kind of incident: your outputs change, your evals shift, and nothing in your diff explains it.

The real answer isn't picking one. It's picking one per route, deliberately, and writing down why. A billing-adjacent classifier that must produce identical output for audit reasons wants a snapshot and a calendar entry. A chat surface where quality improvements are welcome wants an alias and an eval suite that runs on a schedule. What you must not have is thirty model strings scattered across a codebase, each chosen by whoever wrote that file, with no owner.

It is never just the model string

This is the trap that turns a one-line change into a two-week migration, and it's why "we'll swap it when we have to" fails.

Request parameters retire too. On Claude Opus 4.7 and later, temperature, top_p and top_k return a 400 error when set to a non-default value — the recommended replacement is to omit them and steer behavior through prompting. And it's not only the API: the Python SDK v1.0 and later removes those parameters from the request types, so passing them raises a TypeError before a request is ever made.

The same applies across the current generation: the fixed thinking budget (thinking: {type: "enabled", budget_tokens: N}) is gone in favor of adaptive thinking plus an effort level, and assistant-message prefill — the classic trick for forcing a response format — returns a 400 on the whole 4.6-and-later family. If your "quick model swap" touches code written against a 2025-era API, you are not changing a string. You are rewriting the request.

# Written for an older model. Every line here is now a 400 or a TypeError.
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",   # retired October 28, 2025
    max_tokens=4096,
    temperature=0.2,                       # 400 on Opus 4.7+ (non-default)
    thinking={"type": "enabled", "budget_tokens": 8000},   # removed
    messages=[
        {"role": "user", "content": prompt},
        {"role": "assistant", "content": "{"},             # prefill: 400
    ],
)

# The current shape.
response = client.messages.create(
    model="claude-opus-5",
    max_tokens=16000,
    thinking={"type": "adaptive"},
    output_config={
        "effort": "high",
        "format": {"type": "json_schema", "schema": INVOICE_SCHEMA},
    },
    messages=[{"role": "user", "content": prompt}],
)

And prompts are part of the migration. This is the least mechanical and most expensive piece. Prompts tuned against an older model are frequently over-prescriptive for a newer one — step-by-step scaffolding that raised quality two generations ago can now suppress it. A migration that swaps the ID, passes your smoke test, and ships is a migration that quietly degraded output quality in a way you'll attribute to something else three weeks later. This is exactly the failure mode that makes an evaluation suite the load-bearing part of the runbook rather than the optional part.

The runbook

Five steps. The first three are cheap and you should do them this week, whether or not anything of yours is deprecated.

1. Inventory: find every model string you own

You cannot plan a migration you can't enumerate. Two passes:

# Every model ID that lives in code, config, notebooks, IaC, or a CI secret file.
rg -n --hidden -g '!node_modules' -g '!.git' \
   'claude-(fable|mythos|opus|sonnet|haiku)-[0-9a-z.\-]*|claude-[0-9]' .

Then the pass that matters more, because it catches the jobs nobody remembers: in the Claude Console, open Usage → Export, and read the CSV. It breaks usage down by API key and by model, which means it shows you the nightly batch job in a repo you don't own, running under a key that was issued in 2025. That report is the actual inventory. The grep is just the part you control.

The output of this step is a single table — route, model ID, owner, alias-or-snapshot, why. Keep it in the repo, not in a wiki nobody opens.

2. Centralize the strings

Every model ID resolves through one module. Not because indirection is beautiful, but because a migration should be one reviewable diff, and because it gives you a place to hang the metadata:

# models.py — the only file in the repo that contains a model ID.
from dataclasses import dataclass

@dataclass(frozen=True)
class ModelChoice:
    id: str
    owner: str          # who approves a change
    pinned_reason: str  # why this shape, not the other one
    review_by: str      # ISO date, from the published retirement floor

CHAT       = ModelChoice("claude-opus-5", "platform", "quality-led surface, alias by choice", "2027-07-24")
CLASSIFIER = ModelChoice("claude-haiku-4-5", "platform", "high volume, latency-sensitive", "2026-10-15")

review_by is not a guess. It is the published "not sooner than" date, copied in. Now your calendar and your code agree.

3. A canary that fails loudly and cheaply

A retirement should be caught by a scheduled job, not by a user. This is about eight lines and costs a fraction of a cent per run:

# canary.py — run daily in CI. Alerts on the one error that is never transient.
import anthropic, sys
from models import CHAT, CLASSIFIER

client = anthropic.Anthropic()
dead = []

for choice in (CHAT, CLASSIFIER):
    try:
        client.messages.create(
            model=choice.id,
            max_tokens=1,
            messages=[{"role": "user", "content": "ping"}],
        )
    except anthropic.NotFoundError:
        dead.append(choice)

if dead:
    for choice in dead:
        print(f"MODEL GONE: {choice.id} (owner: {choice.owner})", file=sys.stderr)
    sys.exit(1)

For the inventory half, client.models.list() gives you what your key can currently reach, with id, display_name, max_input_tokens, max_tokens and a capabilities tree — useful for catching the day a model stops appearing, and for asserting that a replacement actually supports the features your route depends on before you switch to it.

4. The eval gate, built before you need it

Sixty days is plenty of time to change a string and not nearly enough to answer "is the new one better on our task?" if you're starting from zero. The gate needs three things and they take an afternoon each:

A frozen set of real inputs. Fifty to two hundred requests sampled from production, with the messy ones deliberately over-represented. Not synthetic examples.
A grader that isn't a vibe. Exact match where you can get it, schema validation where the output is structured, an LLM judge with a written rubric where you can't. The judge is fine — an unwritten rubric is not.
A cost and latency baseline. Because the honest outcome of a migration is sometimes "quality held, p95 doubled," and you want to know that before your users do.

Run it against the old model and the new one on the same inputs, same day. The comparison is the deliverable, and it's the artifact that lets you say "yes, ship it" in an afternoon instead of a fortnight. If you're standing this up for the first time, this is the same machinery you need for every other change you'll ever make to an LLM feature — building it properly once pays for itself on the second migration.

5. Migrate the request, then the prompt, then the model

Order matters, because it isolates variables:

Modernize the request shape on the model you're already running — remove sampling parameters, replace fixed thinking budgets with adaptive thinking plus an effort level, replace prefill with structured outputs. Ship it. Nothing about behavior should change, and if something does, you've learned it in isolation.
Swap the model ID behind a flag, run the eval gate, compare.
Re-tune the prompt for the new model — usually by deleting scaffolding rather than adding it — and re-run the gate.
Roll out by percentage, keeping the old model reachable until its retirement date, not until your rollout finishes.

Doing all four at once produces a result you can't attribute. Doing them in order takes the same total effort and tells you which change did what.

What the provider actually owes you, and what it doesn't

Worth being clear-eyed here, because the answer is better than the ecosystem average and still not a guarantee.

You get: at least 60 days' notice before retirement of a publicly released model, email to affected accounts with active deployments, published tentative floors for active models, a recommended replacement for every deprecated model, and a self-service usage audit. Anthropic has also published commitments on model deprecation and preservation, including long-term preservation of model weights, and states openly that retirement exists to free capacity — with real downsides for people who depended on a specific model's behavior.

You don't get: a promise that the replacement behaves like the old one on your task. That has never been promised by anyone, and it's the only part that requires actual work from you.

One more thing worth knowing if you're multi-cloud: those dates apply to Anthropic-operated platforms — the Claude API, Claude Platform on AWS, and Microsoft Foundry. Amazon Bedrock and Google Cloud set their own retirement schedules, so a model's status and dates can differ there. If your failover path routes to a different platform, that path has its own calendar, and it is not this one.

The checklist

Print this. It's the whole article.

[ ] Every model ID in the codebase resolves through one module.
[ ] That module records, per route: owner, alias-or-snapshot, and the reason.
[ ] The published "not sooner than" date for each model is in the code and in a shared calendar.
[ ] NotFoundError / HTTP 404 is caught separately and pages a human. It is never retried.
[ ] A daily canary calls every configured model with max_tokens=1 and fails the build when one is gone.
[ ] The Console usage export has been read at least once this quarter, by a human, looking for keys and models nobody claims.
[ ] An eval suite with real inputs and a written rubric exists today, not on announcement day.
[ ] The request shape is modernized independently of the model swap.
[ ] Prompt re-tuning is scheduled as part of every migration, not treated as optional.
[ ] Anyone routing through Bedrock or Vertex has checked those platforms' separate schedules.

None of this is clever. All of it is the difference between a migration you schedule and an outage you explain.

I build and teach this stack at Cursuri-AI.ro — including taking LLM features to production and shipping a full AI product end to end. If your team has a model string you can't account for, the inventory step is genuinely the highest-value hour you'll spend this month.

Your LLM Stream Works on localhost and Dies in Production — Fixing SSE End to End

galian — Sat, 22 Aug 2026 11:39:18 +0000

Streaming an LLM response looks like the easiest feature in your product. The SDK gives you an iterator, you print tokens, it works on your laptop in under a minute.

Then you deploy it, and one of these happens: the response arrives all at once after 40 seconds instead of token by token. Or the connection drops mid-answer with no error anywhere in your logs. Or a user closes the tab and you keep paying for 60,000 tokens nobody will ever read.

None of those are model problems. They're the six things that break in the space between api.anthropic.com and a browser tab — a proxy, a load balancer, an HTTP client, and a JavaScript API that each have opinions about long-lived responses.

I teach AI engineering at Cursuri-AI.ro, an AI education platform in Eastern Europe, and this is the failure surface I see most often in production reviews — because every layer of it looks fine in isolation. Here's the whole path, one break at a time.

First: streaming is not a UX nicety anymore

It's worth being clear about why you're doing this, because it changes how you treat failures.

The perceived-latency argument is the famous one, and it's real. But on current models, streaming is also a correctness requirement for a growing share of requests. Claude Opus 5, Sonnet 5, and the 4.6/4.7/4.8 family support up to 128K output tokens, and Anthropic's SDKs will refuse a non-streaming request they estimate will exceed the connection's tolerance — the Python SDK raises a ValueError rather than let you build something that hangs and drops. The default client timeout is 10 minutes (note the units differ by SDK: seconds in Python and Ruby, milliseconds in TypeScript), and with thinking on by default on Opus 5, a hard task can spend minutes generating before the first visible character.

So: any request with a large max_tokens, a long input, or a reasoning-heavy prompt is a streaming request. Not for the animation — for the connection.

import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model="claude-opus-5",
    max_tokens=64000,
    thinking={"type": "adaptive", "display": "summarized"},
    messages=[{"role": "user", "content": "Analyze this incident report..."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

    final = stream.get_final_message()

One thing in that snippet is not cosmetic. On Opus 5, Opus 4.8/4.7, Fable 5, and Sonnet 5, the thinking display default is "omitted" — the thinking block opens, emits a single signature_delta, and closes, with no thinking_delta events. If you stream reasoning to users and leave the default in place, your UI shows a long dead pause and then a wall of text. display: "summarized" is what gives you something to render during the think. It costs nothing extra: thinking happens and is billed identically under every display setting.

Break #1: your proxy is holding the tokens hostage

This is the number-one "streaming doesn't work in production" bug, and the tell is unmistakable: locally you see tokens appear one by one; deployed, the whole response lands at once at the end.

Nothing in your app is wrong. nginx is buffering. From the nginx documentation, proxy_buffering defaults to on, and when buffering is enabled nginx reads the response from the upstream into its own buffers before passing it along. When it's off, "the response is passed to a client synchronously, immediately as it is received."

There are two ways to fix it, and you want the second one.

location /api/chat/stream {
    proxy_pass http://app;
    proxy_buffering off;
    proxy_read_timeout 300s;   # default is 60s
}

That works, but it puts a per-route infrastructure rule in a file your application team doesn't own. nginx also honors a response header: "Buffering can also be enabled or disabled by passing yes or no in the X-Accel-Buffering response header field." So the streaming endpoint can turn buffering off for itself:

from fastapi.responses import StreamingResponse

def sse_response(generator):
    return StreamingResponse(
        generator,
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",   # the important one
        },
    )

The header travels with the endpoint. Add a route, get correct behavior. No config drift, no "works in staging" incident six months later when someone rebuilds the ingress.

While you're there, note proxy_read_timeout — it defaults to 60s in nginx. A model thinking hard for 90 seconds before its first token will be cut off by your own proxy, and the error you'll see is a generic upstream timeout with no mention of streaming at all.

Break #2: the load balancer kills you during the pause

Same class of bug, one layer out, and it bites hardest exactly on the requests you care most about.

An AWS Application Load Balancer's idle_timeout.timeout_seconds attribute has a valid range of 1–4000 seconds and a default of 60 (see LoadBalancerAttribute in the ELB API reference). "Idle" means no bytes in either direction. An LLM that thinks for 75 seconds before emitting its first token produces exactly that: a silent, live, perfectly healthy TCP connection that your load balancer decides is dead.

You can raise the timeout, and you probably should. But raising it alone is a fragile fix, because it only pushes the cliff further out. The robust fix is to make sure the connection is never actually idle — which SSE has a purpose-built mechanism for.

The SSE wire format treats a line beginning with a colon as a comment. MDN puts it plainly: "A colon as the first character of a line is in essence a comment, and is ignored," and "The comment line can be used to prevent connections from timing out; a server can send a comment periodically to keep the connection alive."

import asyncio

async def sse_stream(request, params):
    queue: asyncio.Queue = asyncio.Queue()

    async def heartbeat():
        while True:
            await asyncio.sleep(15)
            await queue.put(": keepalive\n\n")   # ignored by every SSE client

    # ... producer task pushes real events onto the same queue ...

Fifteen seconds is a good default — comfortably under a 60-second idle timeout, cheap enough that nobody notices. The bytes are discarded by the client and keep every hop on the path convinced the connection is alive.

Anthropic does the same thing on its side, by the way: "Event streams may also include any number of ping events." Your parser needs to expect them and ignore them, which leads directly to the next break.

Break #3: the errors arrive with HTTP 200

Here's the part that catches teams who have otherwise done everything right.

Once the stream has started, the HTTP status code is already sent. It's 200. It will stay 200 no matter what happens next. Failures after that point arrive as events inside the body, and if your parser only handles the happy path, they vanish silently.

The Messages API documents this directly: "The API may occasionally send errors in the event stream. For example, during periods of high usage, you may receive an overloaded_error, which would normally correspond to an HTTP 529 in a non-streaming context":

event: error
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}

If you're using the SDK helpers this is handled for you. If you're parsing raw SSE — which you are, at the browser end, and often at the server end too — you need to branch on it explicitly.

Two more things belong in the same parser:

Unknown event types must not throw. The docs are explicit: "In accordance with the versioning policy, new event types may be added, and your code should handle unknown event types gracefully." A parser with an else: raise in it is a scheduled outage with an unknown date. Log and skip.

stop_reason is not always end_turn. On Opus 4.7 and later, safety classifiers can decline a request with HTTP 200 and stop_reason: "refusal", with a category in stop_details. Check stop_reason before you read content. (stop_details is populated only for refusals and is null for every other stop reason, so guard before reading it.) On Opus 5 and Fable 5 you can also opt into server-side fallbacks — betas: ["server-side-fallback-2026-07-01"] with fallbacks: "default" — which routes by refusal category. If you do, expect a fallback content block in the stream at each model boundary: a content_block_start / content_block_stop pair with no deltas between them. A parser that assumes every block has deltas will choke on it.

Here's the shape of a parser that survives all of it:

for event in raw_events:
    t = event.get("type")

    if t == "content_block_delta":
        d = event["delta"]
        if d["type"] == "text_delta":
            yield sse("token", {"text": d["text"]})
        elif d["type"] == "thinking_delta":
            yield sse("thinking", {"text": d["thinking"]})
        # input_json_delta / signature_delta: accumulate, don't render

    elif t == "message_delta":
        # NOTE: usage counts in message_delta are CUMULATIVE, not incremental
        usage = event.get("usage") or {}
        output_tokens = usage.get("output_tokens", output_tokens)
        stop_reason = event["delta"].get("stop_reason", stop_reason)

    elif t == "error":
        yield sse("error", {"code": event["error"]["type"]})
        return

    elif t in ("ping", "content_block_start", "content_block_stop", "message_start", "message_stop"):
        pass

    else:
        log.info("unknown stream event, ignoring", extra={"event_type": t})

That comment about cumulative usage is worth internalizing — the docs flag it with a warning box. Adding up message_delta usage across events gives you a token count that grows quadratically and a cost dashboard that is confidently wrong.

Break #4: the user left and you're still paying

A user asks a question, gets three sentences in, sees the answer isn't what they wanted, and closes the tab.

What happens to the 60,000-token generation you started? In a lot of production apps: it runs to completion, bills in full, and writes its result to a database row nobody will read. Multiply by your bounce rate.

Cancellation has to be propagated explicitly, at every hop. Server side, Starlette (and therefore FastAPI) exposes the disconnect signal:

async def generate(request, params):
    with client.messages.stream(**params) as stream:
        for text in stream.text_stream:
            if await request.is_disconnected():
                break          # exiting the with-block closes the upstream connection
            yield sse("token", {"text": text})

Exiting the context manager is what actually matters — it closes the HTTP connection to Anthropic, and generation stops. A break without the context manager, or a background task that owns the stream and outlives the request, keeps burning tokens.

Browser side, fetch cancellation goes through AbortController:

const controller = new AbortController();
stopButton.onclick = () => controller.abort();

const res = await fetch("/api/chat/stream", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ prompt }),
  signal: controller.signal,
});

Aborting closes the TCP connection, which is what your is_disconnected() check observes. The chain only works if every link is present.

Break #5: `EventSource` can't do what your chat app needs

The browser's native SSE client is EventSource, and reaching for it is the obvious move. It's also, for most LLM chat UIs, the wrong one.

The constructor takes a URL and an options object whose only meaningful member is withCredentials. There is no place to put an HTTP method, a request body, or headers. That rules out sending a prompt as a POST body and rules out an Authorization header. And its built-in auto-reconnect — "By default, if the connection between the client and server closes, the connection is restarted" — is actively hostile for LLM generation: a dropped connection silently fires a brand-new generation, at full cost, with no memory of the tokens already delivered. One flaky connection becomes a duplicate bill.

Use fetch with a ReadableStream reader instead. You get POST, headers, AbortController, and — crucially — no reconnect you didn't ask for:

const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buf = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += value;

  const frames = buf.split("\n\n");
  buf = frames.pop();                       // keep the incomplete tail

  for (const frame of frames) {
    const dataLines = frame
      .split("\n")
      .filter((l) => l.startsWith("data:"))  // ':' comment lines fall out here
      .map((l) => l.slice(5).trimStart());
    if (!dataLines.length) continue;         // heartbeat frame
    handle(JSON.parse(dataLines.join("\n")));
  }
}

Two details that cause real bugs. First, TCP does not deliver your frames whole — reader.read() will hand you half an event, and the buf.split("\n\n") / buf.pop() pattern is what keeps you from parsing a truncated JSON object. Second, multiple consecutive data: lines in one frame are concatenated with newlines between them, per the spec; joining them with "" corrupts any payload containing a newline.

Break #6: partial output is a state you have to design for

Every stream can end three ways: complete, cancelled, failed mid-flight. Most codebases only persist the first.

That produces two ugly symptoms. A user refreshes after a network blip and their half-written answer is simply gone — the connection is stateless, so the tokens are too. And your cost tracking under-reports, because you only record usage on clean completions, while every abandoned generation was billed in full.

The fix is not resumable streams — that's a hard, mostly unnecessary feature. It's persisting the assistant message as it's produced, with a status:

Insert the message row with status = "streaming" before the first token.
Flush accumulated text to it periodically (every ~50 tokens or every second — not every token, unless you enjoy write amplification).
On message_stop, set status = "complete" and write the final usage from get_final_message().
On disconnect or error, set status = "partial" or "failed" and record whatever usage you observed.

Now a refresh renders the partial answer with an honest "generation was interrupted" marker, and your cost dashboard counts abandoned generations — which is the number that tells you whether Break #4 is costing you real money. Wiring persistence, usage accounting, and streaming state together is exactly the kind of plumbing we build end to end in our course on shipping a production AI SaaS, because it's where "demo works" and "product works" actually diverge.

The checklist

Everything above, compressed into things you can go verify this afternoon:

#	Check	Where
1	`proxy_buffering off` or `X-Accel-Buffering: no` on streaming routes	nginx / app
2	Proxy read timeout and LB idle timeout raised above your p99 time-to-first-token	infra
3	Heartbeat comment frames every ~15s	app
4	Parser handles `ping`, `error`, and unknown event types without throwing	app + client
5	`stop_reason` checked before reading `content`; `stop_details` guarded	app
6	`message_delta` usage treated as cumulative, not incremental	app
7	`thinking.display: "summarized"` if you render reasoning	app
8	Client disconnect propagated to the upstream stream (context manager exits)	app
9	`AbortController` wired to a visible stop control	client
10	Partial messages persisted with a status; abandoned usage recorded	app

If you want to know whether any of this is actually costing you, the honest test is a load test with a 30% mid-stream abandonment rate, run against staging, with token usage measured at both ends. The gap between what your dashboard reports and what your Anthropic console reports is the size of the problem.

Streaming is where a lot of AI products quietly lose both money and trust — and it's a plumbing problem, not a prompting one. If you want the full path from a first API call to an application that holds up under real traffic, that's the arc of our Advanced LLM Integration in Production course; the failure-mode discipline behind it — knowing whether a change actually helped — comes from LLM Evaluation and Testing. And if your next feature is voice, every one of these six breaks gets harder, which is its own topic in Voice AI and Realtime Multimodal Agents.

Sources: Streaming Messages — Claude API docs · Using server-sent events — MDN · ngx_http_proxy_module — nginx · LoadBalancerAttribute — AWS ELB API Reference

Anthropic Filed to Go Public. Here's What Actually Changes for Developers Building on Claude

galian — Fri, 14 Aug 2026 11:40:32 +0000

On June 1, 2026, Anthropic PBC confidentially submitted a draft registration statement on Form S-1 to the SEC for a proposed IPO of its common stock. That's the whole confirmed story. The number of shares hasn't been set, the price hasn't been set, no exchange has been named, no ticker has been named, and the announcement says plainly that any offering "will depend on market conditions and other factors."

Everything else you've read this summer — October listing, Nasdaq, $60B raise, which banks are on the cover — is press speculation dressed up in specific numbers. Some of it will turn out right. None of it is confirmed by the company, and none of it should be in your planning documents yet.

I teach AI engineering at Cursuri-AI.ro, an AI education platform in Eastern Europe, and I don't have a stock tip for you. What I do have is the version of this story that matters if you have production traffic going to api.anthropic.com: an IPO changes the incentives of the company on the other end of your API key, and there are four consequences worth engineering around before the S-1 goes public.

First, the numbers that are actually confirmed

Three figures anchor everything else, and all three come from Anthropic or from reporting on its own disclosures:

Metric	Figure
Run-rate revenue	Surpassed $47B, up from ~$9B at the end of 2025
Series H	$65B raised at a $965B post-money valuation
S-1 status	Confidential draft submitted June 1, 2026 — SEC review ongoing

A roughly 5x run-rate increase in under two quarters is the part worth sitting with. That is not a company that needs to go public to survive. It's a company converting enterprise API and coding-agent demand into revenue faster than almost any software business in history, and choosing to put that on a public balance sheet.

It also isn't happening in isolation. SpaceX — which now owns xAI outright and rebranded the combined entity SpaceXAI — listed on Nasdaq under SPCX on June 12, 2026, closing its first day at $160.95, up 19%. Frontier AI is moving from "priced by a handful of private rounds" to "priced by the public market every trading day." That shift is the actual subject of this post.

Consequence #1: your vendor gets a quarterly clock

Private Anthropic optimized on a multi-year horizon and answered to a small set of investors who had already agreed to the thesis. Public Anthropic answers, every ninety days, to a market that will form opinions about gross margin, revenue concentration, and compute spend.

I want to be careful here, because the lazy version of this argument is "public companies get greedy and raise prices." That's not a forecast I can support, and the recent evidence points the other way: Claude Opus 5 shipped in July 2026 at the same $5/$25 per million tokens that Opus 4.8 cost, and Sonnet 5 sits at $2/$10. Inference prices per unit of capability have been falling, not rising.

The realistic pressure isn't the sticker price. It's everything around it:

Free and low-tier generosity is the easiest margin lever to pull, and the least visible in a headline.
Rate limits and capacity allocation start to follow revenue, not goodwill. Enterprise contracts get the GPUs.
Deprecation timelines tighten, because retiring an old model is pure margin.

None of those show up as a price increase. All of them show up in your error rate.

Consequence #2: model retirement becomes an operational risk, not a footnote

This is the one I'd act on this week, and I say that as someone who has eaten the failure.

We had an integration pinned to a hardcoded Anthropic model ID. That model was retired. The result wasn't a graceful warning in a dashboard — it was a live 404 in production, in a customer-facing feature, discovered by users. The lesson generalizes past any one vendor: every hardcoded model ID is a scheduled outage with an unknown date.

Every frontier lab now runs a formal retirement process — xAI's docs currently carry a dedicated "Model Retirement" migration page, and Anthropic publishes deprecation notices. A public company has a stronger incentive to run that process on a tighter schedule. Your codebase should treat model identity as configuration, not as a string literal:

# config, not code — one place to change, one place to audit
MODELS = {
    "default":  os.getenv("LLM_DEFAULT",  "claude-sonnet-5"),
    "hard":     os.getenv("LLM_HARD",     "claude-opus-5"),
    "cheap":    os.getenv("LLM_CHEAP",    "claude-haiku-4-5-20251001"),
}

def complete(task: str, **kwargs):
    model = MODELS[TASK_TIER.get(task, "default")]
    return client.messages.create(model=model, **kwargs)

The test for whether you're exposed is simple: can you change which model serves a workload without a deploy? If the answer is no, that's the highest-leverage refactor on your list, and it's worth doing before you need it at 2am. Getting from "API calls sprinkled across services" to "an application where a model swap is a config change" is the arc we walk through in our course on building AI applications with the Python SDK.

Consequence #3: the S-1 will be the best vendor due-diligence document ever published

This is the genuinely good news, and almost nobody is talking about it.

Frontier labs are financial black boxes. We infer their economics from leaks, from investor decks that reach journalists, and from third-party trackers that disagree with each other by tens of billions. A public S-1 ends that for one of them. When Anthropic's full prospectus lands, you will be able to read, under penalty of securities law:

Gross margin on inference. The single number that tells you how much room exists under current API prices.
Revenue concentration. How much of that $47B comes from the top handful of customers. If one hyperscaler or one coding-agent partner is a huge slice, that's a dependency in your supply chain too.
Compute commitments. Multi-year obligations to chip and cloud suppliers — the fixed costs that determine how price-flexible the company can be in a downturn.
The risk factors section. Labs are required to enumerate, in writing, what could go wrong: litigation, regulation, model liability, safety incidents, key-personnel loss. It's the most candid document a lab will ever publish about itself.

If you're the person at your company who signs off on an AI vendor, put a reminder in your calendar for the day the public S-1 drops. Read the risk factors and the concentration disclosures first. That's a better afternoon of vendor research than any analyst report you'll pay for.

Consequence #4: "which model" stops being a taste question

A public market prices frontier labs against each other continuously, and that competition lands in your API bill and your eval scores. The current spread is already wide enough that model choice is a real engineering decision rather than a preference:

Model	Input / output per 1M tokens
Claude Fable 5	$10 / $50
Claude Opus 5	$5 / $25
Claude Sonnet 5	$2 / $10
Grok 4.6	$2 / $6

An 8x spread on output tokens between the top and bottom of that table means the same agent architecture can cost wildly different amounts depending on routing. And routing decisions made on vibes are how teams end up paying flagship prices for classification work.

The prerequisite for routing well is an eval harness that can answer "did quality hold when I moved this workload down a tier?" with data instead of a hunch. Without it, every model launch is a risk you absorb. With it, every launch is a shopping opportunity. That harness is the core of our LLM evals in production course, and the comparative side — what each frontier family is actually good at — is what we break down in our AI model comparison course.

What I'd actually do this quarter

Five items, in the order I'd tackle them:

Grep for hardcoded model IDs. Move every one to configuration. This is a half-day of work that prevents a class of outage.
Build or fix the eval gate. You cannot safely change models, effort levels, or providers without one. Everything else on this list depends on it.
Instrument cost per task finished — not cost per token. Token price is a distraction; what you care about is what it costs to complete one unit of real work, including retries and reasoning tokens.
Read your contract's deprecation terms. How much notice are you actually owed before a model you depend on goes away? If you're on a standard developer plan, the answer is probably "whatever the public policy says," which can change.
Prove that failover works. Not "we have a second provider configured" — actually run a game day where the primary returns 429s and 500s and see what your users experience.

None of this is IPO-specific advice. It's just ordinary production hygiene that an IPO makes newly urgent, because the difference between a well-run integration and a fragile one only becomes visible when the vendor's incentives shift.

The bottom line

Anthropic filing to go public is, on balance, good for developers. A company with $47B in run-rate revenue and public-market accountability is a more predictable dependency than a private lab whose economics you have to guess at. You get disclosure, you get a documented risk profile, and you get a competitor that has to keep winning on capability because the scoreboard is now public.

What you also get is a vendor with a quarterly clock, which means the slack in free tiers, generous rate limits, and leisurely deprecation windows is likely to tighten before it loosens. The teams that come out ahead won't be the ones who predicted the ticker. They'll be the ones who spent this quarter making a model swap a config change instead of an incident.

If you want structured, hands-on training on any of this — evals, model selection, production LLM applications — that's what we build at Cursuri-AI.ro.

Sources: Anthropic — confidential draft S-1 submission · TechCrunch — Anthropic files to go public · CNBC — SpaceX IPO, SPCX first-day close

The EU AI Act's Transparency Rules Went Live — Here's What You Actually Have to Ship

galian — Sun, 09 Aug 2026 21:42:39 +0000

Most AI regulation, for most developers, has been something that happens to other people. Risk classifications, conformity assessments, notified bodies — the kind of thing where you nod, assume legal will handle it, and go back to your streaming handler.

Article 50 is different, and it went live on 2 August 2026. It's the part of the EU AI Act that doesn't care whether your system is "high risk." It cares about one thing: can the person on the other end tell that this is AI? If the answer is no, you owe them a disclosure — and the disclosure is a product decision, a UI decision, and in one case a file-format decision. All three land on engineering.

The Commission adopted its guidelines on Article 50 on 20 July 2026, and the AI Office published a Code of Practice on Transparency of AI-Generated Content on 10 June 2026. Between those two documents and the article text itself, the shape of what you have to build is now reasonably clear. This is that shape, translated into work items.

First: does this apply to you?

Two questions, and you're probably in scope on both.

Are you a provider or a deployer? The Act splits duties. A provider develops an AI system (or has one developed) and places it on the market under its own name — if you built the chatbot or the image generator, that's you. A deployer uses an AI system under its own authority — if you dropped someone else's model into your support widget, that's you. Article 50 assigns two obligations to providers and two to deployers, and plenty of teams are both.

Does the EU reach you? The Act's territorial hooks (Article 2) are not "are you an EU company." They're closer to "is the system placed on the EU market, or is its output used in the EU." A US startup with EU users is in scope. This is the GDPR pattern, and it caught a lot of people by surprise the first time.

Not in scope: things that don't interact with people or generate content for them. The Commission's guidelines explicitly put spam filters and automated translation tools outside Article 50(1), and voice assistants and chatbots inside it.

Obligation 1 — tell people they're talking to an AI

Providers shall ensure that AI systems intended to interact directly with natural persons are designed and developed in such a way that the natural persons concerned are informed that they are interacting with an AI system, unless this is obvious from the point of view of a natural person who is reasonably well-informed, observant and circumspect. — Article 50(1)

This is the one that hits the most products, and it's the easiest to get wrong in a way that looks compliant.

The "obvious" carve-out is narrower than you want it to be. The test isn't "our users are technical." It's a reasonably well-informed, observant and circumspect person in the circumstances and context of use. A widget labelled "AI Assistant" in a developer tool is plausibly obvious. The same engine answering an inbound phone call in a warm, human-sounding voice is not — and voice is exactly where the gap is widest right now, because realtime voice agents are good enough that the "obvious" defence has quietly stopped being true.

Where the disclosure has to live. Article 50(5) settles the argument you're about to have with someone in a planning meeting: the information must be provided at the latest at the time of the first interaction or exposure, and it must be clear and distinguishable. That rules out:

burying it in the Terms of Service
a footnote in the privacy policy
a tooltip behind a hover on a mobile UI
disclosing on turn three, after the user has already asked something personal

Article 50(5) also requires conformity with applicable accessibility requirements — which in practice means your disclosure has to survive a screen reader, not just a design review.

What that looks like in a chat surface is boring, and boring is the point:

<!-- Rendered before the first assistant message, not after it. -->
<div class="ai-disclosure" role="note">
  <strong>You're chatting with an AI assistant.</strong>
  Answers are generated automatically.
  <a href="/support/human">Talk to a person</a>
</div>

For voice, the equivalent is a spoken line in the first turn — before you collect anything, and in the language of the call. For an API you sell to other developers, the honest move is to pass the obligation downstream explicitly: document it, and give integrators a disclosure string they can render, because when they ship your model to end users under their own brand, the deployer duties become theirs and the design duty stays yours.

One more thing worth building while you're in there: a handoff path to a human. Article 50 doesn't mandate it. But the disclosure lands very differently when it's followed by an escape hatch, and support teams that ship AI assistants without one tend to discover the reason the hard way.

Obligation 2 — mark synthetic output so machines can detect it

Providers of AI systems […] generating synthetic audio, image, video or text content, shall ensure the outputs […] are marked in a machine-readable format and detectable as artificially generated or manipulated. — Article 50(2)

This is the genuinely hard engineering item, and it's the one with a different deadline (more on that below).

Two properties are required, and they're not the same thing:

Machine-readable marking — metadata that a downstream system can parse.
Detectability — the output can be recognised as artificially generated or manipulated.

The Act asks for solutions that are "effective, interoperable, robust and reliable as far as this is technically feasible" — a standard that explicitly bends to the state of the art. It also carves out assistive editing functions and systems that do not substantially alter the input data. Your auto-crop and your denoise filter are not in scope. Your "generate a product photo from this prompt" endpoint is.

What the Code of Practice points at. The AI Office's Code, published 10 June 2026, describes a layered approach rather than a single mechanism: signed, timestamped provenance metadata — where C2PA is the standard identified as meeting those criteria — plus an imperceptible watermark embedded in the content itself, robust enough to survive ordinary transformations like compression, cropping, scaling and format conversion. The Code is voluntary and, at the time of writing, going through an adequacy assessment by the Commission and the AI Board. Adhering to it is a route to demonstrating compliance; not adhering means you have to show equivalently adequate means of your own.

In practice, for images and video, that means attaching C2PA Content Credentials at generation time. The Content Authenticity Initiative ships open-source tooling for this — c2pa-rs with Python, JS, C++, Swift and Android bindings, plus a c2patool CLI — so this is a library integration, not a research project.

The assertion that carries "this was AI-generated" is the IPTC digital source type, referenced inside a c2pa.actions assertion:

{
  "assertions": [
    {
      "label": "c2pa.actions",
      "data": {
        "actions": [
          {
            "action": "c2pa.created",
            "digitalSourceType":
              "http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia"
          }
        ]
      }
    }
  ]
}

trainedAlgorithmicMedia is the IPTC code for content created by a generative model. There are neighbouring codes for composites and for algorithmically edited media — pick the one that actually describes what your pipeline did, because "created" on a system that only retouched is its own kind of wrong. Verify the current assertion shape against the C2PA spec before you ship; the standard is still moving.

Three things nobody tells you in the compliance deck:

Metadata gets stripped. Plenty of platforms re-encode uploads and discard provenance metadata on the way in. Signing at generation is necessary; assuming it survives the internet is not. This is precisely why the Code pairs metadata with a watermark instead of trusting either alone.
Text is the weak link. Machine-readable marking of text has no equivalent of C2PA that works after a copy-paste. Statistical watermarking of token distributions exists, degrades under paraphrase, and doesn't survive a user retyping the paragraph. The Act's "as far as technically feasible" language is doing real work here — but "hard" is not "exempt," and documenting your reasoning is part of the deliverable.
Sign server-side. Any marking applied in the browser is marking a determined user can skip. The signature belongs on the generation path, before the bytes reach a client.

If you're generating images or video in a product today, this obligation is now a line item in your media pipeline, not a policy question.

Obligation 3 — emotion recognition and biometric categorisation

Deployers of an emotion recognition system or a biometric categorisation system shall inform the natural persons exposed thereto of the operation of the system, and shall process the personal data in accordance with [the GDPR and related instruments]. — Article 50(3)

Shorter, and mostly a matter of knowing that it applies to you. If your product infers emotional state from voice, face or text, or sorts people into categories from biometric data, you inform the people exposed to it — and you're squarely in GDPR territory on top, usually with special-category data.

Before you scope the disclosure: check Article 5 first. Some emotion recognition — in the workplace and in education — is prohibited outright, not merely subject to transparency, and has been since February 2025. Article 50 is the wrong chapter to be reading if that's your use case.

Obligation 4 — deepfakes and public-interest text

Deployers of an AI system that generates or manipulates image, audio or video content constituting a deep fake, shall disclose that the content has been artificially generated or manipulated. — Article 50(4)

Note the split from Obligation 2: marking the file is the provider's duty; disclosing to the audience is the deployer's. If you use a third-party model to produce a synthetic spokesperson for a campaign, the vendor's C2PA manifest doesn't discharge your obligation. You still have to tell the audience.

Two carve-outs matter:

Artistic and satirical works. Where the content is part of an evidently artistic, creative, satirical or fictional work, the disclosure shrinks to revealing the existence of generated content in an appropriate manner that does not hamper the display or enjoyment of the work. A film doesn't need a permanent banner across the frame.

Text on matters of public interest. AI-generated or manipulated text published to inform the public on matters of public interest must be disclosed — unless it has undergone human review or editorial control and a natural or legal person holds editorial responsibility. This is the clause every content-heavy site should read twice. An unreviewed AI-written news summary needs a label. The same article with a named editor who checked it and owns it does not. If your publishing workflow can't currently prove which of those two happened, that's the actual gap — and it's a workflow problem before it's a legal one, which is why content operations built on AI now need an audit trail as much as a style guide.

The deadlines, which are not all the same

The Digital Omnibus on AI — published in the Official Journal on 24 July 2026 and in force since 27 July — shifted several AI Act dates. Article 50 came out of it mostly intact:

What	Applies from
Article 50(1) AI-interaction disclosure	2 August 2026
Article 50(3) emotion recognition / biometric categorisation	2 August 2026
Article 50(4) deepfake and public-interest text disclosure	2 August 2026
Article 50(2) marking, systems placed on the market on or after 2 Aug 2026	2 August 2026
Article 50(2) marking, systems placed on the market before 2 Aug 2026	2 December 2026
Annex III high-risk obligations (Chapter III)	deferred to 2 December 2027
High-risk AI embedded in regulated products (Annex I)	deferred to 2 August 2028

That last pair is the source of most of the confusion in the room right now: the high-risk regime got a long deferral, and a lot of teams heard "the AI Act got pushed back" and stopped reading. Article 50 did not get pushed back. The only grace period is the marking obligation for generative systems that were already on the market, and it expires on 2 December 2026.

Enforcement sits with national market surveillance authorities, the AI Office and — for EU institutions — the European Data Protection Supervisor. Breaching Article 50 carries fines of up to €15 million or 3% of worldwide annual turnover, whichever is higher.

A ship checklist

Pin this to the epic:

[ ] Inventory every surface where an AI system talks to a person — chat, voice, email autoresponders, in-app agents. Each one gets a first-interaction disclosure or a written argument for why it's obvious.
[ ] Disclosure rendered before the first AI output, accessible, in the user's language, not in the ToS.
[ ] Every generative endpoint identified as provider-side or deployer-side. Write it down; vendor contracts should say the same thing.
[ ] C2PA Content Credentials signed server-side on image/audio/video generation, with the correct IPTC digitalSourceType.
[ ] Watermarking assessed for each modality; where it isn't feasible, the reasoning is documented rather than assumed.
[ ] Deepfake disclosure at the publication surface, not just in the file metadata.
[ ] Editorial-review provenance recorded for AI-assisted public-interest text — who reviewed, when, who owns it.
[ ] Emotion recognition / biometric categorisation checked against Article 5 prohibitions before anything else.
[ ] Disclosure copy and placement covered by a test, so the next redesign doesn't silently delete it.

Conclusion

The thing that makes Article 50 unusual, as regulation goes, is how little of it is paperwork. There's no conformity assessment here, no technical documentation dossier, no notified body. There's a banner that has to render before the first message, a signature that has to happen on the generation path, a label that has to reach the audience, and a record of who reviewed what. Four engineering tickets, roughly, and none of them are hard.

They're just easy to defer — and the deferral is what gets expensive. Article 50 is now enforceable, the guidelines are published, the Code of Practice exists, and the tooling for the hard part is open source. Compliance here is mostly a question of whether someone put it in the sprint.

The teams that will have the least trouble with this aren't the ones with the biggest legal department. They're the ones that were already willing to tell users, plainly, what the machine was doing. Turns out that was always the good product decision — it just became the required one.

Written by the team behind Cursuri-AI.ro, an AI education platform with hands-on English-language courses on AI, data privacy and EU AI Act compliance, production LLM integration, and shipping AI products.

Sources & further reading:

EU AI Act — Article 50: Transparency Obligations for Providers and Deployers of Certain AI Systems (full text, exemptions, codes of practice)
European Commission — Guidelines on transparency obligations for providers and deployers of certain AI systems (adopted 20 July 2026)
European Commission — Code of Practice on Transparency of AI-generated Content (published 10 June 2026)
Content Authenticity Initiative — open-source C2PA SDKs and c2patool
IPTC — Digital Source Type NewsCodes vocabulary (trainedAlgorithmicMedia and related values)

This article is educational content written by engineers, not legal advice. Article 50 interacts with the GDPR, the DSA, national implementing rules and sector regulation, and the Digital Omnibus changed several dates in 2026 — verify against current official sources and your own counsel before shipping.

Claude Opus 5 Is Out — Here Are the API Changes That Will Actually Break Your Code

galian — Tue, 28 Jul 2026 22:38:55 +0000

Anthropic shipped Claude Opus 5 on July 24, 2026, and if you build on the Claude API there's a version of this launch story you can safely skip: the benchmark charts. The version you can't skip is the API contract, because for the first time in a while an Opus release changes how existing requests behave — and one parameter combination that was perfectly valid on Opus 4.8 now returns a 400 error.

I write and teach about AI engineering at Cursuri-AI.ro, Eastern Europe's AI education platform, and this post is the writeup I wish every model launch came with: not "how smart is it," but "what do I need to change in my code, in what order, and what silently behaves differently if I change nothing." Everything below comes from Anthropic's official "What's new in Claude Opus 5" documentation — no leaked numbers, no vibes.

Quick disclaimer: this space moves monthly and this is a launch-window snapshot. Verify against the official docs before you wire anything to production.

The spec sheet, in one table

Spec	Claude Opus 5
Model ID	`claude-opus-5` (`anthropic.claude-opus-5` on Bedrock)
Context window	1M tokens — both default and maximum (no smaller variant)
Max output	128k tokens
Thinking	On by default
Pricing	$5 / $25 per million input/output tokens — unchanged from Opus 4.8
Fast mode	$10 / $50, research preview, Claude API only

Two of these are bigger deals than they look. The 1M context window isn't an opt-in variant — it's just what the model has, and the docs specifically claim consistent instruction following, tool calling, and reasoning throughout the window. And 128k max output makes single-request long deliverables (big refactors, full reports) realistic — with a caveat about max_tokens we'll get to, because it's now doing more work than it used to.

Opus 4.8 stays available on every platform, so nothing forces a same-day migration. But the changes below are the kind you want to understand before your first "why is this request failing" incident, not after.

Change #1: thinking is on by default

On Opus 4.8, a request without a thinking field ran without extended thinking. You opted in with thinking: {"type": "adaptive"}.

On Opus 5, that same field-less request now runs with thinking on. The model decides when and how much to think on each turn, and the effort parameter is the knob that controls thinking depth. If your code already sends thinking: {"type": "adaptive"}, you're fine — that value remains valid and is equivalent to the new default.

Why this can bite you even though it sounds like a free upgrade:

max_tokens is a hard limit on total output — thinking plus response text. A workload that ran happily with max_tokens: 2000 on Opus 4.8 (no thinking, short answers) can now spend a chunk of that budget on reasoning before it writes a single visible token. The official guidance is explicit: revisit max_tokens for every workload that previously ran without thinking.

Change #2: the actual breaking change — disabled thinking + high effort = 400

Here's the one to grep your codebase for today:

thinking: {"type": "disabled"} is accepted only when effort is high or below.
thinking: {"type": "disabled"} combined with effort xhigh or max returns a 400 error, on every request.
This is generally available behavior (not beta) from Opus 5 onward — and it's a breaking change from Opus 4.8, where disabling thinking was independent of the effort level.

If you currently run thinking-disabled at high effort levels, you have exactly two exits: keep thinking disabled and drop effort to high or below, or keep your effort level and delete the thinking field entirely.

There's a second, subtler reason to just let thinking stay on. The docs note that with thinking disabled, Opus 5 can occasionally write a tool call into its text output instead of emitting a proper tool_use block, or leak internal XML tags into the visible response. If you parse tool calls in production, that's not a cosmetic footnote — it's a failure mode your handler needs to survive. Anthropic's recommendation is to keep thinking enabled and control cost with lower effort levels instead.

Change #3: `effort` is now the control lever that matters

With thinking on by default, effort becomes the central parameter of your integration. The full ladder on Opus 5: low, medium, high (the default), xhigh, and max. The docs make a claim worth taking seriously: Opus 5 converts additional effort into better results more reliably than any earlier Opus model — which means the level you pick carries more weight than it did on 4.8. At the other end, low and medium are explicitly called out for producing strong quality at a fraction of the tokens and latency.

A request with everything turned up looks like this:

import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model="claude-opus-5",
    max_tokens=64000,
    output_config={"effort": "max"},
    messages=[{"role": "user", "content": "..."}],
) as stream:
    response = stream.get_final_message()

Three deliberate details in that snippet: there's no thinking field (the default is what you want), max_tokens is large (at xhigh/max the model needs room to think and act across tool calls), and it's streamed — at 64k-token budgets, non-streaming requests can hit the time limit.

The sane strategy is the documented one: start at the default high, then adjust based on your own evals, not vibes. Step down where quality holds — you pocket the tokens and latency. Step up to xhigh/max for the genuinely hard work. If you don't have an eval harness that can answer "did quality hold when I dropped effort?", that's the single highest-leverage thing to build this quarter — it's the discipline we treat as foundational in our course on LLM evals in production.

Change #4: the model behaves differently, even if you change nothing

The docs have a refreshingly honest section on differences you'll notice without touching your code:

Default responses run longer — both user-facing answers and written deliverables. If your product has strict length constraints, enforce them in the prompt.
In agentic sessions, the model narrates its progress more often. Good for UX transparency; tune it down where you want silence.
In multi-agent frameworks, it delegates to subagents more readily — budget accordingly, subagents are tokens too.
It verifies its own work without being told to. This is the one that demands action: verification instructions inherited from older prompts — "include a final verification step," "use a subagent to verify" — should be removed, because on Opus 5 they cause over-verification. You pay twice, in tokens and latency, for work the model already does.

This is a pattern every migration teaches: prompts get calibrated against the previous model's weaknesses, and those calibrations become friction on the next model. A model migration without a prompt audit is half a migration. That workflow — prompts, tools, verification, on real repos — is exactly what we drill in our Claude Code and agentic coding course.

The smaller changes you'll actually be happy about

Prompt caching minimum drops to 512 tokens, from 1,024 on Opus 4.8. System prompts that were too short to cache start caching with zero code changes. If you run compact prompts at volume, this shows up on your invoice by itself.
Mid-conversation tool changes (beta): you can add or remove tools between turns while preserving the prompt cache, with the mid-conversation-tool-changes-2026-07-01 beta header. For phase-based agents (explore → edit → verify, different tools each), this removes a whole architectural compromise — no more front-loading every tool "just in case."
A "default" mode for fallbacks (beta): the fallbacks parameter can now apply Anthropic's recommended fallback models by refusal category, instead of a hand-maintained model list. Use the server-side-fallback-2026-07-01 header (the older 2026-06-01 header only accepts explicit lists).
Fast mode exists but is API-only: $10/$50 per million tokens, research preview, not currently on Bedrock, Google Cloud, or Microsoft Foundry. If you route through a cloud provider and were counting on it, adjust your plan.

The migration checklist, in the order things break

Grep for the fatal combination: thinking: {"type": "disabled"} anywhere near effort xhigh/max → that's a 400 now. Decide per integration: drop the thinking field, or lower effort.
Revisit max_tokens on every request that ran without thinking on 4.8 — the budget now covers reasoning + response.
Swap the model ID to claude-opus-5 — from a config variable, not a hardcoded string. (Model retirements turn hardcoded IDs into production 404s; ask me how I know.)
Audit your prompts for verification instructions and remove them — over-verification is pure waste on Opus 5.
Turn on streaming for anything with a large max_tokens.
If you must run thinking-disabled, make your parser survive tool calls in text and stray XML tags.
Re-run your evals before shifting traffic. "Better on average" is not "better on your tasks."

If your architecture makes step 3 scary — model IDs scattered across services, no config layer, no eval gate — that's a structural problem worth fixing once, properly. Going from "API calls sprinkled through the codebase" to "an application where a model swap is a config change" is the arc of our building AI applications with Python course.

The bottom line

Opus 5 doesn't ask you to learn a new API — it asks you to re-check the assumptions your existing code was built on. Thinking is on unless you say otherwise, effort is the lever that matters, max_tokens now pays for cognition too, and one previously-valid parameter combo is a hard error. In exchange you get a 1M-token window as the default, 128k output, cheaper caching, and betas that clean up real architectural pain.

Teams that treat this as a checklist migration — rather than a find-and-replace on the model name — will come out with integrations that are cheaper and more reliable than what they had. And that discipline, built once, pays out again on every launch that follows.

If you want structured, hands-on training on any of this — evals, agentic coding, production LLM apps — that's what we build at Cursuri-AI.ro.

Multi-Agent AI Systems: When One Agent Isn't Enough — and When Two Is Already Too Many

galian — Wed, 22 Jul 2026 20:54:34 +0000

Somewhere around the third time your agent's context window filled up with search results it no longer needed, you had the thought every agent builder eventually has: what if I split this into multiple agents? One to research, one to write, one to review. Maybe a "manager" agent on top. It sounds like an org chart, and org charts feel like architecture.

Sometimes that instinct is exactly right — Anthropic's research system, built as an orchestrator with parallel subagents, outperformed its best single-agent setup by a wide margin on internal evals. And sometimes it's exactly wrong — Cognition (the team behind Devin) published a piece bluntly titled "Don't Build Multi-Agents", arguing that for their domain, splitting the work is how you manufacture inconsistency. Both teams are right, because they're solving different problems.

This article is the decision framework between those two positions: what multi-agent architectures actually buy you, what they cost, the orchestration pattern that works in production, and the failure modes that don't make it into the launch posts.

What a multi-agent system actually is

Strip the buzz and it's simple: a multi-agent system is one where an LLM-driven agent can delegate work to other LLM-driven agents, each running in its own context window, and use their results. The common production shape is orchestrator–worker: a lead agent owns the goal, decomposes it into subtasks, spawns subagents to execute them (often in parallel), and synthesizes what comes back.

That "own context window" clause is the entire point. It's not about the anthropomorphic org chart — it's about context isolation. Everything else follows from it.

Notice what this is not: a pipeline of prompts you call in sequence from your own code is not a multi-agent system — it's just a program (and often, that's exactly what you should build instead). The multi-agent label earns its keep when the decomposition itself is dynamic — when an agent decides at runtime what to delegate, to whom, and with what instructions.

The three things fan-out actually buys you

1. Context isolation. A subagent that greps a codebase or reads twenty web pages burns thousands of tokens on intermediate noise — raw file contents, search results, dead ends. In a single-agent design, all of that lands in the one context window you have, where it dilutes attention for every subsequent step. In an orchestrator–worker design, the noise lives and dies in the subagent's window; the orchestrator receives a compressed conclusion. You've effectively multiplied your usable context by the number of workers.

2. Parallelism. Independent subtasks — "evaluate these five libraries," "check each of these twelve services for the deprecated call" — can run concurrently instead of serially. For research-shaped and audit-shaped work, this is a wall-clock improvement measured in multiples, not percent.

3. Separation of concerns. Each subagent gets a narrow prompt, a curated toolset, and one job. A focused agent with six relevant tools reliably beats a generalist juggling twenty — the same reason you curate tool sets aggressively in context engineering for agents, applied at the architecture level.

Anthropic's engineering write-up on their multi-agent research system put numbers on this: the orchestrator-with-parallel-subagents architecture outperformed their single-agent baseline by 90.2% on an internal research eval, and they found that token usage — how much relevant thinking-and-reading the system could do — explained most of the performance variance. Fan-out is, at bottom, a way to spend more tokens productively on one problem than a single context window physically allows.

What it costs you

The same write-up is refreshingly honest about the bill: in their data, a single agent used roughly 4× more tokens than a chat interaction, and multi-agent systems roughly 15×. That's the entry fee. Multi-agent architectures only make economic sense when the task's value clears it.

The subtler cost is coordination. The orchestrator communicates with subagents through task descriptions — and every ambiguity in those descriptions becomes a subagent doing confidently the wrong thing in a context you can't see. Early versions of Anthropic's system had subagents duplicating each other's work, wandering off-scope, and searching for things other subagents had already found — not because the model was weak, but because the delegation instructions were vague. The fix was unglamorous prompt engineering: each delegation needs an explicit objective, output format, tool guidance, and boundaries where the subtask ends.

The case against — and it's a strong one

Cognition's counter-argument deserves to be taken seriously, because it identifies the precise condition under which multi-agent designs fail: when subtasks are not actually independent.

Their core principles: share full context and trace history, and remember that actions carry implicit decisions. When two subagents work in parallel from partial views of the task, each makes small implicit decisions — naming, structure, interpretation of an ambiguous requirement. When their outputs meet, those decisions conflict. For coding work, where part A and part B must compile, link, and agree on conventions, parallel agents with partial context produce exactly the inconsistency you'd expect from two contractors who never spoke to each other.

So the honest synthesis of the two positions isn't "Anthropic vs. Cognition" — it's a property of the task:

Read-heavy, decomposable, breadth-first work (research, audits, evaluation sweeps, codebase exploration) parallelizes beautifully. Subagent outputs are findings that an orchestrator can merge, and conflicts between them are cheap to resolve.
Write-heavy, interdependent, convention-laden work (feature implementation, refactoring, anything where outputs must cohere) punishes fan-out. Here a single agent with a well-managed context — compaction, external memory, just-in-time retrieval — is the stronger architecture.

A useful mnemonic: fan out to read, stay single to write. It's not absolute (isolated write tasks in separate worktrees parallelize fine), but it's the right default, and knowing when to break it is precisely the judgment that a serious course on AI agent architecture spends real time building.

The orchestration patterns that survive production

If your task passes the test above, these are the shapes that hold up:

Orchestrator–worker (the default). One lead agent decomposes, delegates, synthesizes. Workers don't talk to each other — results flow through the orchestrator. This keeps the communication topology a star, not a mesh, which is the difference between debuggable and not. Resist the peer-to-peer "agents chatting with agents" fantasy; in production it mostly produces token-expensive misunderstandings.

Pipeline with isolated stages. Each item flows through fixed stages (find → verify → summarize), each stage its own agent call with its own clean context. No barrier between items — item A can be in stage 3 while item B is in stage 1. Deterministic control flow in your code, LLM judgment inside each stage. This hybrid — code decides the structure, agents fill the slots — is the most underrated pattern in the space, and it's where most "do we even need a framework?" questions dissolve.

Adversarial verification. Spawn N independent agents to refute a finding rather than confirm it, and keep only what survives. This is fan-out used not for speed but for confidence — independent contexts mean independent mistakes, which is exactly what majority voting needs to work. It's the cheapest defense against the plausible-but-wrong output that a single agent will happily double down on.

Three implementation rules that pay for themselves regardless of pattern:

Delegate like you're writing a ticket for a contractor. Objective, output format, tools to use, scope boundaries. If a competent human would need to ask a clarifying question, your subagent will instead guess — silently.
Return data, not prose. Subagents should report structured results (schema-validated if your stack allows it), because the orchestrator is a program consuming output, not a reader enjoying it.
Match effort to value. Simple lookups get one worker with a small budget; open-ended research earns a fleet. An orchestrator that spawns ten subagents for a question one search would answer is the multi-agent version of the microservices monorepo with three users.

The failure modes nobody demos

Every one of these is a thing that happens after the demo works:

Duplicated work. Two subagents independently discover the same fact at full token price, because the orchestrator's task descriptions didn't partition the space. Partition explicitly.
Compounding errors. In a pipeline, a stage-one hallucination becomes stage-two's trusted input. Multi-agent systems don't just propagate errors — they launder them: by the final synthesis, the wrong fact arrives with the confidence of having been "processed" three times. This is why verification stages belong early, not only at the end.
The telephone game. Every orchestrator-mediated hop compresses information. Three hops in, "the test fails intermittently on CI under load" has become "the tests are broken." Keep hierarchies shallow — one level of delegation covers almost every real use case.
Lost partial work. Ten parallel subagents, one crashes at minute eight. If your harness can't resume without re-running the other nine, you'll pay the 15× token bill more than once. Checkpointing and resumability are boring, and they are the difference between a system and a demo.

And the meta-failure that enables all of the above: not measuring. A multi-agent system has more knobs than anything else you'll build with LLMs — decomposition granularity, worker count, delegation prompts, synthesis strategy — and every knob is a chance to regress silently. The teams that ship these systems treat evaluation as first-class infrastructure: a representative task set, end-to-end outcome scoring (with an LLM judge where rubrics allow), and a re-run on every architectural change. "It seemed better on the three queries I tried" is how 15× token bills get approved for 1× results.

A decision checklist

Before you split one agent into several, put the task through this:

Question	If yes →	If no →
Can subtasks run with genuinely independent context?	fan-out is viable	stay single-agent
Is the work read/analyze-heavy rather than write/produce-heavy?	fan-out helps	single agent + context engineering
Does the task value clear a ~15× token cost?	proceed	simplify the architecture
Would a fixed pipeline in code + single agent calls do it?	build that instead	orchestrator–worker
Do you have evals to catch coordination regressions?	ship	build evals first

Note how often the honest answer is the boring one. A single agent with disciplined context management — compaction, external memory, sub-tasking only when context demands it — is the right architecture for most of what gets built today, and wiring either variant into a real product with retries, budgets, and observability is its own production discipline on top of the architecture choice.

Conclusion

Multi-agent systems aren't the next tier above single agents — they're a specialized tool that trades tokens and coordination complexity for context capacity and parallelism. That trade is spectacular for breadth-first, read-heavy, decomposable work, and actively harmful for interdependent, convention-heavy production of artifacts. Fan out to read; stay single to write; let deterministic code own the control flow either way; and put an eval behind every knob.

The teams that get this right aren't the ones with the most agents. They're the ones who can articulate, for their specific task, what a second context window buys — and who kept the architecture exactly one notch more complex than that answer requires.

I build and teach this material at Cursuri-AI.ro, an AI education platform with hands-on English-language tracks on AI agent architecture, context engineering, LLM evaluation, and production integration.

Sources & further reading:

Anthropic — How we built our multi-agent research system (orchestrator–worker pattern, 90.2% eval result, token-cost data, delegation lessons)
Anthropic — Building effective agents (workflows vs. agents, orchestrator–workers, "simplest thing that works")
Cognition — Don't Build Multi-Agents (context-sharing principles, the case for single-threaded agents)

This article is educational content. Architectures, model capabilities, and costs evolve quickly — validate against your own workloads and current documentation.

Fine-Tuning vs RAG vs Prompting: How to Actually Decide in 2026

galian — Wed, 15 Jul 2026 11:49:01 +0000

There's a predictable arc to most LLM projects. Something doesn't work, someone says "we should fine-tune it," a month disappears into dataset wrangling and GPU bills, and the model comes back... about as wrong as before — because the actual problem was that it never had the right facts in front of it. Fine-tuning was never going to fix that.

The three techniques — prompting, retrieval-augmented generation (RAG), and fine-tuning — are not a ladder you climb from cheap to fancy. They solve different problems, and choosing the wrong one is expensive in exactly the way that's hard to notice: it looks like progress while it burns weeks.

This is a decision framework. Not "here's what each one is" — you can get that anywhere — but the concrete questions that tell you which one your problem actually needs, and the failure signatures that mean you picked wrong.

The one distinction that resolves most arguments

Before any framework, internalize this split, because it settles 80% of the "should we fine-tune?" debates on its own:

RAG changes what the model knows. It injects facts into the context at inference time.
Fine-tuning changes how the model behaves. It adjusts the weights to shift style, format, structure, and task-specific skill.
Prompting changes what the model is told to do right now, using the knowledge and behavior it already has.

So the first question is never "which technique is best?" It's "is my problem a knowledge gap or a behavior gap?"

The model gives outdated, made-up, or "I don't have access to that" answers about your data → knowledge gap → RAG.
The model knows the facts but won't reliably produce the format, tone, or task structure you need → behavior gap → fine-tuning (maybe).
You haven't seriously tried telling it clearly what to do yet → prompting, first, always.

Get this wrong and no amount of engineering saves you. Fine-tuning a model to "know your docs" is the classic error: you can bake a few facts into weights, but they go stale the moment your docs change, you can't cite sources, and you've spent training compute to build a worse version of a lookup. Knowledge that changes belongs in retrieval, not in weights.

Always start with prompting (yes, even now)

Prompting is not the beginner tier you graduate from. In 2026, with frontier models, a well-constructed prompt plus a few good examples solves a startling share of problems that teams assume need training. It's the fastest, cheapest, most inspectable option, and it should be your baseline before you're allowed to say the word "fine-tune."

Reach for prompting when:

You're still discovering what "good output" even looks like. Prompts are editable in seconds; datasets are not.
The task is reasoning, transformation, or generation the model already broadly knows how to do.
You need to ship this week.

The techniques that make prompting punch above its weight are unglamorous but real: precise role and task framing, few-shot examples that demonstrate the exact output shape, chain-of-thought for multi-step reasoning, and rigid output contracts (structured/JSON) so downstream code can trust the result. Most "the model can't do this" conclusions are actually "we asked badly" conclusions. Squeezing the ceiling out of prompting before spending on anything heavier is a discipline in itself — it's the whole point of a prompt engineering masterclass, and the ROI of getting it right first is enormous because everything downstream inherits a better baseline.

The prompting ceiling — how you know you've hit it: you've iterated seriously, added good examples, and the model still fails — and the failure is either (a) it doesn't know facts it couldn't possibly know, or (b) it can't hold a consistent behavior across inputs no matter how you phrase the instruction. (a) points to RAG. (b) might point to fine-tuning. Not before.

Reach for RAG when the problem is knowledge

RAG is the answer whenever the model needs to work with information it wasn't trained on: your internal documentation, a product catalog, last week's tickets, a knowledge base that updates daily, anything private or fresh.

Choose RAG when:

Answers must be grounded in a specific corpus and you need to cite sources.
The knowledge changes — pricing, policies, docs, inventory. You update an index, not a model.
Hallucination on facts is unacceptable and you need an audit trail of where an answer came from.
The knowledge base is large, or partly access-controlled per user.

The reason RAG beats fine-tuning for knowledge isn't subtle: updating a document store is trivial and instant; updating weights is a training run. RAG gives you freshness, provenance, and per-user access control for free — none of which fine-tuning can offer. When your facts have a shelf life, retrieval is the only correct architecture, and building it well (chunking, hybrid search, re-ranking) is where the real engineering lives — the substance of a dedicated course on RAG and retrieval-augmented generation.

RAG's own ceiling: retrieval fixes what the model knows, not how it behaves. If your RAG answers are factually correct but come out in the wrong format, wrong tone, or don't follow your house style no matter how you prompt — that residual behavior gap is exactly where fine-tuning finally earns its place, on top of RAG, not instead of it.

Fine-tune when the problem is behavior — and only then

Fine-tuning is the right tool, but for a narrower set of problems than its reputation suggests. It shines at teaching consistent behavior that's hard to specify in a prompt: a very specific output structure, a domain's tone and terminology, a classification or extraction task where you have lots of labeled examples, or a skill the base model does clumsily.

Legitimately reach for fine-tuning when:

You need consistent style, format, or structure at a level prompting can't hold across the full input distribution.
You have a narrow, high-volume, well-defined task (classification, extraction, a specific transformation) and enough quality labeled data.
You want to bake in a behavior so you can drop it from the prompt — shorter prompts, lower per-call cost, faster responses at scale.
Latency or cost at scale matters and a smaller fine-tuned model can match a bigger prompted one.

Two things make modern fine-tuning far less scary than its reputation. First, you almost never do full fine-tuning — parameter-efficient methods like LoRA/QLoRA train a tiny set of adapter weights, cutting the compute and memory cost by orders of magnitude while getting most of the benefit. Second, the bottleneck is data quality, not model choice: a few hundred to a few thousand clean, consistent, representative examples beat a huge noisy pile every time. The hard part of fine-tuning was never running the training job — it's building the dataset, choosing PEFT trade-offs, and evaluating the result without fooling yourself, which is precisely the ground a fine-tuning course has to cover to be worth anything.

When fine-tuning is the wrong answer — the red flags:

"We'll fine-tune it on our docs so it knows them." → No. That's RAG. Fine-tuned facts go stale and can't be cited.
"We haven't really tried prompting." → Do that first; you may not need to train at all.
"The requirements change weekly." → Fine-tuning bakes behavior in; if the target moves, you're re-training constantly. Keep it in the prompt until it stabilizes.
"We have 40 examples." → Usually not enough for reliable behavior change; strong prompting with those 40 as few-shot examples will likely beat it.

The combinations are the real answer

Framing these as rivals is the beginner mistake. In production, the strongest systems combine them, because they operate on different axes — knowledge, behavior, and instruction — and stack cleanly:

RAG + prompting is the workhorse for most knowledge-grounded assistants: retrieve the right context, then a well-engineered prompt instructs the model to answer only from it and cite sources. No training required.
Fine-tuning + RAG is the high end: fine-tune for the domain's behavior (tone, format, task skill), and use RAG for the facts. The model behaves exactly right and stays current — behavior in the weights, knowledge in the index.
Fine-tuning + prompting collapses a long, brittle instruction into learned behavior, so your prompts get short and your inference gets cheaper.

Orchestrating these — deciding which layer owns which responsibility, and routing a request through retrieval, tools, and the model in the right order — is its own engineering discipline, and it's the core of a course on advanced LLM integration. The mental model to keep: knowledge → retrieval, behavior → weights, instruction → prompt. Put each requirement on the axis it actually lives on.

The decision, in one pass

Run your problem through this, in order. Stop at the first that fits:

Have you genuinely exhausted prompting — clear instructions, good few-shot examples, structured output? If not → prompt. (This is where most projects should still be.)
Is the failure a knowledge gap — missing, stale, or private facts; needs citations? → RAG.
Is the failure a behavior gap — format/tone/task consistency the prompt can't hold, and you have quality labeled data and the target is stable? → fine-tune (LoRA first).
Is it both? → RAG for the facts, fine-tuning for the behavior. In that order.

And underneath all of it: you cannot make this decision without evaluation. "It seems better" is not data. Before you choose, build a small eval set — representative inputs with known-good outputs — so you can measure whether prompting already clears the bar, whether RAG actually retrieves the right context, and whether a fine-tune moved the metric or just moved the failures around. Teams that skip this pick techniques by vibes and discover the mistake in production; teams that treat evals as first-class make the cheap correct choice on purpose. The eval set is what turns this framework from an opinion into a decision.

Conclusion

The reason so many LLM projects stall isn't a shortage of technique — it's reaching for the wrong one and mistaking motion for progress. Fine-tuning a model to "learn facts," RAG-ing a problem that was really a bad prompt, or grinding on prompts when the model fundamentally lacks the data: each fails in a way that looks like effort.

Anchor on the split and you'll rarely go wrong. Knowledge that changes → RAG. Behavior you can't prompt into place → fine-tuning. Everything else → prompt, and prompt well. Start cheap, measure honestly, and add complexity only when an eval — not a hunch — tells you the current layer has topped out. The best architecture isn't the most sophisticated one; it's the one that puts each requirement on the axis where it actually belongs.

Sources & further reading:

Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models
Dettmers et al. — QLoRA: Efficient Finetuning of Quantized LLMs

This article is educational content. Models, tooling, and cost trade-offs evolve quickly; validate any approach against your own data and current provider documentation before committing to it in production.

Vector Embeddings Explained: Build Semantic Search in Python

galian — Mon, 13 Jul 2026 11:25:24 +0000

Search for "reset my password" in a keyword-based system and a help article titled "How to recover your account credentials" won't match — not one word overlaps. Yet any human knows they mean the same thing. Closing that gap between characters and meaning is what vector embeddings do, and they're the quiet engine behind semantic search, RAG, recommendation systems, and most of the "AI that understands you" experiences shipped since 2023.

This is a practical guide. We'll cover what an embedding actually is, why cosine similarity is the operation you'll use constantly, and then build a real, working semantic search engine in Python — first with pure NumPy so you see the mechanics, then with the tools you'd actually reach for in production. By the end you'll have code that runs and a mental model that transfers to every embedding-powered feature you build next.

What an embedding actually is

An embedding is a list of numbers — a vector — that represents a piece of content as a point in high-dimensional space. An embedding model (a neural network trained on enormous text corpora) reads your text and outputs, say, 384 or 1,536 floating-point numbers. The magic isn't the numbers themselves; it's the property the training instills: texts with similar meaning land close together in that space, and unrelated texts land far apart.

That's the whole idea. "How do I reset my password?" and "I forgot my login credentials" produce vectors that sit near each other. "What's the weather in Cluj?" produces a vector off in a completely different region. The model has effectively turned meaning into geometry — and geometry is something a computer can measure with plain arithmetic.

A few properties worth internalizing before we write code:

Dimensionality is fixed per model. A given model always outputs the same length (e.g. 384 for all-MiniLM-L6-v2, 1,536 for OpenAI's text-embedding-3-small). You can't mix vectors from different models — they live in different spaces.
The individual numbers are not interpretable. Dimension 200 doesn't mean "formality" or "topic." Meaning is distributed across all dimensions at once. Don't try to read them.
Distance is the entire point. You almost never care about a vector's absolute position — only how close it is to other vectors.

Cosine similarity: the one operation you'll use everywhere

To ask "how similar are these two texts?", you compare their vectors. The near-universal choice for text embeddings is cosine similarity: it measures the angle between two vectors, ignoring their magnitude.

Why the angle and not, say, straight-line (Euclidean) distance? Because for text embeddings, direction encodes meaning while length often encodes uninteresting things like text length or token count. Two documents about the same topic point the same way even if one is a sentence and the other a paragraph. Cosine similarity captures exactly that.

The formula is just the dot product of the two vectors divided by the product of their magnitudes:

cos(θ) = (A · B) / (‖A‖ · ‖B‖)

It returns a value from -1 to 1, though for most modern text embeddings you'll see results land in roughly the 0-to-1 range:

~1.0 — nearly identical meaning
~0.5 — loosely related
~0.0 — unrelated

That single number, computed against a corpus of stored vectors, is semantic search. Everything else is optimization.

Build a semantic search engine from scratch

Let's make it concrete. We'll use sentence-transformers, which runs a capable embedding model locally — no API key, no network calls, so you can run this offline right now.

pip install sentence-transformers numpy

Step 1 — Embed a corpus

from sentence_transformers import SentenceTransformer
import numpy as np

# A small, fast, widely used model. 384-dimensional output.
model = SentenceTransformer("all-MiniLM-L6-v2")

documents = [
    "How to reset your account password",
    "Refund policy for annual subscriptions",
    "Setting up two-factor authentication",
    "Our office hours and contact details",
    "How to recover a locked account",
]

# Encode the whole corpus once. Shape: (5, 384)
doc_embeddings = model.encode(documents)
print(doc_embeddings.shape)  # (5, 384)

That doc_embeddings array is your search index. In a real app you compute it once, when a document is created or updated, and store it — never on every query.

Step 2 — Cosine similarity in NumPy

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> np.ndarray:
    """Cosine similarity between vector `a` and each row of matrix `b`."""
    a_norm = a / np.linalg.norm(a)
    b_norm = b / np.linalg.norm(b, axis=1, keepdims=True)
    return b_norm @ a_norm

Ten lines, no dependencies beyond NumPy. This is the actual core of semantic search — the rest is plumbing.

Step 3 — Search

def search(query: str, k: int = 3):
    query_vec = model.encode(query)
    scores = cosine_similarity(query_vec, doc_embeddings)
    ranked = np.argsort(scores)[::-1][:k]  # top-k, highest first
    return [(documents[i], float(scores[i])) for i in ranked]

for doc, score in search("I forgot my login credentials"):
    print(f"{score:.3f}  {doc}")

Run it, and you'll get something like:

0.62  How to reset your account password
0.55  How to recover a locked account
0.31  Setting up two-factor authentication

Notice what happened: the query "I forgot my login credentials" shares zero words with "How to reset your account password," yet it ranked first. A keyword search would have returned nothing. That's the payoff — you matched on meaning, not on string overlap. This shift from lexical to semantic matching is the foundation every retrieval-augmented system builds on, and it's the starting point of a structured course on RAG and retrieval-augmented generation that goes from this toy index to production retrieval.

From toy to production: what changes

The NumPy version is perfect for learning and fine for a few thousand documents. Past that, three things force an upgrade.

You need a vector database

Computing cosine similarity against every stored vector on every query is O(n) — fine at 5 documents, painful at 5 million. Vector databases solve this with Approximate Nearest Neighbor (ANN) indexes (HNSW is the common one) that trade a sliver of accuracy for enormous speed, returning near-neighbors in milliseconds over huge corpora.

You have good open-source options:

pgvector — a Postgres extension. If your data already lives in Postgres, this is often the pragmatic choice: vectors and relational data in one place, one backup story.
Chroma / Qdrant / Weaviate / Milvus — purpose-built vector stores with richer filtering and scaling stories.
FAISS — a library (not a server) from Meta for fast similarity search when you want to manage the index yourself.

A minimal Chroma example shows how little the mental model changes:

import chromadb

client = chromadb.Client()
collection = client.create_collection("docs")

collection.add(documents=documents, ids=[f"d{i}" for i in range(len(documents))])

results = collection.query(query_texts=["I forgot my login credentials"], n_results=3)
print(results["documents"])

Chroma embeds the text for you and handles the index. Same concept, production ergonomics.

Chunking matters more than the model

You rarely embed whole documents. A 30-page PDF becomes one vector that's an average of everything and a good match for nothing. In practice you chunk documents into passages (a few hundred tokens, often with slight overlap) and embed each chunk. Get chunking wrong and even a great embedding model returns mush — which is one of the most common reasons retrieval systems quietly underperform. Chunking strategy, overlap, and metadata are exactly the unglamorous details that separate a demo from a dependable system, and they're covered in depth in a course on advanced LLM integration for production apps.

Choosing and swapping embedding models

Embedding models differ in dimensionality, speed, cost, and language coverage — and critically, you must embed your corpus and your queries with the same model. Change the model and you re-embed everything. For multilingual apps (Romanian included), pick a model with strong multilingual training rather than an English-first one, or your non-English recall will suffer silently. Public benchmarks like the MTEB leaderboard on Hugging Face are the sane starting point for comparing models on retrieval quality rather than vibes.

Hybrid search: when semantic alone isn't enough

Here's a lesson that surprises people: pure semantic search is worse than keyword search for certain queries. Ask for an exact product code, a specific error like ERR_CONN_REFUSED, a person's name, or an acronym, and embeddings can betray you — they match on meaning, and a precise identifier has little semantic meaning to spread around. The embedding for ERR_CONN_REFUSED sits near "connection problems" generally, so a document about a different connection error can outrank the exact match.

The production answer is hybrid search: run both a keyword search (classic lexical scoring like BM25) and a semantic search, then combine the rankings. Keyword search nails exact terms, identifiers, and rare words; semantic search nails paraphrase and intent. Together they cover each other's blind spots.

The standard way to merge the two result lists is Reciprocal Rank Fusion (RRF) — a simple, robust formula that scores each document by its rank in each list rather than by raw scores that live on incompatible scales:

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> dict[str, float]:
    """Fuse multiple ranked lists of doc IDs into one score per doc."""
    scores: dict[str, float] = {}
    for ranking in rankings:            # e.g. [keyword_results, semantic_results]
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return dict(sorted(scores.items(), key=lambda x: x[1], reverse=True))

Because RRF only looks at positions, you don't have to normalize BM25 scores against cosine similarities — a notorious apples-to-oranges trap. Most serious vector databases (Weaviate, Qdrant, and others) now offer hybrid search with RRF built in, precisely because "just embeddings" quietly underperforms on real, messy query logs. If you take one production lesson from this article, make it this: measure your recall on real queries, and reach for hybrid the moment exact-match queries show up.

Where embeddings show up beyond search

Semantic search is the gateway, but the same vectors power a lot more:

RAG (retrieval-augmented generation) — retrieve relevant chunks by embedding similarity, then feed them to an LLM as grounding context. Embeddings are the retrieval half; without good retrieval, generation hallucinates.
Deduplication & clustering — near-duplicate detection and topic clustering fall out of distances almost for free.
Recommendations — "items similar to this one" is a nearest-neighbor query in embedding space.
Classification — embed labeled examples, then classify new items by nearest neighbors, often without training a dedicated model.

The through-line: any time you need "similar in meaning" rather than "matches exactly," embeddings are the tool. Building these features end to end — API to product, with the retrieval and orchestration wired up properly — is the spine of a hands-on course on building AI applications in Python with the OpenAI and Anthropic SDKs.

Common mistakes that cost you hours

A few traps that catch almost everyone the first time:

Re-embedding the corpus on every query. Embed documents once, store the vectors, embed only the incoming query at search time.
Mixing models. Query vectors and document vectors must come from the same embedding model. A silent mismatch produces garbage rankings with no error.
Forgetting to normalize. If you compute raw dot products instead of cosine similarity (and your vectors aren't already unit-normalized), longer texts get an unfair boost. Normalize, or use a library that does.
Embedding documents that are too large. One vector per giant document averages meaning into uselessness. Chunk first.
Trusting a single similarity threshold forever. The "good enough" cutoff depends on your model and data. Measure it on real queries; don't hardcode 0.8 because a blog post said so.

Conclusion

Vector embeddings are one of those ideas that feels like magic until you see the mechanics — and then it's just geometry. Text becomes a point in space, meaning becomes distance, and search becomes "find the nearest points." You built exactly that in a few lines of Python, from raw NumPy cosine similarity to a Chroma-backed index, and the same core idea scales from a toy corpus to millions of documents behind an ANN index.

Start where we started: embed a handful of your own documents, run a query that shares no keywords with the right answer, and watch it surface anyway. Once that clicks, RAG, recommendations, and semantic features stop looking like separate topics and start looking like one tool applied five ways. If you want the structured path from here — retrieval, chunking, evaluation, and production wiring — Cursuri-AI.ro builds it step by step.

Sources & further reading:

Sentence-Transformers — Official documentation
Hugging Face — MTEB: Massive Text Embedding Benchmark leaderboard
pgvector — Open-source vector similarity search for Postgres
Chroma — Open-source embedding database

This article is educational content. Model names, dimensions, and library APIs evolve; verify current details in the official documentation before building production systems.

Context Engineering for AI Agents: Beyond Prompt Engineering

galian — Thu, 09 Jul 2026 13:08:24 +0000

You wrote a great prompt. It worked beautifully in the playground — one question, one clean answer. Then you wired the same model into an agent that runs twenty steps, calls six tools, and reads back their output, and somewhere around step twelve it started forgetting the goal, calling the wrong tool, or confidently acting on something it misread three steps ago. The prompt didn't get worse. The context did.

This is the gap that context engineering fills. Prompt engineering is about writing one good instruction. Context engineering is about managing the entire set of tokens a model sees at inference time — across a long, multi-step run — so the signal stays high and the model keeps making good decisions. Anthropic frames it as the natural progression of prompt engineering, and if you're building anything agentic in 2026, it's the discipline that separates a demo from a system.

What context engineering actually is

Start with a precise definition. Context is the full set of tokens you include when you sample from a large language model. Not just your prompt — the system instructions, the tool definitions, the examples, the running message history, the retrieved documents, the tool results fed back in. Everything in the window.

Context engineering is the set of strategies for curating and maintaining the optimal set of those tokens during inference. The goal, in one line: find the smallest set of high-signal tokens that reliably produces the outcome you want.

The reason this is a distinct discipline from prompt engineering is the shape of the problem. A prompt is something you write once and it stays put. Context in an agent is dynamic — it grows on every turn as the model reads files, calls tools, and accumulates history. You're not authoring a static string anymore; you're managing a budget that fills up on its own, and deciding continuously what earns a place in it and what gets thrown out. That's an engineering problem, and it's the foundation the whole prompt-to-production journey builds on.

Why "just add more context" is the wrong instinct

The intuitive move, when an agent makes a mistake, is to give it more: more instructions, more examples, more history, more retrieved documents. Sometimes that helps. Very often it makes things worse, and here's why.

A model's effective attention is a finite resource. Every token you add competes with every other token for the model's limited ability to attend to what matters. Past a certain point, adding context doesn't add capability — it dilutes it. The relevant fact is now buried among ten irrelevant ones, and the model attends to the wrong thing.

This shows up empirically. The "lost in the middle" effect — documented by Liu et al. — found that models attend most reliably to information at the start and end of a long context, and least reliably to what's stuck in the middle. As context grows, retrieval of any single fact inside it gets less reliable, a degradation sometimes called context rot. A 200K-token window does not mean you should put 200K tokens in it. Capacity is not the same as attention.

So the mental model to internalize: context is a budget, not a bucket. You're not trying to fill it. You're trying to spend it on the highest-signal tokens available and refuse everything else. Every technique below is a way to enforce that discipline.

The four things competing for your window

Before the techniques, know your spenders. In a running agent, four categories of tokens fight for the same finite budget:

The system prompt — your instructions, role, constraints. Usually small, high-value, and stable.
Tool definitions — the schemas describing every tool the agent can call. These are sneakily expensive: each tool's description sits in context on every turn, whether or not it's used.
Message history — the accumulating transcript of the conversation and the agent's own steps. This is the one that grows without bound and quietly eats the window.
Retrieved / external data — documents, search results, file contents, database rows pulled in to ground the model.

The overall guidance from Anthropic is worth memorizing: keep each of these informative yet tight. Not empty — an under-specified system prompt or a missing tool leaves the model guessing. But not bloated either. The art is the calibration, and it's different for each category. Let's work through the techniques that manage them.

Technique 1: Compaction — summarize the history before it drowns you

Message history is the runaway spender. A long agent run accumulates hundreds of turns of "called tool, got 4KB of JSON back, reasoned about it, called the next tool." Most of those raw tool outputs are dead weight three steps later — you needed the conclusion, not the 4KB.

Compaction is the fix: periodically replace a chunk of verbose history with a tight summary that preserves the decisions, the key facts, and the current state, while dropping the raw noise. When the agent has finished investigating something, you don't need the full transcript of the investigation in context — you need "here's what I found and what it means for the task."

Practical rules:

Compact at natural boundaries, not mid-reasoning — after a sub-task completes, when a phase ends.
Preserve the load-bearing details: open questions, decisions made, constraints discovered, current state. Drop the intermediate chatter and the raw dumps.
Keep the goal pinned. The single most common long-run failure is the agent losing the plot on the original objective. The goal should survive every compaction.

Done well, compaction is what lets an agent run for hundreds of steps without either overflowing its window or forgetting why it started.

Technique 2: External memory — let the agent write things down

The window is not the only place to keep information. The most effective long-running agents treat the context window as working memory and offload durable state to external memory — a file, a scratchpad, a structured store the agent reads from and writes to deliberately.

Instead of carrying every fact in-context forever, the agent writes a note ("the auth module uses JWT with 15-minute expiry; the bug is in the refresh path") to a persistent store, and pulls it back only when relevant. The context window stays lean; the knowledge doesn't get lost. This is exactly how a human engineer works — you don't hold the entire codebase in your head, you keep notes and open the file when you need it.

This pattern — persistent, deliberately-managed memory outside the window — is the core of building agents that hold state over long horizons, and it's what turns a stateless model into something that can work a problem across a session without drowning.

Technique 3: Sub-agents — isolate context so the main thread stays clean

Here's a structural technique that most people never turn on. When a sub-task is going to burn a lot of tokens — "find every call site of this function across the repo," "research these five libraries" — don't do it in the main agent's context. Delegate it to a sub-agent: a separate agent instance with its own isolated window that does the noisy work and reports back a clean result.

The win is context hygiene. The thousands of tokens of file contents and search output that the investigation churns through stay in the sub-agent's context and die with it. The main agent gets back a two-paragraph summary, and its own window stays focused on the actual task instead of silting up with intermediate noise. As a bonus, independent sub-tasks run in parallel instead of serially.

The rule of thumb: delegate when work is independent, parallelizable, or context-heavy; keep it inline when it's sequential and cheap. Knowing when to fan out to sub-agents and how to orchestrate them without stepping on each other is a central skill in building AI agents and automation, and it's one of the highest-leverage context moves you have.

Technique 4: Just-in-time retrieval — pull data when needed, not upfront

There are two ways to get external data into an agent. The naive way is to preload: at the start, dump everything the agent might conceivably need into context — the whole document, all the schemas, every config file. The problem is obvious once you see it: you're spending your budget on maybes, and most of it goes unused while crowding out what matters.

The better pattern is just-in-time retrieval: give the agent the ability to fetch data (a search tool, a file reader, a database query) and let it pull exactly what it needs, exactly when it needs it. Instead of "here are all 40 files," it's "here's a tool to read a file — go get the one you need." The agent loads the relevant chunk into context at the moment of use, acts on it, and (with compaction) lets it fall away afterward.

This mirrors how retrieval-augmented systems already work, and getting the retrieval layer right — what to fetch, how to rank it, how much to bring back — is where advanced LLM integration earns its keep. Preloading feels safer; just-in-time is what actually scales.

Technique 5: Tool curation — the failure mode hiding in plain sight

Remember that tool definitions sit in context on every turn. That makes a bloated tool set a double tax: it burns tokens continuously, and it degrades decisions. Anthropic calls out one of the most common failure modes directly — tool sets that cover too much functionality or create ambiguous, overlapping choices about which tool to use.

The tell is a sharp one: if a human engineer can't say for certain which tool should be used in a given situation, an AI agent can't either. Fifteen tools with fuzzy, overlapping responsibilities will produce worse behavior than six sharp, non-overlapping ones — and cost more tokens doing it.

So curate the toolbox like you'd curate an API:

Fewer, sharper tools. Each with a clear, distinct job and an unambiguous "use this when…"
No overlap. Two tools that could both plausibly handle the same request is a decision point where the agent will sometimes pick wrong.
Prune ruthlessly. A tool that's rarely the right choice is paying rent in your context on every single turn. Cut it.

Tool curation is the least glamorous technique here and often the highest-ROI. It's pure subtraction, and subtraction is exactly what context engineering rewards.

How you know it's working: measure it

Every technique above is a change to your context, and changes to context are exactly the kind of thing that feels better while being worse — or vice versa. If your only signal is "I ran a few queries and it seemed fine," you're tuning blind, and you'll ship a regression the day you compact one turn too aggressively and the agent starts forgetting a constraint.

The fix is the same as anywhere in production ML: an evaluation set. Assemble representative tasks with known good outcomes, and re-run them every time you change how context is managed. Then you can say "compaction at phase boundaries held task success at 0.9 while cutting average tokens 40%" instead of "I think it's better now." Treating evaluation as first-class rather than an afterthought is what turns context engineering from a craft into engineering — a number that moves when you change something, not a vibe.

Putting it together

Context engineering isn't a framework you install; it's a posture you adopt toward the model's window. The whole discipline collapses to one principle applied relentlessly: spend the finite budget on the smallest set of high-signal tokens that does the job.

In practice, for a real agent, that means:

Compact the history at natural boundaries so a long run doesn't drown in its own transcript.
Offload durable state to external memory instead of carrying it forever.
Delegate noisy, independent work to sub-agents so the main window stays clean.
Retrieve just-in-time instead of preloading everything you might need.
Curate the tools hard — fewer, sharper, no overlap.
Measure with evals so every change is verified, not hoped.

None of these are exotic, and that's the point. The model is already capable. What makes the capability hold up over a twenty-step run isn't a cleverer prompt — it's disciplined management of everything the model reads along the way.

Conclusion

Prompt engineering taught us to write one good instruction. Context engineering is what you need the moment that instruction has to survive a long, tool-using, self-accumulating agent run — which is to say, the moment you build anything real. The failure you saw at step twelve was never the model getting dumber. It was the context getting noisier, and no one curating it.

Adopt the budget mindset, apply the five techniques, and put an eval set behind every change. Do that, and the same model that fell apart at step twelve will run to step fifty and still know exactly what it's doing — because you engineered what it was looking at the whole way.

The courses linked throughout are part of Cursuri-AI.ro, an AI-learning platform with hands-on, current tracks on context engineering, AI agents, LLM integration, and evaluating AI systems in production.

Sources & further reading:

Anthropic — Effective context engineering for AI agents (definition, the four context components, tool-set failure modes, compaction and memory)
Anthropic — Effective harnesses for long-running agents
Liu et al. — Lost in the Middle: How Language Models Use Long Contexts

This article is educational content. Model behavior, context limits, and tooling evolve quickly; validate approaches against your own workloads and current official documentation.

Run LLMs Locally with Ollama in 2026: The Practical Developer Guide

galian — Sun, 05 Jul 2026 21:11:24 +0000

For years, "run the model locally" was the option you mentioned and then didn't take: the models were too weak, the tooling too fiddly, and the cloud APIs too convenient. In 2026 that calculus has genuinely shifted. Open-weight models in the 12–35B range now handle real coding and agent workloads, Apple Silicon got a dedicated inference engine, and Ollama quietly became a drop-in backend for the same tools you already use against cloud APIs — including Claude Code.

I teach practical AI engineering at Cursuri-AI.ro, Eastern Europe's AI education platform, and local inference has gone from a curiosity module to one of the questions companies ask us most — usually spelled "how do we use LLMs without sending our data anywhere?" This guide is the answer I give developers: what changed, what hardware you actually need, which models are worth pulling, and how to plug it all into a real workflow.

As always with this space: versions and model rankings move monthly. Everything below is verified against Ollama's official blog and docs as of early July 2026 — re-check before you build a budget or an architecture on it.

Why local, and why now

Three arguments have survived contact with production; the rest is mostly vibes.

Privacy and data residency. With a local model, prompts and outputs never leave your machine (or your VPC, if you self-host on a server). For anyone dealing with client data, medical text, legal documents, or GDPR-sensitive workloads, this eliminates the entire "what does the provider do with my data" conversation instead of managing it through contracts. This is the single biggest adoption driver we see in Europe, and it's the backbone of our course on local LLMs, self-hosting, and privacy.

Cost shape. Cloud APIs bill per token; local inference bills you once, in hardware you may already own. For high-volume, latency-tolerant workloads — batch classification, summarization pipelines, internal tooling — a mid-range GPU that's already on someone's desk can absorb work that would otherwise be a real monthly line item. (For low-volume or frontier-quality work, cloud still wins. More on that below.)

No external dependency. A local model doesn't get deprecated, rate-limited, price-changed, or suspended out from under you. After the model-availability surprises of the last year, "at least one workload runs on weights we control" has become a reasonable line item in a resilience plan, not paranoia.

What actually changed in Ollama in 2026

If you last touched Ollama when it was "a nice wrapper around llama.cpp," the 2026 releases are the reason to look again. All of this is from Ollama's official blog:

Anthropic API compatibility (January, v0.14.0). Ollama now exposes a native Anthropic-style /v1/messages endpoint. This is the sleeper feature of the year: Anthropic-native tools — most notably Claude Code — can talk to a local model directly, with no proxy or translation layer. There's a matching OpenAI-compatible endpoint too, so Codex and OpenAI-SDK apps work the same way.
ollama launch (January). A single command that configures and starts a coding agent against a local model — ollama launch claude sets up Claude Code, prompts you to pick a model, and you're in.
Experimental image generation (January). Early days, but the scope of "local model" is no longer text-only.
MLX engine on Apple Silicon (March preview → June release). Ollama moved its Mac inference path to Apple's MLX framework, which exploits unified memory. Ollama's own framing for the June release: its highest performance on Apple Silicon yet — faster output with reduced memory usage.
Ollama 0.30 and 0.31 (June). Version 0.30 brought improved performance and broader GGUF model compatibility through llama.cpp; 0.31 made Gemma 4 significantly faster on Apple Silicon via multi-token prediction (MTP), enabled by default.

The theme is clear: Ollama is positioning itself less as a hobbyist toy and more as the standard local backend for agentic tooling.

Getting started in five minutes

Install (macOS and Windows have installers at ollama.com/download; on Linux):

curl -fsSL https://ollama.com/install.sh | sh

Pull and run a model:

ollama pull gemma4
ollama run gemma4

That's a working local chat. Ollama also starts a local server on port 11434, which is where the interesting part begins — every API-based tool you have can point at it.

Useful daily commands: ollama ls (installed models), ollama ps (what's loaded and where — CPU vs GPU), ollama rm <model> (free disk space; models are multi-gigabyte).

Hardware: the honest sizing guide

The rule of thumb that matters: a model quantized to 4 bits needs very roughly 0.5–0.7 GB of memory per billion parameters, plus overhead for context. Everything else follows from that.

Your hardware	What runs comfortably	Experience
8 GB RAM, no GPU	3–8B models, quantized	Fine for chat, drafting, classification. Slow but usable on CPU
16 GB RAM (Apple Silicon)	8–14B models	Good daily-driver territory; MLX made this tier notably faster in 2026
24 GB+ (M-series Pro/Max or a 24 GB GPU)	27–35B models	Where local coding models get genuinely useful
48 GB+ unified memory / multi-GPU	Large MoE models	Server-class local inference

Two nuances that save people disappointment:

Quantization is why any of this works. Models ship in compressed 4–8 bit variants (the GGUF ecosystem) that trade a small quality loss for a 2–4× memory reduction. Ollama's default tags are already quantized — you rarely need to think about it, but it explains why a "27B model" fits in 24 GB.
Mixture-of-experts (MoE) models need memory for their total parameters but compute like their active subset. NVIDIA's Nemotron-3-Super, for example, is a 120B model with 12B active parameters: it runs faster than its size suggests, but you still need the RAM to hold it.

Context length eats memory too — an agent session with 32K+ tokens of context adds real overhead on top of the weights. If you're sizing for coding agents, budget for that, not just the model file.

The mid-2026 open-weight lineup worth knowing

Rankings churn monthly, so treat this as a map, not a leaderboard. From Ollama's model library, the families that matter right now:

Gemma 4 (12B–31B) — Google's open family, currently the most-pulled model on Ollama. Multimodal, tuned for reasoning and agentic work, and the main beneficiary of the MLX/MTP speedups on Macs.
Qwen3.5 / Qwen3.6 (0.8B–122B) — the ecosystem's Swiss army knife. Qwen3.5 spans everything from edge-tiny to server-large; Qwen3.6 (27B–35B) focuses on agentic coding. qwen3-coder is Ollama's own recommendation for coding-agent use.
GLM-5 family — flagship-class open weights (GLM-5 is 744B total / 40B active); strong at coding and long-horizon tasks. Too big for most desktops locally, but available as :cloud variants (see below) and self-hostable on serious hardware.
Nemotron-3-Super (120B MoE, 12B active) — NVIDIA's entry, aimed at multi-agent applications.
MiniMax-M3 — notable for a 1M-token context window, if your workload is long-document analysis.
Specialists: GLM-OCR for document understanding, TranslateGemma (4B–27B, 55 languages) for translation, LFM2 (24B) for on-device deployment, Ornith (9B–35B) for agentic coding.

Sensible defaults: on a 16 GB Mac, start with gemma4:12b. On 24 GB+, try qwen3-coder for code and gemma4:27b for general work. Then run your tasks on them — a model's rank on someone's benchmark tells you little about your use case.

The part that changes your workflow: Ollama as a drop-in API

Ollama's server speaks both major API dialects, which means "switch to a local model" is now a base-URL change, not a rewrite.

OpenAI-compatible (/v1) — any OpenAI-SDK app works:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resp = client.chat.completions.create(
    model="gemma4",
    messages=[{"role": "user", "content": "Explain GGUF quantization in one paragraph."}],
)
print(resp.choices[0].message.content)

Anthropic-compatible (/v1/messages) — and this is the one with teeth, because it means Claude Code runs against local models. Per Ollama's official docs:

export ANTHROPIC_AUTH_TOKEN=ollama       # accepted but not validated
export ANTHROPIC_BASE_URL=http://localhost:11434

claude --model qwen3-coder

Or let Ollama do the wiring for you:

ollama launch claude

Honest caveats before you get excited:

Ollama recommends at least 32K tokens of context for Claude Code, and its model suggestions for coding are qwen3-coder locally (30B — you want 24 GB+ of VRAM/unified memory) or glm-4.7:cloud / minimax-m2.1:cloud via Ollama's cloud, which keeps the same API surface but runs the weights remotely.
The compatibility layer doesn't cover everything: no prompt caching, no token-counting endpoint, no forced tool selection, no batches API, no PDF inputs (images must be base64). If your workflow leans on those, you'll feel it.
A 30B local model is not Opus, and it isn't trying to be. It's "capable pair of hands on an airplane / on confidential code," not "frontier reasoning."

The pattern that actually works in practice is routing: local models for the private, high-volume, or offline work; frontier cloud models for the hard reasoning. Deciding which tier a task belongs to — and building the escalation path — is an architecture skill, and it's exactly the kind of decision we drill in our AI system architecture course.

When local is the wrong choice

Being a fan of local inference means knowing where it loses:

Frontier-quality reasoning. For the hardest tasks, top cloud models remain clearly ahead of anything you can run on a workstation. If wrong answers are expensive, don't fight this.
Low-volume workloads. If you make a few thousand API calls a month, per-token billing is cheaper than any GPU. Local pays off at volume, at privacy constraints, or at both.
Ops you don't want. A self-hosted model is a service you now run: updates, monitoring, capacity. ollama run on a laptop is trivial; a team-wide inference server is real infrastructure.
Multimodal breadth and long-tail capabilities. Cloud APIs still bundle more (native PDF understanding, larger tool ecosystems, batch APIs) than the local stack replicates.

One more thing people conflate: running a model locally is different from customizing one. If your actual goal is a model that speaks your domain language or follows your house style, that's a fine-tuning question — LoRA adapters on an open-weight base, then serving the result through Ollama. That pipeline (when to fine-tune vs when to just engineer the prompt) is its own discipline, covered in our fine-tuning course.

Frequently asked questions

Is Ollama free?

The tool itself is open source and free. The models each carry their own licenses — Gemma, Qwen, GLM and friends have different terms, some with restrictions on commercial use. Check the license tab on the model's Ollama page before you ship a product on it. Ollama's optional cloud models are a paid service.

What hardware do I need to run LLMs locally in 2026?

As a rule of thumb at 4-bit quantization: 8 GB of RAM runs 3–8B models, 16 GB runs 8–14B comfortably (especially on Apple Silicon with the MLX engine), and 24 GB+ opens up the 27–35B class where local coding models get genuinely useful. More context = more memory on top of the weights.

Can a local model replace GPT or Claude?

For a growing set of tasks — summarization, classification, drafting, routine coding on mid-size codebases — yes, credibly. For frontier reasoning and the highest-stakes accuracy, no. Production teams typically route: local for private/high-volume work, cloud for the hard 10%.

Can I really use Claude Code with Ollama?

Yes. Since Ollama v0.14.0 (January 2026) there's native Anthropic Messages API compatibility: set ANTHROPIC_BASE_URL=http://localhost:11434, run claude --model qwen3-coder — or just ollama launch claude. Expect a capable assistant, not Opus-level reasoning, and note that prompt caching and a few other API features aren't supported through the compatibility layer.

Ollama vs llama.cpp vs vLLM — which should I use?

Ollama for developer experience: one command, model management, dual API compatibility. llama.cpp (which powers Ollama's GGUF path) for maximum control and minimal footprint. vLLM for high-throughput multi-user serving on server GPUs. Most developers should start with Ollama and only move down the stack when they hit a concrete limit.

The skill underneath the tool

Here's the uncomfortable part: pulling a model is the easy 5%. The value shows up when you can answer the questions around it — which model for which task, how to measure whether the local model is good enough for your workload instead of guessing, how to build the routing and fallback so privacy-sensitive work stays local while hard problems escalate to a frontier model. That's engineering judgment, not tooling trivia.

That judgment is what we teach at our AI education platform — hands-on courses built around real repositories and an interactive AI instructor, covering the full local-to-cloud spectrum: self-hosting and privacy, fine-tuning, architecture, and the agentic workflow on top.

Conclusion

In 2026, local LLMs crossed the line from hobby to infrastructure option. Ollama's dual API compatibility means your existing tools — including Claude Code — can run against open weights with a base-URL change; the MLX engine made a 16 GB MacBook a legitimate inference machine; and the open-weight lineup in the 12–35B range is good enough for a real slice of production work.

The play isn't "cancel your API keys." It's knowing which slice of your workload belongs on weights you control — then running it there deliberately, measured, with an escalation path for everything else. Start with ollama run gemma4 tonight; you're one evening away from having an informed opinion.

Written by the team at Cursuri-AI.ro — practical, hands-on AI engineering courses for developers and professionals across Eastern Europe, from local LLMs and self-hosting to agentic coding, evals, and AI system architecture.

Sources: Ollama Blog · Anthropic API compatibility — Ollama Docs · Claude Code with Anthropic API compatibility — Ollama · Ollama Model Library · Download Ollama

DEV Community: Cursuri AI

The Batch API Halves Your Claude Bill — Unless Your Prompt Cache Was Already Doing the Work

What "50% off" actually applies to

A worked example: 10,000 classifications against a 50K prefix

The break-even, for any workload

Raising the batch hit rate: the documented recipe

Running it in production without losing results

custom_id is your idempotency key

Chunk by count and by bytes

Validate one request synchronously first

Submit with an intent marker

Poll gently, and don't expect a progress bar

Four result types, and "succeeded" isn't the same as usable

Re-submit only what failed

The clocks: 24 hours, 29 days

Where batch is the right tool

The checklist

FAQ

Want to go deeper?

Claude Opus 5.5 Migration: Four Errors You'll See and Three You Won't

What changed, in one table

The four errors you'll see

1. "thinking.type.disabled" is not supported for this model

2. tool_choice: type "tool" and "any" are not supported for this model

3. 'claude-opus-5-5' does not support tool types: computer_20251124.

4. Invalid `signature` in `thinking` block

The three changes you won't see

1. The default effort dropped a level, and the benchmarks didn't

2. Your agent's progress messages went quiet

3. Your unattended agent stops halfway

One more non-error: refusals have new categories

The money, honestly

Opus 5.5 or Fable 5.1?

The migration checklist

FAQ

The bottom line

The Model in Your Config Has an Expiry Date — a Runbook for LLM Deprecations

The vocabulary you're missing two-thirds of

The dates are published before the deprecation, and almost nobody reads them

Aliases are a strategy, not a shield

It is never just the model string

The runbook

1. Inventory: find every model string you own

2. Centralize the strings

3. A canary that fails loudly and cheaply

4. The eval gate, built before you need it

5. Migrate the request, then the prompt, then the model

What the provider actually owes you, and what it doesn't

The checklist

Your LLM Stream Works on localhost and Dies in Production — Fixing SSE End to End

First: streaming is not a UX nicety anymore

Break #1: your proxy is holding the tokens hostage

Break #2: the load balancer kills you during the pause

Break #3: the errors arrive with HTTP 200

Break #4: the user left and you're still paying

Break #5: EventSource can't do what your chat app needs

Break #6: partial output is a state you have to design for

The checklist

Anthropic Filed to Go Public. Here's What Actually Changes for Developers Building on Claude

First, the numbers that are actually confirmed

Consequence #1: your vendor gets a quarterly clock

Consequence #2: model retirement becomes an operational risk, not a footnote

Consequence #3: the S-1 will be the best vendor due-diligence document ever published

Consequence #4: "which model" stops being a taste question

What I'd actually do this quarter

The bottom line

The EU AI Act's Transparency Rules Went Live — Here's What You Actually Have to Ship

First: does this apply to you?

Obligation 1 — tell people they're talking to an AI

Obligation 2 — mark synthetic output so machines can detect it

Obligation 3 — emotion recognition and biometric categorisation

Obligation 4 — deepfakes and public-interest text

The deadlines, which are not all the same

A ship checklist

Conclusion

Claude Opus 5 Is Out — Here Are the API Changes That Will Actually Break Your Code

The spec sheet, in one table

Change #1: thinking is on by default

Change #2: the actual breaking change — disabled thinking + high effort = 400

Change #3: effort is now the control lever that matters

`custom_id` is your idempotency key

1. `"thinking.type.disabled" is not supported for this model`

2. `tool_choice: type "tool" and "any" are not supported for this model`

3. `'claude-opus-5-5' does not support tool types: computer_20251124.`

Break #5: `EventSource` can't do what your chat app needs

Change #3: `effort` is now the control lever that matters