DEV Community: galian

The Model in Your Config Has an Expiry Date — a Runbook for LLM Deprecations

galian — Mon, 31 Aug 2026 22:05:44 +0000

There is a category of production incident that has no bug, no bad deploy, and no root cause inside your codebase. Your code is byte-identical to the version that worked yesterday. Your tests pass. Your dependencies are pinned. And a feature is returning 404s.

The model string in your config refers to something that no longer exists.

This is the strangest failure mode in the whole LLM stack, because it is the only one that arrives entirely on somebody else's schedule — and the only one you were told about, in writing, months in advance, in an email nobody on the on-call rotation read.

I teach AI engineering at Cursuri-AI.ro, and model retirement is the incident I see teams handle worst, not because it's hard, but because it doesn't feel like engineering work until the day it does. Here's the whole thing: what actually breaks, what the provider tells you and when, and the runbook that turns a surprise outage into a boring Tuesday ticket.

The vocabulary you're missing two-thirds of

Most teams have two mental states for a model: "it works" and "it's old." Anthropic's model deprecations page defines four, and the gap between them is where the incident lives:

Active — fully supported, recommended.
Legacy — no longer receiving updates, may be deprecated in future.
Deprecated — still functional, no longer recommended, has an assigned retirement date.
Retired — gone. Requests fail.

The important one is deprecated. A deprecated model works perfectly. Latency is normal, quality is normal, your dashboards are green. Nothing in the response payload tells you a clock is running. The only signal is a documentation page and an email to the account owner — who is usually in finance.

And "retired" means what it says. The model ID stops resolving, and the API answers the way it answers any unknown model ID: HTTP 404, not_found_error, whose documented cause is "invalid endpoint or model ID." Your retry logic will not help you, because 404 is not retryable. If your error handling catches one broad exception class and retries everything, you'll spend the outage generating three times the traffic and no useful log lines.

import anthropic

client = anthropic.Anthropic()

try:
    response = client.messages.create(
        model=settings.CHAT_MODEL,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
except anthropic.NotFoundError:
    # This is not a blip. This is a retired or misspelled model ID.
    # Page someone. Do not retry.
    raise
except anthropic.RateLimitError:
    ...
except anthropic.APIConnectionError:
    ...

That first except block is maybe ninety seconds of work, and it's the difference between an alert that says "the model is gone" and an alert that says "elevated error rate, service degraded."

The dates are published before the deprecation, and almost nobody reads them

Here is the part that genuinely surprises people, and it's the single most useful fact in this article.

Anthropic publishes tentative retirement dates for models that are currently active. Not deprecated — active. As of the end of August 2026, the model status table reads, in part:

Model	State	Earliest retirement
`claude-sonnet-4-5-20250929`	Active	Not sooner than September 29, 2026
`claude-haiku-4-5-20251001`	Active	Not sooner than October 15, 2026
`claude-opus-4-5-20251101`	Active	Not sooner than November 24, 2026
`claude-opus-4-6`	Active	Not sooner than February 5, 2027
`claude-opus-5`	Active	Not sooner than July 24, 2027
`claude-fable-5`	Active	Not sooner than June 9, 2027

Read the first row again. If you are running claude-sonnet-4-5-20250929 in production today, its published floor is roughly four weeks out. That is not a retirement announcement — "not sooner than" is a floor, not a date, and no deprecation has been announced for it. But it is a planning input you already have, for free, and it costs you nothing to put in a calendar.

The counterpart is the models that already went: claude-opus-4-1-20250805 was deprecated on June 5, 2026 and retired on August 5, 2026 — three weeks before this article. claude-opus-4-20250514 and claude-sonnet-4-20250514 were deprecated April 14, 2026 and retired June 15, 2026.

Do the arithmetic on those pairs: 61 days, and 62 days. Anthropic commits to "at least 60 days' notice before model retirement for publicly released models," and in practice the notice window is exactly that. Sixty days is enough time to run a migration. It is not enough time to build an eval suite, get budget approval, and negotiate with a team that owns the prompt. That work has to already exist when the email lands.

Aliases are a strategy, not a shield

Anthropic exposes two shapes of model ID: aliases like claude-opus-5 or claude-sonnet-4-6, and dated snapshots like claude-sonnet-4-5-20250929. Note that the current generation are complete as aliases — claude-opus-5 is the whole ID, and appending a date suffix to it produces a 404 as surely as a retired model does.

Neither shape saves you:

A dated snapshot is reproducible and will never change behavior under you — and it is precisely what appears in the retirement table, by name, with a date.
An alias may keep resolving across point releases, but the alias is scoped to a model family. When the family goes, the alias goes. And an alias that silently starts pointing somewhere new is its own kind of incident: your outputs change, your evals shift, and nothing in your diff explains it.

The real answer isn't picking one. It's picking one per route, deliberately, and writing down why. A billing-adjacent classifier that must produce identical output for audit reasons wants a snapshot and a calendar entry. A chat surface where quality improvements are welcome wants an alias and an eval suite that runs on a schedule. What you must not have is thirty model strings scattered across a codebase, each chosen by whoever wrote that file, with no owner.

It is never just the model string

This is the trap that turns a one-line change into a two-week migration, and it's why "we'll swap it when we have to" fails.

Request parameters retire too. On Claude Opus 4.7 and later, temperature, top_p and top_k return a 400 error when set to a non-default value — the recommended replacement is to omit them and steer behavior through prompting. And it's not only the API: the Python SDK v1.0 and later removes those parameters from the request types, so passing them raises a TypeError before a request is ever made.

The same applies across the current generation: the fixed thinking budget (thinking: {type: "enabled", budget_tokens: N}) is gone in favor of adaptive thinking plus an effort level, and assistant-message prefill — the classic trick for forcing a response format — returns a 400 on the whole 4.6-and-later family. If your "quick model swap" touches code written against a 2025-era API, you are not changing a string. You are rewriting the request.

# Written for an older model. Every line here is now a 400 or a TypeError.
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",   # retired October 28, 2025
    max_tokens=4096,
    temperature=0.2,                       # 400 on Opus 4.7+ (non-default)
    thinking={"type": "enabled", "budget_tokens": 8000},   # removed
    messages=[
        {"role": "user", "content": prompt},
        {"role": "assistant", "content": "{"},             # prefill: 400
    ],
)

# The current shape.
response = client.messages.create(
    model="claude-opus-5",
    max_tokens=16000,
    thinking={"type": "adaptive"},
    output_config={
        "effort": "high",
        "format": {"type": "json_schema", "schema": INVOICE_SCHEMA},
    },
    messages=[{"role": "user", "content": prompt}],
)

And prompts are part of the migration. This is the least mechanical and most expensive piece. Prompts tuned against an older model are frequently over-prescriptive for a newer one — step-by-step scaffolding that raised quality two generations ago can now suppress it. A migration that swaps the ID, passes your smoke test, and ships is a migration that quietly degraded output quality in a way you'll attribute to something else three weeks later. This is exactly the failure mode that makes an evaluation suite the load-bearing part of the runbook rather than the optional part.

The runbook

Five steps. The first three are cheap and you should do them this week, whether or not anything of yours is deprecated.

1. Inventory: find every model string you own

You cannot plan a migration you can't enumerate. Two passes:

# Every model ID that lives in code, config, notebooks, IaC, or a CI secret file.
rg -n --hidden -g '!node_modules' -g '!.git' \
   'claude-(fable|mythos|opus|sonnet|haiku)-[0-9a-z.\-]*|claude-[0-9]' .

Then the pass that matters more, because it catches the jobs nobody remembers: in the Claude Console, open Usage → Export, and read the CSV. It breaks usage down by API key and by model, which means it shows you the nightly batch job in a repo you don't own, running under a key that was issued in 2025. That report is the actual inventory. The grep is just the part you control.

The output of this step is a single table — route, model ID, owner, alias-or-snapshot, why. Keep it in the repo, not in a wiki nobody opens.

2. Centralize the strings

Every model ID resolves through one module. Not because indirection is beautiful, but because a migration should be one reviewable diff, and because it gives you a place to hang the metadata:

# models.py — the only file in the repo that contains a model ID.
from dataclasses import dataclass

@dataclass(frozen=True)
class ModelChoice:
    id: str
    owner: str          # who approves a change
    pinned_reason: str  # why this shape, not the other one
    review_by: str      # ISO date, from the published retirement floor

CHAT       = ModelChoice("claude-opus-5", "platform", "quality-led surface, alias by choice", "2027-07-24")
CLASSIFIER = ModelChoice("claude-haiku-4-5", "platform", "high volume, latency-sensitive", "2026-10-15")

review_by is not a guess. It is the published "not sooner than" date, copied in. Now your calendar and your code agree.

3. A canary that fails loudly and cheaply

A retirement should be caught by a scheduled job, not by a user. This is about eight lines and costs a fraction of a cent per run:

# canary.py — run daily in CI. Alerts on the one error that is never transient.
import anthropic, sys
from models import CHAT, CLASSIFIER

client = anthropic.Anthropic()
dead = []

for choice in (CHAT, CLASSIFIER):
    try:
        client.messages.create(
            model=choice.id,
            max_tokens=1,
            messages=[{"role": "user", "content": "ping"}],
        )
    except anthropic.NotFoundError:
        dead.append(choice)

if dead:
    for choice in dead:
        print(f"MODEL GONE: {choice.id} (owner: {choice.owner})", file=sys.stderr)
    sys.exit(1)

For the inventory half, client.models.list() gives you what your key can currently reach, with id, display_name, max_input_tokens, max_tokens and a capabilities tree — useful for catching the day a model stops appearing, and for asserting that a replacement actually supports the features your route depends on before you switch to it.

4. The eval gate, built before you need it

Sixty days is plenty of time to change a string and not nearly enough to answer "is the new one better on our task?" if you're starting from zero. The gate needs three things and they take an afternoon each:

A frozen set of real inputs. Fifty to two hundred requests sampled from production, with the messy ones deliberately over-represented. Not synthetic examples.
A grader that isn't a vibe. Exact match where you can get it, schema validation where the output is structured, an LLM judge with a written rubric where you can't. The judge is fine — an unwritten rubric is not.
A cost and latency baseline. Because the honest outcome of a migration is sometimes "quality held, p95 doubled," and you want to know that before your users do.

Run it against the old model and the new one on the same inputs, same day. The comparison is the deliverable, and it's the artifact that lets you say "yes, ship it" in an afternoon instead of a fortnight. If you're standing this up for the first time, this is the same machinery you need for every other change you'll ever make to an LLM feature — building it properly once pays for itself on the second migration.

5. Migrate the request, then the prompt, then the model

Order matters, because it isolates variables:

Modernize the request shape on the model you're already running — remove sampling parameters, replace fixed thinking budgets with adaptive thinking plus an effort level, replace prefill with structured outputs. Ship it. Nothing about behavior should change, and if something does, you've learned it in isolation.
Swap the model ID behind a flag, run the eval gate, compare.
Re-tune the prompt for the new model — usually by deleting scaffolding rather than adding it — and re-run the gate.
Roll out by percentage, keeping the old model reachable until its retirement date, not until your rollout finishes.

Doing all four at once produces a result you can't attribute. Doing them in order takes the same total effort and tells you which change did what.

What the provider actually owes you, and what it doesn't

Worth being clear-eyed here, because the answer is better than the ecosystem average and still not a guarantee.

You get: at least 60 days' notice before retirement of a publicly released model, email to affected accounts with active deployments, published tentative floors for active models, a recommended replacement for every deprecated model, and a self-service usage audit. Anthropic has also published commitments on model deprecation and preservation, including long-term preservation of model weights, and states openly that retirement exists to free capacity — with real downsides for people who depended on a specific model's behavior.

You don't get: a promise that the replacement behaves like the old one on your task. That has never been promised by anyone, and it's the only part that requires actual work from you.

One more thing worth knowing if you're multi-cloud: those dates apply to Anthropic-operated platforms — the Claude API, Claude Platform on AWS, and Microsoft Foundry. Amazon Bedrock and Google Cloud set their own retirement schedules, so a model's status and dates can differ there. If your failover path routes to a different platform, that path has its own calendar, and it is not this one.

The checklist

Print this. It's the whole article.

[ ] Every model ID in the codebase resolves through one module.
[ ] That module records, per route: owner, alias-or-snapshot, and the reason.
[ ] The published "not sooner than" date for each model is in the code and in a shared calendar.
[ ] NotFoundError / HTTP 404 is caught separately and pages a human. It is never retried.
[ ] A daily canary calls every configured model with max_tokens=1 and fails the build when one is gone.
[ ] The Console usage export has been read at least once this quarter, by a human, looking for keys and models nobody claims.
[ ] An eval suite with real inputs and a written rubric exists today, not on announcement day.
[ ] The request shape is modernized independently of the model swap.
[ ] Prompt re-tuning is scheduled as part of every migration, not treated as optional.
[ ] Anyone routing through Bedrock or Vertex has checked those platforms' separate schedules.

None of this is clever. All of it is the difference between a migration you schedule and an outage you explain.

I build and teach this stack at Cursuri-AI.ro — including taking LLM features to production and shipping a full AI product end to end. If your team has a model string you can't account for, the inventory step is genuinely the highest-value hour you'll spend this month.

Your LLM Stream Works on localhost and Dies in Production — Fixing SSE End to End

galian — Sat, 22 Aug 2026 11:39:18 +0000

Streaming an LLM response looks like the easiest feature in your product. The SDK gives you an iterator, you print tokens, it works on your laptop in under a minute.

Then you deploy it, and one of these happens: the response arrives all at once after 40 seconds instead of token by token. Or the connection drops mid-answer with no error anywhere in your logs. Or a user closes the tab and you keep paying for 60,000 tokens nobody will ever read.

None of those are model problems. They're the six things that break in the space between api.anthropic.com and a browser tab — a proxy, a load balancer, an HTTP client, and a JavaScript API that each have opinions about long-lived responses.

I teach AI engineering at Cursuri-AI.ro, an AI education platform in Eastern Europe, and this is the failure surface I see most often in production reviews — because every layer of it looks fine in isolation. Here's the whole path, one break at a time.

First: streaming is not a UX nicety anymore

It's worth being clear about why you're doing this, because it changes how you treat failures.

The perceived-latency argument is the famous one, and it's real. But on current models, streaming is also a correctness requirement for a growing share of requests. Claude Opus 5, Sonnet 5, and the 4.6/4.7/4.8 family support up to 128K output tokens, and Anthropic's SDKs will refuse a non-streaming request they estimate will exceed the connection's tolerance — the Python SDK raises a ValueError rather than let you build something that hangs and drops. The default client timeout is 10 minutes (note the units differ by SDK: seconds in Python and Ruby, milliseconds in TypeScript), and with thinking on by default on Opus 5, a hard task can spend minutes generating before the first visible character.

So: any request with a large max_tokens, a long input, or a reasoning-heavy prompt is a streaming request. Not for the animation — for the connection.

import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model="claude-opus-5",
    max_tokens=64000,
    thinking={"type": "adaptive", "display": "summarized"},
    messages=[{"role": "user", "content": "Analyze this incident report..."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

    final = stream.get_final_message()

One thing in that snippet is not cosmetic. On Opus 5, Opus 4.8/4.7, Fable 5, and Sonnet 5, the thinking display default is "omitted" — the thinking block opens, emits a single signature_delta, and closes, with no thinking_delta events. If you stream reasoning to users and leave the default in place, your UI shows a long dead pause and then a wall of text. display: "summarized" is what gives you something to render during the think. It costs nothing extra: thinking happens and is billed identically under every display setting.

Break #1: your proxy is holding the tokens hostage

This is the number-one "streaming doesn't work in production" bug, and the tell is unmistakable: locally you see tokens appear one by one; deployed, the whole response lands at once at the end.

Nothing in your app is wrong. nginx is buffering. From the nginx documentation, proxy_buffering defaults to on, and when buffering is enabled nginx reads the response from the upstream into its own buffers before passing it along. When it's off, "the response is passed to a client synchronously, immediately as it is received."

There are two ways to fix it, and you want the second one.

location /api/chat/stream {
    proxy_pass http://app;
    proxy_buffering off;
    proxy_read_timeout 300s;   # default is 60s
}

That works, but it puts a per-route infrastructure rule in a file your application team doesn't own. nginx also honors a response header: "Buffering can also be enabled or disabled by passing yes or no in the X-Accel-Buffering response header field." So the streaming endpoint can turn buffering off for itself:

from fastapi.responses import StreamingResponse

def sse_response(generator):
    return StreamingResponse(
        generator,
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",   # the important one
        },
    )

The header travels with the endpoint. Add a route, get correct behavior. No config drift, no "works in staging" incident six months later when someone rebuilds the ingress.

While you're there, note proxy_read_timeout — it defaults to 60s in nginx. A model thinking hard for 90 seconds before its first token will be cut off by your own proxy, and the error you'll see is a generic upstream timeout with no mention of streaming at all.

Break #2: the load balancer kills you during the pause

Same class of bug, one layer out, and it bites hardest exactly on the requests you care most about.

An AWS Application Load Balancer's idle_timeout.timeout_seconds attribute has a valid range of 1–4000 seconds and a default of 60 (see LoadBalancerAttribute in the ELB API reference). "Idle" means no bytes in either direction. An LLM that thinks for 75 seconds before emitting its first token produces exactly that: a silent, live, perfectly healthy TCP connection that your load balancer decides is dead.

You can raise the timeout, and you probably should. But raising it alone is a fragile fix, because it only pushes the cliff further out. The robust fix is to make sure the connection is never actually idle — which SSE has a purpose-built mechanism for.

The SSE wire format treats a line beginning with a colon as a comment. MDN puts it plainly: "A colon as the first character of a line is in essence a comment, and is ignored," and "The comment line can be used to prevent connections from timing out; a server can send a comment periodically to keep the connection alive."

import asyncio

async def sse_stream(request, params):
    queue: asyncio.Queue = asyncio.Queue()

    async def heartbeat():
        while True:
            await asyncio.sleep(15)
            await queue.put(": keepalive\n\n")   # ignored by every SSE client

    # ... producer task pushes real events onto the same queue ...

Fifteen seconds is a good default — comfortably under a 60-second idle timeout, cheap enough that nobody notices. The bytes are discarded by the client and keep every hop on the path convinced the connection is alive.

Anthropic does the same thing on its side, by the way: "Event streams may also include any number of ping events." Your parser needs to expect them and ignore them, which leads directly to the next break.

Break #3: the errors arrive with HTTP 200

Here's the part that catches teams who have otherwise done everything right.

Once the stream has started, the HTTP status code is already sent. It's 200. It will stay 200 no matter what happens next. Failures after that point arrive as events inside the body, and if your parser only handles the happy path, they vanish silently.

The Messages API documents this directly: "The API may occasionally send errors in the event stream. For example, during periods of high usage, you may receive an overloaded_error, which would normally correspond to an HTTP 529 in a non-streaming context":

event: error
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}

If you're using the SDK helpers this is handled for you. If you're parsing raw SSE — which you are, at the browser end, and often at the server end too — you need to branch on it explicitly.

Two more things belong in the same parser:

Unknown event types must not throw. The docs are explicit: "In accordance with the versioning policy, new event types may be added, and your code should handle unknown event types gracefully." A parser with an else: raise in it is a scheduled outage with an unknown date. Log and skip.

stop_reason is not always end_turn. On Opus 4.7 and later, safety classifiers can decline a request with HTTP 200 and stop_reason: "refusal", with a category in stop_details. Check stop_reason before you read content. (stop_details is populated only for refusals and is null for every other stop reason, so guard before reading it.) On Opus 5 and Fable 5 you can also opt into server-side fallbacks — betas: ["server-side-fallback-2026-07-01"] with fallbacks: "default" — which routes by refusal category. If you do, expect a fallback content block in the stream at each model boundary: a content_block_start / content_block_stop pair with no deltas between them. A parser that assumes every block has deltas will choke on it.

Here's the shape of a parser that survives all of it:

for event in raw_events:
    t = event.get("type")

    if t == "content_block_delta":
        d = event["delta"]
        if d["type"] == "text_delta":
            yield sse("token", {"text": d["text"]})
        elif d["type"] == "thinking_delta":
            yield sse("thinking", {"text": d["thinking"]})
        # input_json_delta / signature_delta: accumulate, don't render

    elif t == "message_delta":
        # NOTE: usage counts in message_delta are CUMULATIVE, not incremental
        usage = event.get("usage") or {}
        output_tokens = usage.get("output_tokens", output_tokens)
        stop_reason = event["delta"].get("stop_reason", stop_reason)

    elif t == "error":
        yield sse("error", {"code": event["error"]["type"]})
        return

    elif t in ("ping", "content_block_start", "content_block_stop", "message_start", "message_stop"):
        pass

    else:
        log.info("unknown stream event, ignoring", extra={"event_type": t})

That comment about cumulative usage is worth internalizing — the docs flag it with a warning box. Adding up message_delta usage across events gives you a token count that grows quadratically and a cost dashboard that is confidently wrong.

Break #4: the user left and you're still paying

A user asks a question, gets three sentences in, sees the answer isn't what they wanted, and closes the tab.

What happens to the 60,000-token generation you started? In a lot of production apps: it runs to completion, bills in full, and writes its result to a database row nobody will read. Multiply by your bounce rate.

Cancellation has to be propagated explicitly, at every hop. Server side, Starlette (and therefore FastAPI) exposes the disconnect signal:

async def generate(request, params):
    with client.messages.stream(**params) as stream:
        for text in stream.text_stream:
            if await request.is_disconnected():
                break          # exiting the with-block closes the upstream connection
            yield sse("token", {"text": text})

Exiting the context manager is what actually matters — it closes the HTTP connection to Anthropic, and generation stops. A break without the context manager, or a background task that owns the stream and outlives the request, keeps burning tokens.

Browser side, fetch cancellation goes through AbortController:

const controller = new AbortController();
stopButton.onclick = () => controller.abort();

const res = await fetch("/api/chat/stream", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ prompt }),
  signal: controller.signal,
});

Aborting closes the TCP connection, which is what your is_disconnected() check observes. The chain only works if every link is present.

Break #5: `EventSource` can't do what your chat app needs

The browser's native SSE client is EventSource, and reaching for it is the obvious move. It's also, for most LLM chat UIs, the wrong one.

The constructor takes a URL and an options object whose only meaningful member is withCredentials. There is no place to put an HTTP method, a request body, or headers. That rules out sending a prompt as a POST body and rules out an Authorization header. And its built-in auto-reconnect — "By default, if the connection between the client and server closes, the connection is restarted" — is actively hostile for LLM generation: a dropped connection silently fires a brand-new generation, at full cost, with no memory of the tokens already delivered. One flaky connection becomes a duplicate bill.

Use fetch with a ReadableStream reader instead. You get POST, headers, AbortController, and — crucially — no reconnect you didn't ask for:

const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buf = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += value;

  const frames = buf.split("\n\n");
  buf = frames.pop();                       // keep the incomplete tail

  for (const frame of frames) {
    const dataLines = frame
      .split("\n")
      .filter((l) => l.startsWith("data:"))  // ':' comment lines fall out here
      .map((l) => l.slice(5).trimStart());
    if (!dataLines.length) continue;         // heartbeat frame
    handle(JSON.parse(dataLines.join("\n")));
  }
}

Two details that cause real bugs. First, TCP does not deliver your frames whole — reader.read() will hand you half an event, and the buf.split("\n\n") / buf.pop() pattern is what keeps you from parsing a truncated JSON object. Second, multiple consecutive data: lines in one frame are concatenated with newlines between them, per the spec; joining them with "" corrupts any payload containing a newline.

Break #6: partial output is a state you have to design for

Every stream can end three ways: complete, cancelled, failed mid-flight. Most codebases only persist the first.

That produces two ugly symptoms. A user refreshes after a network blip and their half-written answer is simply gone — the connection is stateless, so the tokens are too. And your cost tracking under-reports, because you only record usage on clean completions, while every abandoned generation was billed in full.

The fix is not resumable streams — that's a hard, mostly unnecessary feature. It's persisting the assistant message as it's produced, with a status:

Insert the message row with status = "streaming" before the first token.
Flush accumulated text to it periodically (every ~50 tokens or every second — not every token, unless you enjoy write amplification).
On message_stop, set status = "complete" and write the final usage from get_final_message().
On disconnect or error, set status = "partial" or "failed" and record whatever usage you observed.

Now a refresh renders the partial answer with an honest "generation was interrupted" marker, and your cost dashboard counts abandoned generations — which is the number that tells you whether Break #4 is costing you real money. Wiring persistence, usage accounting, and streaming state together is exactly the kind of plumbing we build end to end in our course on shipping a production AI SaaS, because it's where "demo works" and "product works" actually diverge.

The checklist

Everything above, compressed into things you can go verify this afternoon:

#	Check	Where
1	`proxy_buffering off` or `X-Accel-Buffering: no` on streaming routes	nginx / app
2	Proxy read timeout and LB idle timeout raised above your p99 time-to-first-token	infra
3	Heartbeat comment frames every ~15s	app
4	Parser handles `ping`, `error`, and unknown event types without throwing	app + client
5	`stop_reason` checked before reading `content`; `stop_details` guarded	app
6	`message_delta` usage treated as cumulative, not incremental	app
7	`thinking.display: "summarized"` if you render reasoning	app
8	Client disconnect propagated to the upstream stream (context manager exits)	app
9	`AbortController` wired to a visible stop control	client
10	Partial messages persisted with a status; abandoned usage recorded	app

If you want to know whether any of this is actually costing you, the honest test is a load test with a 30% mid-stream abandonment rate, run against staging, with token usage measured at both ends. The gap between what your dashboard reports and what your Anthropic console reports is the size of the problem.

Streaming is where a lot of AI products quietly lose both money and trust — and it's a plumbing problem, not a prompting one. If you want the full path from a first API call to an application that holds up under real traffic, that's the arc of our Advanced LLM Integration in Production course; the failure-mode discipline behind it — knowing whether a change actually helped — comes from LLM Evaluation and Testing. And if your next feature is voice, every one of these six breaks gets harder, which is its own topic in Voice AI and Realtime Multimodal Agents.

Sources: Streaming Messages — Claude API docs · Using server-sent events — MDN · ngx_http_proxy_module — nginx · LoadBalancerAttribute — AWS ELB API Reference

Anthropic Filed to Go Public. Here's What Actually Changes for Developers Building on Claude

galian — Fri, 14 Aug 2026 11:40:32 +0000

On June 1, 2026, Anthropic PBC confidentially submitted a draft registration statement on Form S-1 to the SEC for a proposed IPO of its common stock. That's the whole confirmed story. The number of shares hasn't been set, the price hasn't been set, no exchange has been named, no ticker has been named, and the announcement says plainly that any offering "will depend on market conditions and other factors."

Everything else you've read this summer — October listing, Nasdaq, $60B raise, which banks are on the cover — is press speculation dressed up in specific numbers. Some of it will turn out right. None of it is confirmed by the company, and none of it should be in your planning documents yet.

I teach AI engineering at Cursuri-AI.ro, an AI education platform in Eastern Europe, and I don't have a stock tip for you. What I do have is the version of this story that matters if you have production traffic going to api.anthropic.com: an IPO changes the incentives of the company on the other end of your API key, and there are four consequences worth engineering around before the S-1 goes public.

First, the numbers that are actually confirmed

Three figures anchor everything else, and all three come from Anthropic or from reporting on its own disclosures:

Metric	Figure
Run-rate revenue	Surpassed $47B, up from ~$9B at the end of 2025
Series H	$65B raised at a $965B post-money valuation
S-1 status	Confidential draft submitted June 1, 2026 — SEC review ongoing

A roughly 5x run-rate increase in under two quarters is the part worth sitting with. That is not a company that needs to go public to survive. It's a company converting enterprise API and coding-agent demand into revenue faster than almost any software business in history, and choosing to put that on a public balance sheet.

It also isn't happening in isolation. SpaceX — which now owns xAI outright and rebranded the combined entity SpaceXAI — listed on Nasdaq under SPCX on June 12, 2026, closing its first day at $160.95, up 19%. Frontier AI is moving from "priced by a handful of private rounds" to "priced by the public market every trading day." That shift is the actual subject of this post.

Consequence #1: your vendor gets a quarterly clock

Private Anthropic optimized on a multi-year horizon and answered to a small set of investors who had already agreed to the thesis. Public Anthropic answers, every ninety days, to a market that will form opinions about gross margin, revenue concentration, and compute spend.

I want to be careful here, because the lazy version of this argument is "public companies get greedy and raise prices." That's not a forecast I can support, and the recent evidence points the other way: Claude Opus 5 shipped in July 2026 at the same $5/$25 per million tokens that Opus 4.8 cost, and Sonnet 5 sits at $2/$10. Inference prices per unit of capability have been falling, not rising.

The realistic pressure isn't the sticker price. It's everything around it:

Free and low-tier generosity is the easiest margin lever to pull, and the least visible in a headline.
Rate limits and capacity allocation start to follow revenue, not goodwill. Enterprise contracts get the GPUs.
Deprecation timelines tighten, because retiring an old model is pure margin.

None of those show up as a price increase. All of them show up in your error rate.

Consequence #2: model retirement becomes an operational risk, not a footnote

This is the one I'd act on this week, and I say that as someone who has eaten the failure.

We had an integration pinned to a hardcoded Anthropic model ID. That model was retired. The result wasn't a graceful warning in a dashboard — it was a live 404 in production, in a customer-facing feature, discovered by users. The lesson generalizes past any one vendor: every hardcoded model ID is a scheduled outage with an unknown date.

Every frontier lab now runs a formal retirement process — xAI's docs currently carry a dedicated "Model Retirement" migration page, and Anthropic publishes deprecation notices. A public company has a stronger incentive to run that process on a tighter schedule. Your codebase should treat model identity as configuration, not as a string literal:

# config, not code — one place to change, one place to audit
MODELS = {
    "default":  os.getenv("LLM_DEFAULT",  "claude-sonnet-5"),
    "hard":     os.getenv("LLM_HARD",     "claude-opus-5"),
    "cheap":    os.getenv("LLM_CHEAP",    "claude-haiku-4-5-20251001"),
}

def complete(task: str, **kwargs):
    model = MODELS[TASK_TIER.get(task, "default")]
    return client.messages.create(model=model, **kwargs)

The test for whether you're exposed is simple: can you change which model serves a workload without a deploy? If the answer is no, that's the highest-leverage refactor on your list, and it's worth doing before you need it at 2am. Getting from "API calls sprinkled across services" to "an application where a model swap is a config change" is the arc we walk through in our course on building AI applications with the Python SDK.

Consequence #3: the S-1 will be the best vendor due-diligence document ever published

This is the genuinely good news, and almost nobody is talking about it.

Frontier labs are financial black boxes. We infer their economics from leaks, from investor decks that reach journalists, and from third-party trackers that disagree with each other by tens of billions. A public S-1 ends that for one of them. When Anthropic's full prospectus lands, you will be able to read, under penalty of securities law:

Gross margin on inference. The single number that tells you how much room exists under current API prices.
Revenue concentration. How much of that $47B comes from the top handful of customers. If one hyperscaler or one coding-agent partner is a huge slice, that's a dependency in your supply chain too.
Compute commitments. Multi-year obligations to chip and cloud suppliers — the fixed costs that determine how price-flexible the company can be in a downturn.
The risk factors section. Labs are required to enumerate, in writing, what could go wrong: litigation, regulation, model liability, safety incidents, key-personnel loss. It's the most candid document a lab will ever publish about itself.

If you're the person at your company who signs off on an AI vendor, put a reminder in your calendar for the day the public S-1 drops. Read the risk factors and the concentration disclosures first. That's a better afternoon of vendor research than any analyst report you'll pay for.

Consequence #4: "which model" stops being a taste question

A public market prices frontier labs against each other continuously, and that competition lands in your API bill and your eval scores. The current spread is already wide enough that model choice is a real engineering decision rather than a preference:

Model	Input / output per 1M tokens
Claude Fable 5	$10 / $50
Claude Opus 5	$5 / $25
Claude Sonnet 5	$2 / $10
Grok 4.6	$2 / $6

An 8x spread on output tokens between the top and bottom of that table means the same agent architecture can cost wildly different amounts depending on routing. And routing decisions made on vibes are how teams end up paying flagship prices for classification work.

The prerequisite for routing well is an eval harness that can answer "did quality hold when I moved this workload down a tier?" with data instead of a hunch. Without it, every model launch is a risk you absorb. With it, every launch is a shopping opportunity. That harness is the core of our LLM evals in production course, and the comparative side — what each frontier family is actually good at — is what we break down in our AI model comparison course.

What I'd actually do this quarter

Five items, in the order I'd tackle them:

Grep for hardcoded model IDs. Move every one to configuration. This is a half-day of work that prevents a class of outage.
Build or fix the eval gate. You cannot safely change models, effort levels, or providers without one. Everything else on this list depends on it.
Instrument cost per task finished — not cost per token. Token price is a distraction; what you care about is what it costs to complete one unit of real work, including retries and reasoning tokens.
Read your contract's deprecation terms. How much notice are you actually owed before a model you depend on goes away? If you're on a standard developer plan, the answer is probably "whatever the public policy says," which can change.
Prove that failover works. Not "we have a second provider configured" — actually run a game day where the primary returns 429s and 500s and see what your users experience.

None of this is IPO-specific advice. It's just ordinary production hygiene that an IPO makes newly urgent, because the difference between a well-run integration and a fragile one only becomes visible when the vendor's incentives shift.

The bottom line

Anthropic filing to go public is, on balance, good for developers. A company with $47B in run-rate revenue and public-market accountability is a more predictable dependency than a private lab whose economics you have to guess at. You get disclosure, you get a documented risk profile, and you get a competitor that has to keep winning on capability because the scoreboard is now public.

What you also get is a vendor with a quarterly clock, which means the slack in free tiers, generous rate limits, and leisurely deprecation windows is likely to tighten before it loosens. The teams that come out ahead won't be the ones who predicted the ticker. They'll be the ones who spent this quarter making a model swap a config change instead of an incident.

If you want structured, hands-on training on any of this — evals, model selection, production LLM applications — that's what we build at Cursuri-AI.ro.

Sources: Anthropic — confidential draft S-1 submission · TechCrunch — Anthropic files to go public · CNBC — SpaceX IPO, SPCX first-day close

The EU AI Act's Transparency Rules Went Live — Here's What You Actually Have to Ship

galian — Sun, 09 Aug 2026 21:42:39 +0000

Most AI regulation, for most developers, has been something that happens to other people. Risk classifications, conformity assessments, notified bodies — the kind of thing where you nod, assume legal will handle it, and go back to your streaming handler.

Article 50 is different, and it went live on 2 August 2026. It's the part of the EU AI Act that doesn't care whether your system is "high risk." It cares about one thing: can the person on the other end tell that this is AI? If the answer is no, you owe them a disclosure — and the disclosure is a product decision, a UI decision, and in one case a file-format decision. All three land on engineering.

The Commission adopted its guidelines on Article 50 on 20 July 2026, and the AI Office published a Code of Practice on Transparency of AI-Generated Content on 10 June 2026. Between those two documents and the article text itself, the shape of what you have to build is now reasonably clear. This is that shape, translated into work items.

First: does this apply to you?

Two questions, and you're probably in scope on both.

Are you a provider or a deployer? The Act splits duties. A provider develops an AI system (or has one developed) and places it on the market under its own name — if you built the chatbot or the image generator, that's you. A deployer uses an AI system under its own authority — if you dropped someone else's model into your support widget, that's you. Article 50 assigns two obligations to providers and two to deployers, and plenty of teams are both.

Does the EU reach you? The Act's territorial hooks (Article 2) are not "are you an EU company." They're closer to "is the system placed on the EU market, or is its output used in the EU." A US startup with EU users is in scope. This is the GDPR pattern, and it caught a lot of people by surprise the first time.

Not in scope: things that don't interact with people or generate content for them. The Commission's guidelines explicitly put spam filters and automated translation tools outside Article 50(1), and voice assistants and chatbots inside it.

Obligation 1 — tell people they're talking to an AI

Providers shall ensure that AI systems intended to interact directly with natural persons are designed and developed in such a way that the natural persons concerned are informed that they are interacting with an AI system, unless this is obvious from the point of view of a natural person who is reasonably well-informed, observant and circumspect. — Article 50(1)

This is the one that hits the most products, and it's the easiest to get wrong in a way that looks compliant.

The "obvious" carve-out is narrower than you want it to be. The test isn't "our users are technical." It's a reasonably well-informed, observant and circumspect person in the circumstances and context of use. A widget labelled "AI Assistant" in a developer tool is plausibly obvious. The same engine answering an inbound phone call in a warm, human-sounding voice is not — and voice is exactly where the gap is widest right now, because realtime voice agents are good enough that the "obvious" defence has quietly stopped being true.

Where the disclosure has to live. Article 50(5) settles the argument you're about to have with someone in a planning meeting: the information must be provided at the latest at the time of the first interaction or exposure, and it must be clear and distinguishable. That rules out:

burying it in the Terms of Service
a footnote in the privacy policy
a tooltip behind a hover on a mobile UI
disclosing on turn three, after the user has already asked something personal

Article 50(5) also requires conformity with applicable accessibility requirements — which in practice means your disclosure has to survive a screen reader, not just a design review.

What that looks like in a chat surface is boring, and boring is the point:

<!-- Rendered before the first assistant message, not after it. -->
<div class="ai-disclosure" role="note">
  <strong>You're chatting with an AI assistant.</strong>
  Answers are generated automatically.
  <a href="/support/human">Talk to a person</a>
</div>

For voice, the equivalent is a spoken line in the first turn — before you collect anything, and in the language of the call. For an API you sell to other developers, the honest move is to pass the obligation downstream explicitly: document it, and give integrators a disclosure string they can render, because when they ship your model to end users under their own brand, the deployer duties become theirs and the design duty stays yours.

One more thing worth building while you're in there: a handoff path to a human. Article 50 doesn't mandate it. But the disclosure lands very differently when it's followed by an escape hatch, and support teams that ship AI assistants without one tend to discover the reason the hard way.

Obligation 2 — mark synthetic output so machines can detect it

Providers of AI systems […] generating synthetic audio, image, video or text content, shall ensure the outputs […] are marked in a machine-readable format and detectable as artificially generated or manipulated. — Article 50(2)

This is the genuinely hard engineering item, and it's the one with a different deadline (more on that below).

Two properties are required, and they're not the same thing:

Machine-readable marking — metadata that a downstream system can parse.
Detectability — the output can be recognised as artificially generated or manipulated.

The Act asks for solutions that are "effective, interoperable, robust and reliable as far as this is technically feasible" — a standard that explicitly bends to the state of the art. It also carves out assistive editing functions and systems that do not substantially alter the input data. Your auto-crop and your denoise filter are not in scope. Your "generate a product photo from this prompt" endpoint is.

What the Code of Practice points at. The AI Office's Code, published 10 June 2026, describes a layered approach rather than a single mechanism: signed, timestamped provenance metadata — where C2PA is the standard identified as meeting those criteria — plus an imperceptible watermark embedded in the content itself, robust enough to survive ordinary transformations like compression, cropping, scaling and format conversion. The Code is voluntary and, at the time of writing, going through an adequacy assessment by the Commission and the AI Board. Adhering to it is a route to demonstrating compliance; not adhering means you have to show equivalently adequate means of your own.

In practice, for images and video, that means attaching C2PA Content Credentials at generation time. The Content Authenticity Initiative ships open-source tooling for this — c2pa-rs with Python, JS, C++, Swift and Android bindings, plus a c2patool CLI — so this is a library integration, not a research project.

The assertion that carries "this was AI-generated" is the IPTC digital source type, referenced inside a c2pa.actions assertion:

{
  "assertions": [
    {
      "label": "c2pa.actions",
      "data": {
        "actions": [
          {
            "action": "c2pa.created",
            "digitalSourceType":
              "http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia"
          }
        ]
      }
    }
  ]
}

trainedAlgorithmicMedia is the IPTC code for content created by a generative model. There are neighbouring codes for composites and for algorithmically edited media — pick the one that actually describes what your pipeline did, because "created" on a system that only retouched is its own kind of wrong. Verify the current assertion shape against the C2PA spec before you ship; the standard is still moving.

Three things nobody tells you in the compliance deck:

Metadata gets stripped. Plenty of platforms re-encode uploads and discard provenance metadata on the way in. Signing at generation is necessary; assuming it survives the internet is not. This is precisely why the Code pairs metadata with a watermark instead of trusting either alone.
Text is the weak link. Machine-readable marking of text has no equivalent of C2PA that works after a copy-paste. Statistical watermarking of token distributions exists, degrades under paraphrase, and doesn't survive a user retyping the paragraph. The Act's "as far as technically feasible" language is doing real work here — but "hard" is not "exempt," and documenting your reasoning is part of the deliverable.
Sign server-side. Any marking applied in the browser is marking a determined user can skip. The signature belongs on the generation path, before the bytes reach a client.

If you're generating images or video in a product today, this obligation is now a line item in your media pipeline, not a policy question.

Obligation 3 — emotion recognition and biometric categorisation

Deployers of an emotion recognition system or a biometric categorisation system shall inform the natural persons exposed thereto of the operation of the system, and shall process the personal data in accordance with [the GDPR and related instruments]. — Article 50(3)

Shorter, and mostly a matter of knowing that it applies to you. If your product infers emotional state from voice, face or text, or sorts people into categories from biometric data, you inform the people exposed to it — and you're squarely in GDPR territory on top, usually with special-category data.

Before you scope the disclosure: check Article 5 first. Some emotion recognition — in the workplace and in education — is prohibited outright, not merely subject to transparency, and has been since February 2025. Article 50 is the wrong chapter to be reading if that's your use case.

Obligation 4 — deepfakes and public-interest text

Deployers of an AI system that generates or manipulates image, audio or video content constituting a deep fake, shall disclose that the content has been artificially generated or manipulated. — Article 50(4)

Note the split from Obligation 2: marking the file is the provider's duty; disclosing to the audience is the deployer's. If you use a third-party model to produce a synthetic spokesperson for a campaign, the vendor's C2PA manifest doesn't discharge your obligation. You still have to tell the audience.

Two carve-outs matter:

Artistic and satirical works. Where the content is part of an evidently artistic, creative, satirical or fictional work, the disclosure shrinks to revealing the existence of generated content in an appropriate manner that does not hamper the display or enjoyment of the work. A film doesn't need a permanent banner across the frame.

Text on matters of public interest. AI-generated or manipulated text published to inform the public on matters of public interest must be disclosed — unless it has undergone human review or editorial control and a natural or legal person holds editorial responsibility. This is the clause every content-heavy site should read twice. An unreviewed AI-written news summary needs a label. The same article with a named editor who checked it and owns it does not. If your publishing workflow can't currently prove which of those two happened, that's the actual gap — and it's a workflow problem before it's a legal one, which is why content operations built on AI now need an audit trail as much as a style guide.

The deadlines, which are not all the same

The Digital Omnibus on AI — published in the Official Journal on 24 July 2026 and in force since 27 July — shifted several AI Act dates. Article 50 came out of it mostly intact:

What	Applies from
Article 50(1) AI-interaction disclosure	2 August 2026
Article 50(3) emotion recognition / biometric categorisation	2 August 2026
Article 50(4) deepfake and public-interest text disclosure	2 August 2026
Article 50(2) marking, systems placed on the market on or after 2 Aug 2026	2 August 2026
Article 50(2) marking, systems placed on the market before 2 Aug 2026	2 December 2026
Annex III high-risk obligations (Chapter III)	deferred to 2 December 2027
High-risk AI embedded in regulated products (Annex I)	deferred to 2 August 2028

That last pair is the source of most of the confusion in the room right now: the high-risk regime got a long deferral, and a lot of teams heard "the AI Act got pushed back" and stopped reading. Article 50 did not get pushed back. The only grace period is the marking obligation for generative systems that were already on the market, and it expires on 2 December 2026.

Enforcement sits with national market surveillance authorities, the AI Office and — for EU institutions — the European Data Protection Supervisor. Breaching Article 50 carries fines of up to €15 million or 3% of worldwide annual turnover, whichever is higher.

A ship checklist

Pin this to the epic:

[ ] Inventory every surface where an AI system talks to a person — chat, voice, email autoresponders, in-app agents. Each one gets a first-interaction disclosure or a written argument for why it's obvious.
[ ] Disclosure rendered before the first AI output, accessible, in the user's language, not in the ToS.
[ ] Every generative endpoint identified as provider-side or deployer-side. Write it down; vendor contracts should say the same thing.
[ ] C2PA Content Credentials signed server-side on image/audio/video generation, with the correct IPTC digitalSourceType.
[ ] Watermarking assessed for each modality; where it isn't feasible, the reasoning is documented rather than assumed.
[ ] Deepfake disclosure at the publication surface, not just in the file metadata.
[ ] Editorial-review provenance recorded for AI-assisted public-interest text — who reviewed, when, who owns it.
[ ] Emotion recognition / biometric categorisation checked against Article 5 prohibitions before anything else.
[ ] Disclosure copy and placement covered by a test, so the next redesign doesn't silently delete it.

Conclusion

The thing that makes Article 50 unusual, as regulation goes, is how little of it is paperwork. There's no conformity assessment here, no technical documentation dossier, no notified body. There's a banner that has to render before the first message, a signature that has to happen on the generation path, a label that has to reach the audience, and a record of who reviewed what. Four engineering tickets, roughly, and none of them are hard.

They're just easy to defer — and the deferral is what gets expensive. Article 50 is now enforceable, the guidelines are published, the Code of Practice exists, and the tooling for the hard part is open source. Compliance here is mostly a question of whether someone put it in the sprint.

The teams that will have the least trouble with this aren't the ones with the biggest legal department. They're the ones that were already willing to tell users, plainly, what the machine was doing. Turns out that was always the good product decision — it just became the required one.

Written by the team behind Cursuri-AI.ro, an AI education platform with hands-on English-language courses on AI, data privacy and EU AI Act compliance, production LLM integration, and shipping AI products.

Sources & further reading:

EU AI Act — Article 50: Transparency Obligations for Providers and Deployers of Certain AI Systems (full text, exemptions, codes of practice)
European Commission — Guidelines on transparency obligations for providers and deployers of certain AI systems (adopted 20 July 2026)
European Commission — Code of Practice on Transparency of AI-generated Content (published 10 June 2026)
Content Authenticity Initiative — open-source C2PA SDKs and c2patool
IPTC — Digital Source Type NewsCodes vocabulary (trainedAlgorithmicMedia and related values)

This article is educational content written by engineers, not legal advice. Article 50 interacts with the GDPR, the DSA, national implementing rules and sector regulation, and the Digital Omnibus changed several dates in 2026 — verify against current official sources and your own counsel before shipping.

Claude Opus 5 Is Out — Here Are the API Changes That Will Actually Break Your Code

galian — Tue, 28 Jul 2026 22:38:55 +0000

Anthropic shipped Claude Opus 5 on July 24, 2026, and if you build on the Claude API there's a version of this launch story you can safely skip: the benchmark charts. The version you can't skip is the API contract, because for the first time in a while an Opus release changes how existing requests behave — and one parameter combination that was perfectly valid on Opus 4.8 now returns a 400 error.

I write and teach about AI engineering at Cursuri-AI.ro, Eastern Europe's AI education platform, and this post is the writeup I wish every model launch came with: not "how smart is it," but "what do I need to change in my code, in what order, and what silently behaves differently if I change nothing." Everything below comes from Anthropic's official "What's new in Claude Opus 5" documentation — no leaked numbers, no vibes.

Quick disclaimer: this space moves monthly and this is a launch-window snapshot. Verify against the official docs before you wire anything to production.

The spec sheet, in one table

Spec	Claude Opus 5
Model ID	`claude-opus-5` (`anthropic.claude-opus-5` on Bedrock)
Context window	1M tokens — both default and maximum (no smaller variant)
Max output	128k tokens
Thinking	On by default
Pricing	$5 / $25 per million input/output tokens — unchanged from Opus 4.8
Fast mode	$10 / $50, research preview, Claude API only

Two of these are bigger deals than they look. The 1M context window isn't an opt-in variant — it's just what the model has, and the docs specifically claim consistent instruction following, tool calling, and reasoning throughout the window. And 128k max output makes single-request long deliverables (big refactors, full reports) realistic — with a caveat about max_tokens we'll get to, because it's now doing more work than it used to.

Opus 4.8 stays available on every platform, so nothing forces a same-day migration. But the changes below are the kind you want to understand before your first "why is this request failing" incident, not after.

Change #1: thinking is on by default

On Opus 4.8, a request without a thinking field ran without extended thinking. You opted in with thinking: {"type": "adaptive"}.

On Opus 5, that same field-less request now runs with thinking on. The model decides when and how much to think on each turn, and the effort parameter is the knob that controls thinking depth. If your code already sends thinking: {"type": "adaptive"}, you're fine — that value remains valid and is equivalent to the new default.

Why this can bite you even though it sounds like a free upgrade:

max_tokens is a hard limit on total output — thinking plus response text. A workload that ran happily with max_tokens: 2000 on Opus 4.8 (no thinking, short answers) can now spend a chunk of that budget on reasoning before it writes a single visible token. The official guidance is explicit: revisit max_tokens for every workload that previously ran without thinking.

Change #2: the actual breaking change — disabled thinking + high effort = 400

Here's the one to grep your codebase for today:

thinking: {"type": "disabled"} is accepted only when effort is high or below.
thinking: {"type": "disabled"} combined with effort xhigh or max returns a 400 error, on every request.
This is generally available behavior (not beta) from Opus 5 onward — and it's a breaking change from Opus 4.8, where disabling thinking was independent of the effort level.

If you currently run thinking-disabled at high effort levels, you have exactly two exits: keep thinking disabled and drop effort to high or below, or keep your effort level and delete the thinking field entirely.

There's a second, subtler reason to just let thinking stay on. The docs note that with thinking disabled, Opus 5 can occasionally write a tool call into its text output instead of emitting a proper tool_use block, or leak internal XML tags into the visible response. If you parse tool calls in production, that's not a cosmetic footnote — it's a failure mode your handler needs to survive. Anthropic's recommendation is to keep thinking enabled and control cost with lower effort levels instead.

Change #3: `effort` is now the control lever that matters

With thinking on by default, effort becomes the central parameter of your integration. The full ladder on Opus 5: low, medium, high (the default), xhigh, and max. The docs make a claim worth taking seriously: Opus 5 converts additional effort into better results more reliably than any earlier Opus model — which means the level you pick carries more weight than it did on 4.8. At the other end, low and medium are explicitly called out for producing strong quality at a fraction of the tokens and latency.

A request with everything turned up looks like this:

import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model="claude-opus-5",
    max_tokens=64000,
    output_config={"effort": "max"},
    messages=[{"role": "user", "content": "..."}],
) as stream:
    response = stream.get_final_message()

Three deliberate details in that snippet: there's no thinking field (the default is what you want), max_tokens is large (at xhigh/max the model needs room to think and act across tool calls), and it's streamed — at 64k-token budgets, non-streaming requests can hit the time limit.

The sane strategy is the documented one: start at the default high, then adjust based on your own evals, not vibes. Step down where quality holds — you pocket the tokens and latency. Step up to xhigh/max for the genuinely hard work. If you don't have an eval harness that can answer "did quality hold when I dropped effort?", that's the single highest-leverage thing to build this quarter — it's the discipline we treat as foundational in our course on LLM evals in production.

Change #4: the model behaves differently, even if you change nothing

The docs have a refreshingly honest section on differences you'll notice without touching your code:

Default responses run longer — both user-facing answers and written deliverables. If your product has strict length constraints, enforce them in the prompt.
In agentic sessions, the model narrates its progress more often. Good for UX transparency; tune it down where you want silence.
In multi-agent frameworks, it delegates to subagents more readily — budget accordingly, subagents are tokens too.
It verifies its own work without being told to. This is the one that demands action: verification instructions inherited from older prompts — "include a final verification step," "use a subagent to verify" — should be removed, because on Opus 5 they cause over-verification. You pay twice, in tokens and latency, for work the model already does.

This is a pattern every migration teaches: prompts get calibrated against the previous model's weaknesses, and those calibrations become friction on the next model. A model migration without a prompt audit is half a migration. That workflow — prompts, tools, verification, on real repos — is exactly what we drill in our Claude Code and agentic coding course.

The smaller changes you'll actually be happy about

Prompt caching minimum drops to 512 tokens, from 1,024 on Opus 4.8. System prompts that were too short to cache start caching with zero code changes. If you run compact prompts at volume, this shows up on your invoice by itself.
Mid-conversation tool changes (beta): you can add or remove tools between turns while preserving the prompt cache, with the mid-conversation-tool-changes-2026-07-01 beta header. For phase-based agents (explore → edit → verify, different tools each), this removes a whole architectural compromise — no more front-loading every tool "just in case."
A "default" mode for fallbacks (beta): the fallbacks parameter can now apply Anthropic's recommended fallback models by refusal category, instead of a hand-maintained model list. Use the server-side-fallback-2026-07-01 header (the older 2026-06-01 header only accepts explicit lists).
Fast mode exists but is API-only: $10/$50 per million tokens, research preview, not currently on Bedrock, Google Cloud, or Microsoft Foundry. If you route through a cloud provider and were counting on it, adjust your plan.

The migration checklist, in the order things break

Grep for the fatal combination: thinking: {"type": "disabled"} anywhere near effort xhigh/max → that's a 400 now. Decide per integration: drop the thinking field, or lower effort.
Revisit max_tokens on every request that ran without thinking on 4.8 — the budget now covers reasoning + response.
Swap the model ID to claude-opus-5 — from a config variable, not a hardcoded string. (Model retirements turn hardcoded IDs into production 404s; ask me how I know.)
Audit your prompts for verification instructions and remove them — over-verification is pure waste on Opus 5.
Turn on streaming for anything with a large max_tokens.
If you must run thinking-disabled, make your parser survive tool calls in text and stray XML tags.
Re-run your evals before shifting traffic. "Better on average" is not "better on your tasks."

If your architecture makes step 3 scary — model IDs scattered across services, no config layer, no eval gate — that's a structural problem worth fixing once, properly. Going from "API calls sprinkled through the codebase" to "an application where a model swap is a config change" is the arc of our building AI applications with Python course.

The bottom line

Opus 5 doesn't ask you to learn a new API — it asks you to re-check the assumptions your existing code was built on. Thinking is on unless you say otherwise, effort is the lever that matters, max_tokens now pays for cognition too, and one previously-valid parameter combo is a hard error. In exchange you get a 1M-token window as the default, 128k output, cheaper caching, and betas that clean up real architectural pain.

Teams that treat this as a checklist migration — rather than a find-and-replace on the model name — will come out with integrations that are cheaper and more reliable than what they had. And that discipline, built once, pays out again on every launch that follows.

If you want structured, hands-on training on any of this — evals, agentic coding, production LLM apps — that's what we build at Cursuri-AI.ro.

Multi-Agent AI Systems: When One Agent Isn't Enough — and When Two Is Already Too Many

galian — Wed, 22 Jul 2026 20:54:34 +0000

Somewhere around the third time your agent's context window filled up with search results it no longer needed, you had the thought every agent builder eventually has: what if I split this into multiple agents? One to research, one to write, one to review. Maybe a "manager" agent on top. It sounds like an org chart, and org charts feel like architecture.

Sometimes that instinct is exactly right — Anthropic's research system, built as an orchestrator with parallel subagents, outperformed its best single-agent setup by a wide margin on internal evals. And sometimes it's exactly wrong — Cognition (the team behind Devin) published a piece bluntly titled "Don't Build Multi-Agents", arguing that for their domain, splitting the work is how you manufacture inconsistency. Both teams are right, because they're solving different problems.

This article is the decision framework between those two positions: what multi-agent architectures actually buy you, what they cost, the orchestration pattern that works in production, and the failure modes that don't make it into the launch posts.

What a multi-agent system actually is

Strip the buzz and it's simple: a multi-agent system is one where an LLM-driven agent can delegate work to other LLM-driven agents, each running in its own context window, and use their results. The common production shape is orchestrator–worker: a lead agent owns the goal, decomposes it into subtasks, spawns subagents to execute them (often in parallel), and synthesizes what comes back.

That "own context window" clause is the entire point. It's not about the anthropomorphic org chart — it's about context isolation. Everything else follows from it.

Notice what this is not: a pipeline of prompts you call in sequence from your own code is not a multi-agent system — it's just a program (and often, that's exactly what you should build instead). The multi-agent label earns its keep when the decomposition itself is dynamic — when an agent decides at runtime what to delegate, to whom, and with what instructions.

The three things fan-out actually buys you

1. Context isolation. A subagent that greps a codebase or reads twenty web pages burns thousands of tokens on intermediate noise — raw file contents, search results, dead ends. In a single-agent design, all of that lands in the one context window you have, where it dilutes attention for every subsequent step. In an orchestrator–worker design, the noise lives and dies in the subagent's window; the orchestrator receives a compressed conclusion. You've effectively multiplied your usable context by the number of workers.

2. Parallelism. Independent subtasks — "evaluate these five libraries," "check each of these twelve services for the deprecated call" — can run concurrently instead of serially. For research-shaped and audit-shaped work, this is a wall-clock improvement measured in multiples, not percent.

3. Separation of concerns. Each subagent gets a narrow prompt, a curated toolset, and one job. A focused agent with six relevant tools reliably beats a generalist juggling twenty — the same reason you curate tool sets aggressively in context engineering for agents, applied at the architecture level.

Anthropic's engineering write-up on their multi-agent research system put numbers on this: the orchestrator-with-parallel-subagents architecture outperformed their single-agent baseline by 90.2% on an internal research eval, and they found that token usage — how much relevant thinking-and-reading the system could do — explained most of the performance variance. Fan-out is, at bottom, a way to spend more tokens productively on one problem than a single context window physically allows.

What it costs you

The same write-up is refreshingly honest about the bill: in their data, a single agent used roughly 4× more tokens than a chat interaction, and multi-agent systems roughly 15×. That's the entry fee. Multi-agent architectures only make economic sense when the task's value clears it.

The subtler cost is coordination. The orchestrator communicates with subagents through task descriptions — and every ambiguity in those descriptions becomes a subagent doing confidently the wrong thing in a context you can't see. Early versions of Anthropic's system had subagents duplicating each other's work, wandering off-scope, and searching for things other subagents had already found — not because the model was weak, but because the delegation instructions were vague. The fix was unglamorous prompt engineering: each delegation needs an explicit objective, output format, tool guidance, and boundaries where the subtask ends.

The case against — and it's a strong one

Cognition's counter-argument deserves to be taken seriously, because it identifies the precise condition under which multi-agent designs fail: when subtasks are not actually independent.

Their core principles: share full context and trace history, and remember that actions carry implicit decisions. When two subagents work in parallel from partial views of the task, each makes small implicit decisions — naming, structure, interpretation of an ambiguous requirement. When their outputs meet, those decisions conflict. For coding work, where part A and part B must compile, link, and agree on conventions, parallel agents with partial context produce exactly the inconsistency you'd expect from two contractors who never spoke to each other.

So the honest synthesis of the two positions isn't "Anthropic vs. Cognition" — it's a property of the task:

Read-heavy, decomposable, breadth-first work (research, audits, evaluation sweeps, codebase exploration) parallelizes beautifully. Subagent outputs are findings that an orchestrator can merge, and conflicts between them are cheap to resolve.
Write-heavy, interdependent, convention-laden work (feature implementation, refactoring, anything where outputs must cohere) punishes fan-out. Here a single agent with a well-managed context — compaction, external memory, just-in-time retrieval — is the stronger architecture.

A useful mnemonic: fan out to read, stay single to write. It's not absolute (isolated write tasks in separate worktrees parallelize fine), but it's the right default, and knowing when to break it is precisely the judgment that a serious course on AI agent architecture spends real time building.

The orchestration patterns that survive production

If your task passes the test above, these are the shapes that hold up:

Orchestrator–worker (the default). One lead agent decomposes, delegates, synthesizes. Workers don't talk to each other — results flow through the orchestrator. This keeps the communication topology a star, not a mesh, which is the difference between debuggable and not. Resist the peer-to-peer "agents chatting with agents" fantasy; in production it mostly produces token-expensive misunderstandings.

Pipeline with isolated stages. Each item flows through fixed stages (find → verify → summarize), each stage its own agent call with its own clean context. No barrier between items — item A can be in stage 3 while item B is in stage 1. Deterministic control flow in your code, LLM judgment inside each stage. This hybrid — code decides the structure, agents fill the slots — is the most underrated pattern in the space, and it's where most "do we even need a framework?" questions dissolve.

Adversarial verification. Spawn N independent agents to refute a finding rather than confirm it, and keep only what survives. This is fan-out used not for speed but for confidence — independent contexts mean independent mistakes, which is exactly what majority voting needs to work. It's the cheapest defense against the plausible-but-wrong output that a single agent will happily double down on.

Three implementation rules that pay for themselves regardless of pattern:

Delegate like you're writing a ticket for a contractor. Objective, output format, tools to use, scope boundaries. If a competent human would need to ask a clarifying question, your subagent will instead guess — silently.
Return data, not prose. Subagents should report structured results (schema-validated if your stack allows it), because the orchestrator is a program consuming output, not a reader enjoying it.
Match effort to value. Simple lookups get one worker with a small budget; open-ended research earns a fleet. An orchestrator that spawns ten subagents for a question one search would answer is the multi-agent version of the microservices monorepo with three users.

The failure modes nobody demos

Every one of these is a thing that happens after the demo works:

Duplicated work. Two subagents independently discover the same fact at full token price, because the orchestrator's task descriptions didn't partition the space. Partition explicitly.
Compounding errors. In a pipeline, a stage-one hallucination becomes stage-two's trusted input. Multi-agent systems don't just propagate errors — they launder them: by the final synthesis, the wrong fact arrives with the confidence of having been "processed" three times. This is why verification stages belong early, not only at the end.
The telephone game. Every orchestrator-mediated hop compresses information. Three hops in, "the test fails intermittently on CI under load" has become "the tests are broken." Keep hierarchies shallow — one level of delegation covers almost every real use case.
Lost partial work. Ten parallel subagents, one crashes at minute eight. If your harness can't resume without re-running the other nine, you'll pay the 15× token bill more than once. Checkpointing and resumability are boring, and they are the difference between a system and a demo.

And the meta-failure that enables all of the above: not measuring. A multi-agent system has more knobs than anything else you'll build with LLMs — decomposition granularity, worker count, delegation prompts, synthesis strategy — and every knob is a chance to regress silently. The teams that ship these systems treat evaluation as first-class infrastructure: a representative task set, end-to-end outcome scoring (with an LLM judge where rubrics allow), and a re-run on every architectural change. "It seemed better on the three queries I tried" is how 15× token bills get approved for 1× results.

A decision checklist

Before you split one agent into several, put the task through this:

Question	If yes →	If no →
Can subtasks run with genuinely independent context?	fan-out is viable	stay single-agent
Is the work read/analyze-heavy rather than write/produce-heavy?	fan-out helps	single agent + context engineering
Does the task value clear a ~15× token cost?	proceed	simplify the architecture
Would a fixed pipeline in code + single agent calls do it?	build that instead	orchestrator–worker
Do you have evals to catch coordination regressions?	ship	build evals first

Note how often the honest answer is the boring one. A single agent with disciplined context management — compaction, external memory, sub-tasking only when context demands it — is the right architecture for most of what gets built today, and wiring either variant into a real product with retries, budgets, and observability is its own production discipline on top of the architecture choice.

Conclusion

Multi-agent systems aren't the next tier above single agents — they're a specialized tool that trades tokens and coordination complexity for context capacity and parallelism. That trade is spectacular for breadth-first, read-heavy, decomposable work, and actively harmful for interdependent, convention-heavy production of artifacts. Fan out to read; stay single to write; let deterministic code own the control flow either way; and put an eval behind every knob.

The teams that get this right aren't the ones with the most agents. They're the ones who can articulate, for their specific task, what a second context window buys — and who kept the architecture exactly one notch more complex than that answer requires.

I build and teach this material at Cursuri-AI.ro, an AI education platform with hands-on English-language tracks on AI agent architecture, context engineering, LLM evaluation, and production integration.

Sources & further reading:

Anthropic — How we built our multi-agent research system (orchestrator–worker pattern, 90.2% eval result, token-cost data, delegation lessons)
Anthropic — Building effective agents (workflows vs. agents, orchestrator–workers, "simplest thing that works")
Cognition — Don't Build Multi-Agents (context-sharing principles, the case for single-threaded agents)

This article is educational content. Architectures, model capabilities, and costs evolve quickly — validate against your own workloads and current documentation.

We just launched our AI course catalog in English — 20 hands-on courses with an AI professor in every lesson

galian — Fri, 17 Jul 2026 07:21:16 +0000

I'm the founder of Cursuri AI, an AI e-learning platform for professionals. "Cursuri" is simply Romanian for "courses" — the platform started in Romania, where it now runs a catalog of 50 AI courses in Romanian, from prompt engineering fundamentals to building production AI agents.

This week we shipped the thing people kept asking for: the platform is now available in English, with an international catalog of 20 courses built for people who want to actually use AI at work — not just read about it.

This post is the announcement, but I also want to be concrete about what the platform does and what's in the catalog, so you can decide in two minutes whether it's for you.

The idea: learning AI should feel like working with AI

Most online courses are passive: you watch or read, you nod along, and a week later you remember 10% of it. That model is especially broken for AI, where the whole point is interaction — prompting, iterating, getting feedback, correcting course.

So we built the platform around active learning instead:

An AI professor in every lesson. Not a generic chatbot bolted onto the side — an assistant that knows exactly which lesson you're in and answers questions in that context. You can type or just talk to it: it works in text and voice. Stuck on why your RAG retrieval returns garbage, or what a "system prompt" actually is? Ask mid-lesson and get a precise answer. There's a full overview of how it works on the AI professor page.
Interactive AI quizzes. Questions adapt to your level, and every answer comes with a detailed explanation — including why the wrong options are wrong, which is where most of the learning actually happens.
Hands-on exercises on the platform. Coding challenges for the engineering track, realistic work scenarios for the business track. You apply things as you learn them, not "someday later."
Automatic AI summaries. Key points extracted from every lesson, so reviewing before an interview or a meeting takes minutes, not hours.
A personal dashboard. Progress, scores, and insights, so you can see whether you're actually getting better or just clicking "next."

And because AI moves absurdly fast, content is kept up to date — courses get refreshed as the tools and models change, and new courses are added to the bundles at no extra cost.

What's in the English catalog

The 20 launch courses are split into two tracks, because "learning AI" means very different things for a backend engineer and a marketing manager.

IT & Engineering track (10 courses)

This is the track I'm personally most excited about, because it covers the stack of skills that 2026-era AI engineering actually demands:

Introduction to AI Engineering — the on-ramp if you're a developer new to the field
The Complete Prompt Engineering Masterclass
RAG: Retrieval-Augmented Generation in Practice
Advanced LLM Integration in Production Applications
AI Agents: Architecting and Automating Autonomous Systems
Context Engineering and Memory for AI Agents — beyond prompting: managing what your agent knows and remembers
Claude Code Mastery: Agentic Coding from the Terminal — multi-file work, git workflows, CI, MCP
MCP (Model Context Protocol): Building Servers and Integrations
Cursor as a Pro: AI-Native IDE, Composer and Multi-Agent workflows
Vibe Coding: From Prompt to Application with Lovable, v0, Bolt and Replit Agent

If you've been meaning to move from "I use ChatGPT sometimes" to "I ship AI features and agentic workflows," this track is a structured path through exactly that.

Business & Professionals track (10 courses)

No code required — built for managers, marketers, founders, analysts, and office professionals:

AI for Business Leaders
Manager in the AI Era: Leading Your Team Through the AI Transformation
AI for Entrepreneurs and Startups: The Complete Guide
AI for Digital Marketing
AI for Content Creation and Copywriting
AI for Sales and CRM
SEO and AEO/GEO in the AI Era — optimizing for Google, AI Overviews and generative engines
Microsoft 365 Copilot for Office Work: Role-Based Productivity
No-Code Data Analysis with AI — ChatGPT, Excel and SQL for non-programmers
AI Image Generation: The Complete Guide from Prompt to Publishing

The full list with detailed curricula is on the course catalog.

Pricing — simple and honest

We deliberately kept it simple, and every plan is billed monthly with cancel-anytime:

Plan	What you get	Price
Single course	Any one course, full access + AI professor	€49/month + VAT
Business	All 10 Business & Professionals courses	€199/month + VAT
IT Pro	All 10 IT & Engineering courses	€399/month + VAT
All Access	The entire international catalog, both tracks	€499/month + VAT

Two things worth calling out:

You're not forced into a bundle. If you only need the RAG course or the Copilot course, subscribe to just that one for €49/month and cancel when you're done.
New courses are included. Bundle subscribers get every new course we release, as it's released.

Full details and a plan comparison are on the pricing page.

Who this is for (and who it isn't for)

It's for you if you're a professional who wants a structured, hands-on path to using AI in your actual job — whether that job is writing code, running a team, or growing a business. It works well for companies too: there's a dedicated track structure and team offering for organizations that want to upskill whole departments.

It's probably not for you if you want academic ML theory — we don't teach you to derive backpropagation. The focus is applied: tools, workflows, and skills you use the same week you learn them.

The ask

If this sounds useful, take a look at cursuri-ai.ro/en — and if you check out a course, I'd genuinely love feedback. We're a team from Romania, competing on quality and depth of content, so every piece of honest criticism from this community makes the platform better.

And if you have questions about the catalog, the AI professor, or where we're taking the platform next — ask away in the comments. I'll answer everything.

Fine-Tuning vs RAG vs Prompting: How to Actually Decide in 2026

galian — Wed, 15 Jul 2026 11:49:01 +0000

There's a predictable arc to most LLM projects. Something doesn't work, someone says "we should fine-tune it," a month disappears into dataset wrangling and GPU bills, and the model comes back... about as wrong as before — because the actual problem was that it never had the right facts in front of it. Fine-tuning was never going to fix that.

The three techniques — prompting, retrieval-augmented generation (RAG), and fine-tuning — are not a ladder you climb from cheap to fancy. They solve different problems, and choosing the wrong one is expensive in exactly the way that's hard to notice: it looks like progress while it burns weeks.

This is a decision framework. Not "here's what each one is" — you can get that anywhere — but the concrete questions that tell you which one your problem actually needs, and the failure signatures that mean you picked wrong.

The one distinction that resolves most arguments

Before any framework, internalize this split, because it settles 80% of the "should we fine-tune?" debates on its own:

RAG changes what the model knows. It injects facts into the context at inference time.
Fine-tuning changes how the model behaves. It adjusts the weights to shift style, format, structure, and task-specific skill.
Prompting changes what the model is told to do right now, using the knowledge and behavior it already has.

So the first question is never "which technique is best?" It's "is my problem a knowledge gap or a behavior gap?"

The model gives outdated, made-up, or "I don't have access to that" answers about your data → knowledge gap → RAG.
The model knows the facts but won't reliably produce the format, tone, or task structure you need → behavior gap → fine-tuning (maybe).
You haven't seriously tried telling it clearly what to do yet → prompting, first, always.

Get this wrong and no amount of engineering saves you. Fine-tuning a model to "know your docs" is the classic error: you can bake a few facts into weights, but they go stale the moment your docs change, you can't cite sources, and you've spent training compute to build a worse version of a lookup. Knowledge that changes belongs in retrieval, not in weights.

Always start with prompting (yes, even now)

Prompting is not the beginner tier you graduate from. In 2026, with frontier models, a well-constructed prompt plus a few good examples solves a startling share of problems that teams assume need training. It's the fastest, cheapest, most inspectable option, and it should be your baseline before you're allowed to say the word "fine-tune."

Reach for prompting when:

You're still discovering what "good output" even looks like. Prompts are editable in seconds; datasets are not.
The task is reasoning, transformation, or generation the model already broadly knows how to do.
You need to ship this week.

The techniques that make prompting punch above its weight are unglamorous but real: precise role and task framing, few-shot examples that demonstrate the exact output shape, chain-of-thought for multi-step reasoning, and rigid output contracts (structured/JSON) so downstream code can trust the result. Most "the model can't do this" conclusions are actually "we asked badly" conclusions. Squeezing the ceiling out of prompting before spending on anything heavier is a discipline in itself — it's the whole point of a prompt engineering masterclass, and the ROI of getting it right first is enormous because everything downstream inherits a better baseline.

The prompting ceiling — how you know you've hit it: you've iterated seriously, added good examples, and the model still fails — and the failure is either (a) it doesn't know facts it couldn't possibly know, or (b) it can't hold a consistent behavior across inputs no matter how you phrase the instruction. (a) points to RAG. (b) might point to fine-tuning. Not before.

Reach for RAG when the problem is knowledge

RAG is the answer whenever the model needs to work with information it wasn't trained on: your internal documentation, a product catalog, last week's tickets, a knowledge base that updates daily, anything private or fresh.

Choose RAG when:

Answers must be grounded in a specific corpus and you need to cite sources.
The knowledge changes — pricing, policies, docs, inventory. You update an index, not a model.
Hallucination on facts is unacceptable and you need an audit trail of where an answer came from.
The knowledge base is large, or partly access-controlled per user.

The reason RAG beats fine-tuning for knowledge isn't subtle: updating a document store is trivial and instant; updating weights is a training run. RAG gives you freshness, provenance, and per-user access control for free — none of which fine-tuning can offer. When your facts have a shelf life, retrieval is the only correct architecture, and building it well (chunking, hybrid search, re-ranking) is where the real engineering lives — the substance of a dedicated course on RAG and retrieval-augmented generation.

RAG's own ceiling: retrieval fixes what the model knows, not how it behaves. If your RAG answers are factually correct but come out in the wrong format, wrong tone, or don't follow your house style no matter how you prompt — that residual behavior gap is exactly where fine-tuning finally earns its place, on top of RAG, not instead of it.

Fine-tune when the problem is behavior — and only then

Fine-tuning is the right tool, but for a narrower set of problems than its reputation suggests. It shines at teaching consistent behavior that's hard to specify in a prompt: a very specific output structure, a domain's tone and terminology, a classification or extraction task where you have lots of labeled examples, or a skill the base model does clumsily.

Legitimately reach for fine-tuning when:

You need consistent style, format, or structure at a level prompting can't hold across the full input distribution.
You have a narrow, high-volume, well-defined task (classification, extraction, a specific transformation) and enough quality labeled data.
You want to bake in a behavior so you can drop it from the prompt — shorter prompts, lower per-call cost, faster responses at scale.
Latency or cost at scale matters and a smaller fine-tuned model can match a bigger prompted one.

Two things make modern fine-tuning far less scary than its reputation. First, you almost never do full fine-tuning — parameter-efficient methods like LoRA/QLoRA train a tiny set of adapter weights, cutting the compute and memory cost by orders of magnitude while getting most of the benefit. Second, the bottleneck is data quality, not model choice: a few hundred to a few thousand clean, consistent, representative examples beat a huge noisy pile every time. The hard part of fine-tuning was never running the training job — it's building the dataset, choosing PEFT trade-offs, and evaluating the result without fooling yourself, which is precisely the ground a fine-tuning course has to cover to be worth anything.

When fine-tuning is the wrong answer — the red flags:

"We'll fine-tune it on our docs so it knows them." → No. That's RAG. Fine-tuned facts go stale and can't be cited.
"We haven't really tried prompting." → Do that first; you may not need to train at all.
"The requirements change weekly." → Fine-tuning bakes behavior in; if the target moves, you're re-training constantly. Keep it in the prompt until it stabilizes.
"We have 40 examples." → Usually not enough for reliable behavior change; strong prompting with those 40 as few-shot examples will likely beat it.

The combinations are the real answer

Framing these as rivals is the beginner mistake. In production, the strongest systems combine them, because they operate on different axes — knowledge, behavior, and instruction — and stack cleanly:

RAG + prompting is the workhorse for most knowledge-grounded assistants: retrieve the right context, then a well-engineered prompt instructs the model to answer only from it and cite sources. No training required.
Fine-tuning + RAG is the high end: fine-tune for the domain's behavior (tone, format, task skill), and use RAG for the facts. The model behaves exactly right and stays current — behavior in the weights, knowledge in the index.
Fine-tuning + prompting collapses a long, brittle instruction into learned behavior, so your prompts get short and your inference gets cheaper.

Orchestrating these — deciding which layer owns which responsibility, and routing a request through retrieval, tools, and the model in the right order — is its own engineering discipline, and it's the core of a course on advanced LLM integration. The mental model to keep: knowledge → retrieval, behavior → weights, instruction → prompt. Put each requirement on the axis it actually lives on.

The decision, in one pass

Run your problem through this, in order. Stop at the first that fits:

Have you genuinely exhausted prompting — clear instructions, good few-shot examples, structured output? If not → prompt. (This is where most projects should still be.)
Is the failure a knowledge gap — missing, stale, or private facts; needs citations? → RAG.
Is the failure a behavior gap — format/tone/task consistency the prompt can't hold, and you have quality labeled data and the target is stable? → fine-tune (LoRA first).
Is it both? → RAG for the facts, fine-tuning for the behavior. In that order.

And underneath all of it: you cannot make this decision without evaluation. "It seems better" is not data. Before you choose, build a small eval set — representative inputs with known-good outputs — so you can measure whether prompting already clears the bar, whether RAG actually retrieves the right context, and whether a fine-tune moved the metric or just moved the failures around. Teams that skip this pick techniques by vibes and discover the mistake in production; teams that treat evals as first-class make the cheap correct choice on purpose. The eval set is what turns this framework from an opinion into a decision.

Conclusion

The reason so many LLM projects stall isn't a shortage of technique — it's reaching for the wrong one and mistaking motion for progress. Fine-tuning a model to "learn facts," RAG-ing a problem that was really a bad prompt, or grinding on prompts when the model fundamentally lacks the data: each fails in a way that looks like effort.

Anchor on the split and you'll rarely go wrong. Knowledge that changes → RAG. Behavior you can't prompt into place → fine-tuning. Everything else → prompt, and prompt well. Start cheap, measure honestly, and add complexity only when an eval — not a hunch — tells you the current layer has topped out. The best architecture isn't the most sophisticated one; it's the one that puts each requirement on the axis where it actually belongs.

Sources & further reading:

Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models
Dettmers et al. — QLoRA: Efficient Finetuning of Quantized LLMs

This article is educational content. Models, tooling, and cost trade-offs evolve quickly; validate any approach against your own data and current provider documentation before committing to it in production.

Vector Embeddings Explained: Build Semantic Search in Python

galian — Mon, 13 Jul 2026 11:25:24 +0000

Search for "reset my password" in a keyword-based system and a help article titled "How to recover your account credentials" won't match — not one word overlaps. Yet any human knows they mean the same thing. Closing that gap between characters and meaning is what vector embeddings do, and they're the quiet engine behind semantic search, RAG, recommendation systems, and most of the "AI that understands you" experiences shipped since 2023.

This is a practical guide. We'll cover what an embedding actually is, why cosine similarity is the operation you'll use constantly, and then build a real, working semantic search engine in Python — first with pure NumPy so you see the mechanics, then with the tools you'd actually reach for in production. By the end you'll have code that runs and a mental model that transfers to every embedding-powered feature you build next.

What an embedding actually is

An embedding is a list of numbers — a vector — that represents a piece of content as a point in high-dimensional space. An embedding model (a neural network trained on enormous text corpora) reads your text and outputs, say, 384 or 1,536 floating-point numbers. The magic isn't the numbers themselves; it's the property the training instills: texts with similar meaning land close together in that space, and unrelated texts land far apart.

That's the whole idea. "How do I reset my password?" and "I forgot my login credentials" produce vectors that sit near each other. "What's the weather in Cluj?" produces a vector off in a completely different region. The model has effectively turned meaning into geometry — and geometry is something a computer can measure with plain arithmetic.

A few properties worth internalizing before we write code:

Dimensionality is fixed per model. A given model always outputs the same length (e.g. 384 for all-MiniLM-L6-v2, 1,536 for OpenAI's text-embedding-3-small). You can't mix vectors from different models — they live in different spaces.
The individual numbers are not interpretable. Dimension 200 doesn't mean "formality" or "topic." Meaning is distributed across all dimensions at once. Don't try to read them.
Distance is the entire point. You almost never care about a vector's absolute position — only how close it is to other vectors.

Cosine similarity: the one operation you'll use everywhere

To ask "how similar are these two texts?", you compare their vectors. The near-universal choice for text embeddings is cosine similarity: it measures the angle between two vectors, ignoring their magnitude.

Why the angle and not, say, straight-line (Euclidean) distance? Because for text embeddings, direction encodes meaning while length often encodes uninteresting things like text length or token count. Two documents about the same topic point the same way even if one is a sentence and the other a paragraph. Cosine similarity captures exactly that.

The formula is just the dot product of the two vectors divided by the product of their magnitudes:

cos(θ) = (A · B) / (‖A‖ · ‖B‖)

It returns a value from -1 to 1, though for most modern text embeddings you'll see results land in roughly the 0-to-1 range:

~1.0 — nearly identical meaning
~0.5 — loosely related
~0.0 — unrelated

That single number, computed against a corpus of stored vectors, is semantic search. Everything else is optimization.

Build a semantic search engine from scratch

Let's make it concrete. We'll use sentence-transformers, which runs a capable embedding model locally — no API key, no network calls, so you can run this offline right now.

pip install sentence-transformers numpy

Step 1 — Embed a corpus

from sentence_transformers import SentenceTransformer
import numpy as np

# A small, fast, widely used model. 384-dimensional output.
model = SentenceTransformer("all-MiniLM-L6-v2")

documents = [
    "How to reset your account password",
    "Refund policy for annual subscriptions",
    "Setting up two-factor authentication",
    "Our office hours and contact details",
    "How to recover a locked account",
]

# Encode the whole corpus once. Shape: (5, 384)
doc_embeddings = model.encode(documents)
print(doc_embeddings.shape)  # (5, 384)

That doc_embeddings array is your search index. In a real app you compute it once, when a document is created or updated, and store it — never on every query.

Step 2 — Cosine similarity in NumPy

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> np.ndarray:
    """Cosine similarity between vector `a` and each row of matrix `b`."""
    a_norm = a / np.linalg.norm(a)
    b_norm = b / np.linalg.norm(b, axis=1, keepdims=True)
    return b_norm @ a_norm

Ten lines, no dependencies beyond NumPy. This is the actual core of semantic search — the rest is plumbing.

Step 3 — Search

def search(query: str, k: int = 3):
    query_vec = model.encode(query)
    scores = cosine_similarity(query_vec, doc_embeddings)
    ranked = np.argsort(scores)[::-1][:k]  # top-k, highest first
    return [(documents[i], float(scores[i])) for i in ranked]

for doc, score in search("I forgot my login credentials"):
    print(f"{score:.3f}  {doc}")

Run it, and you'll get something like:

0.62  How to reset your account password
0.55  How to recover a locked account
0.31  Setting up two-factor authentication

Notice what happened: the query "I forgot my login credentials" shares zero words with "How to reset your account password," yet it ranked first. A keyword search would have returned nothing. That's the payoff — you matched on meaning, not on string overlap. This shift from lexical to semantic matching is the foundation every retrieval-augmented system builds on, and it's the starting point of a structured course on RAG and retrieval-augmented generation that goes from this toy index to production retrieval.

From toy to production: what changes

The NumPy version is perfect for learning and fine for a few thousand documents. Past that, three things force an upgrade.

You need a vector database

Computing cosine similarity against every stored vector on every query is O(n) — fine at 5 documents, painful at 5 million. Vector databases solve this with Approximate Nearest Neighbor (ANN) indexes (HNSW is the common one) that trade a sliver of accuracy for enormous speed, returning near-neighbors in milliseconds over huge corpora.

You have good open-source options:

pgvector — a Postgres extension. If your data already lives in Postgres, this is often the pragmatic choice: vectors and relational data in one place, one backup story.
Chroma / Qdrant / Weaviate / Milvus — purpose-built vector stores with richer filtering and scaling stories.
FAISS — a library (not a server) from Meta for fast similarity search when you want to manage the index yourself.

A minimal Chroma example shows how little the mental model changes:

import chromadb

client = chromadb.Client()
collection = client.create_collection("docs")

collection.add(documents=documents, ids=[f"d{i}" for i in range(len(documents))])

results = collection.query(query_texts=["I forgot my login credentials"], n_results=3)
print(results["documents"])

Chroma embeds the text for you and handles the index. Same concept, production ergonomics.

Chunking matters more than the model

You rarely embed whole documents. A 30-page PDF becomes one vector that's an average of everything and a good match for nothing. In practice you chunk documents into passages (a few hundred tokens, often with slight overlap) and embed each chunk. Get chunking wrong and even a great embedding model returns mush — which is one of the most common reasons retrieval systems quietly underperform. Chunking strategy, overlap, and metadata are exactly the unglamorous details that separate a demo from a dependable system, and they're covered in depth in a course on advanced LLM integration for production apps.

Choosing and swapping embedding models

Embedding models differ in dimensionality, speed, cost, and language coverage — and critically, you must embed your corpus and your queries with the same model. Change the model and you re-embed everything. For multilingual apps (Romanian included), pick a model with strong multilingual training rather than an English-first one, or your non-English recall will suffer silently. Public benchmarks like the MTEB leaderboard on Hugging Face are the sane starting point for comparing models on retrieval quality rather than vibes.

Hybrid search: when semantic alone isn't enough

Here's a lesson that surprises people: pure semantic search is worse than keyword search for certain queries. Ask for an exact product code, a specific error like ERR_CONN_REFUSED, a person's name, or an acronym, and embeddings can betray you — they match on meaning, and a precise identifier has little semantic meaning to spread around. The embedding for ERR_CONN_REFUSED sits near "connection problems" generally, so a document about a different connection error can outrank the exact match.

The production answer is hybrid search: run both a keyword search (classic lexical scoring like BM25) and a semantic search, then combine the rankings. Keyword search nails exact terms, identifiers, and rare words; semantic search nails paraphrase and intent. Together they cover each other's blind spots.

The standard way to merge the two result lists is Reciprocal Rank Fusion (RRF) — a simple, robust formula that scores each document by its rank in each list rather than by raw scores that live on incompatible scales:

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> dict[str, float]:
    """Fuse multiple ranked lists of doc IDs into one score per doc."""
    scores: dict[str, float] = {}
    for ranking in rankings:            # e.g. [keyword_results, semantic_results]
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return dict(sorted(scores.items(), key=lambda x: x[1], reverse=True))

Because RRF only looks at positions, you don't have to normalize BM25 scores against cosine similarities — a notorious apples-to-oranges trap. Most serious vector databases (Weaviate, Qdrant, and others) now offer hybrid search with RRF built in, precisely because "just embeddings" quietly underperforms on real, messy query logs. If you take one production lesson from this article, make it this: measure your recall on real queries, and reach for hybrid the moment exact-match queries show up.

Where embeddings show up beyond search

Semantic search is the gateway, but the same vectors power a lot more:

RAG (retrieval-augmented generation) — retrieve relevant chunks by embedding similarity, then feed them to an LLM as grounding context. Embeddings are the retrieval half; without good retrieval, generation hallucinates.
Deduplication & clustering — near-duplicate detection and topic clustering fall out of distances almost for free.
Recommendations — "items similar to this one" is a nearest-neighbor query in embedding space.
Classification — embed labeled examples, then classify new items by nearest neighbors, often without training a dedicated model.

The through-line: any time you need "similar in meaning" rather than "matches exactly," embeddings are the tool. Building these features end to end — API to product, with the retrieval and orchestration wired up properly — is the spine of a hands-on course on building AI applications in Python with the OpenAI and Anthropic SDKs.

Common mistakes that cost you hours

A few traps that catch almost everyone the first time:

Re-embedding the corpus on every query. Embed documents once, store the vectors, embed only the incoming query at search time.
Mixing models. Query vectors and document vectors must come from the same embedding model. A silent mismatch produces garbage rankings with no error.
Forgetting to normalize. If you compute raw dot products instead of cosine similarity (and your vectors aren't already unit-normalized), longer texts get an unfair boost. Normalize, or use a library that does.
Embedding documents that are too large. One vector per giant document averages meaning into uselessness. Chunk first.
Trusting a single similarity threshold forever. The "good enough" cutoff depends on your model and data. Measure it on real queries; don't hardcode 0.8 because a blog post said so.

Conclusion

Vector embeddings are one of those ideas that feels like magic until you see the mechanics — and then it's just geometry. Text becomes a point in space, meaning becomes distance, and search becomes "find the nearest points." You built exactly that in a few lines of Python, from raw NumPy cosine similarity to a Chroma-backed index, and the same core idea scales from a toy corpus to millions of documents behind an ANN index.

Start where we started: embed a handful of your own documents, run a query that shares no keywords with the right answer, and watch it surface anyway. Once that clicks, RAG, recommendations, and semantic features stop looking like separate topics and start looking like one tool applied five ways. If you want the structured path from here — retrieval, chunking, evaluation, and production wiring — Cursuri-AI.ro builds it step by step.

Sources & further reading:

Sentence-Transformers — Official documentation
Hugging Face — MTEB: Massive Text Embedding Benchmark leaderboard
pgvector — Open-source vector similarity search for Postgres
Chroma — Open-source embedding database

This article is educational content. Model names, dimensions, and library APIs evolve; verify current details in the official documentation before building production systems.

Context Engineering for AI Agents: Beyond Prompt Engineering

galian — Thu, 09 Jul 2026 13:08:24 +0000

You wrote a great prompt. It worked beautifully in the playground — one question, one clean answer. Then you wired the same model into an agent that runs twenty steps, calls six tools, and reads back their output, and somewhere around step twelve it started forgetting the goal, calling the wrong tool, or confidently acting on something it misread three steps ago. The prompt didn't get worse. The context did.

This is the gap that context engineering fills. Prompt engineering is about writing one good instruction. Context engineering is about managing the entire set of tokens a model sees at inference time — across a long, multi-step run — so the signal stays high and the model keeps making good decisions. Anthropic frames it as the natural progression of prompt engineering, and if you're building anything agentic in 2026, it's the discipline that separates a demo from a system.

What context engineering actually is

Start with a precise definition. Context is the full set of tokens you include when you sample from a large language model. Not just your prompt — the system instructions, the tool definitions, the examples, the running message history, the retrieved documents, the tool results fed back in. Everything in the window.

Context engineering is the set of strategies for curating and maintaining the optimal set of those tokens during inference. The goal, in one line: find the smallest set of high-signal tokens that reliably produces the outcome you want.

The reason this is a distinct discipline from prompt engineering is the shape of the problem. A prompt is something you write once and it stays put. Context in an agent is dynamic — it grows on every turn as the model reads files, calls tools, and accumulates history. You're not authoring a static string anymore; you're managing a budget that fills up on its own, and deciding continuously what earns a place in it and what gets thrown out. That's an engineering problem, and it's the foundation the whole prompt-to-production journey builds on.

Why "just add more context" is the wrong instinct

The intuitive move, when an agent makes a mistake, is to give it more: more instructions, more examples, more history, more retrieved documents. Sometimes that helps. Very often it makes things worse, and here's why.

A model's effective attention is a finite resource. Every token you add competes with every other token for the model's limited ability to attend to what matters. Past a certain point, adding context doesn't add capability — it dilutes it. The relevant fact is now buried among ten irrelevant ones, and the model attends to the wrong thing.

This shows up empirically. The "lost in the middle" effect — documented by Liu et al. — found that models attend most reliably to information at the start and end of a long context, and least reliably to what's stuck in the middle. As context grows, retrieval of any single fact inside it gets less reliable, a degradation sometimes called context rot. A 200K-token window does not mean you should put 200K tokens in it. Capacity is not the same as attention.

So the mental model to internalize: context is a budget, not a bucket. You're not trying to fill it. You're trying to spend it on the highest-signal tokens available and refuse everything else. Every technique below is a way to enforce that discipline.

The four things competing for your window

Before the techniques, know your spenders. In a running agent, four categories of tokens fight for the same finite budget:

The system prompt — your instructions, role, constraints. Usually small, high-value, and stable.
Tool definitions — the schemas describing every tool the agent can call. These are sneakily expensive: each tool's description sits in context on every turn, whether or not it's used.
Message history — the accumulating transcript of the conversation and the agent's own steps. This is the one that grows without bound and quietly eats the window.
Retrieved / external data — documents, search results, file contents, database rows pulled in to ground the model.

The overall guidance from Anthropic is worth memorizing: keep each of these informative yet tight. Not empty — an under-specified system prompt or a missing tool leaves the model guessing. But not bloated either. The art is the calibration, and it's different for each category. Let's work through the techniques that manage them.

Technique 1: Compaction — summarize the history before it drowns you

Message history is the runaway spender. A long agent run accumulates hundreds of turns of "called tool, got 4KB of JSON back, reasoned about it, called the next tool." Most of those raw tool outputs are dead weight three steps later — you needed the conclusion, not the 4KB.

Compaction is the fix: periodically replace a chunk of verbose history with a tight summary that preserves the decisions, the key facts, and the current state, while dropping the raw noise. When the agent has finished investigating something, you don't need the full transcript of the investigation in context — you need "here's what I found and what it means for the task."

Practical rules:

Compact at natural boundaries, not mid-reasoning — after a sub-task completes, when a phase ends.
Preserve the load-bearing details: open questions, decisions made, constraints discovered, current state. Drop the intermediate chatter and the raw dumps.
Keep the goal pinned. The single most common long-run failure is the agent losing the plot on the original objective. The goal should survive every compaction.

Done well, compaction is what lets an agent run for hundreds of steps without either overflowing its window or forgetting why it started.

Technique 2: External memory — let the agent write things down

The window is not the only place to keep information. The most effective long-running agents treat the context window as working memory and offload durable state to external memory — a file, a scratchpad, a structured store the agent reads from and writes to deliberately.

Instead of carrying every fact in-context forever, the agent writes a note ("the auth module uses JWT with 15-minute expiry; the bug is in the refresh path") to a persistent store, and pulls it back only when relevant. The context window stays lean; the knowledge doesn't get lost. This is exactly how a human engineer works — you don't hold the entire codebase in your head, you keep notes and open the file when you need it.

This pattern — persistent, deliberately-managed memory outside the window — is the core of building agents that hold state over long horizons, and it's what turns a stateless model into something that can work a problem across a session without drowning.

Technique 3: Sub-agents — isolate context so the main thread stays clean

Here's a structural technique that most people never turn on. When a sub-task is going to burn a lot of tokens — "find every call site of this function across the repo," "research these five libraries" — don't do it in the main agent's context. Delegate it to a sub-agent: a separate agent instance with its own isolated window that does the noisy work and reports back a clean result.

The win is context hygiene. The thousands of tokens of file contents and search output that the investigation churns through stay in the sub-agent's context and die with it. The main agent gets back a two-paragraph summary, and its own window stays focused on the actual task instead of silting up with intermediate noise. As a bonus, independent sub-tasks run in parallel instead of serially.

The rule of thumb: delegate when work is independent, parallelizable, or context-heavy; keep it inline when it's sequential and cheap. Knowing when to fan out to sub-agents and how to orchestrate them without stepping on each other is a central skill in building AI agents and automation, and it's one of the highest-leverage context moves you have.

Technique 4: Just-in-time retrieval — pull data when needed, not upfront

There are two ways to get external data into an agent. The naive way is to preload: at the start, dump everything the agent might conceivably need into context — the whole document, all the schemas, every config file. The problem is obvious once you see it: you're spending your budget on maybes, and most of it goes unused while crowding out what matters.

The better pattern is just-in-time retrieval: give the agent the ability to fetch data (a search tool, a file reader, a database query) and let it pull exactly what it needs, exactly when it needs it. Instead of "here are all 40 files," it's "here's a tool to read a file — go get the one you need." The agent loads the relevant chunk into context at the moment of use, acts on it, and (with compaction) lets it fall away afterward.

This mirrors how retrieval-augmented systems already work, and getting the retrieval layer right — what to fetch, how to rank it, how much to bring back — is where advanced LLM integration earns its keep. Preloading feels safer; just-in-time is what actually scales.

Technique 5: Tool curation — the failure mode hiding in plain sight

Remember that tool definitions sit in context on every turn. That makes a bloated tool set a double tax: it burns tokens continuously, and it degrades decisions. Anthropic calls out one of the most common failure modes directly — tool sets that cover too much functionality or create ambiguous, overlapping choices about which tool to use.

The tell is a sharp one: if a human engineer can't say for certain which tool should be used in a given situation, an AI agent can't either. Fifteen tools with fuzzy, overlapping responsibilities will produce worse behavior than six sharp, non-overlapping ones — and cost more tokens doing it.

So curate the toolbox like you'd curate an API:

Fewer, sharper tools. Each with a clear, distinct job and an unambiguous "use this when…"
No overlap. Two tools that could both plausibly handle the same request is a decision point where the agent will sometimes pick wrong.
Prune ruthlessly. A tool that's rarely the right choice is paying rent in your context on every single turn. Cut it.

Tool curation is the least glamorous technique here and often the highest-ROI. It's pure subtraction, and subtraction is exactly what context engineering rewards.

How you know it's working: measure it

Every technique above is a change to your context, and changes to context are exactly the kind of thing that feels better while being worse — or vice versa. If your only signal is "I ran a few queries and it seemed fine," you're tuning blind, and you'll ship a regression the day you compact one turn too aggressively and the agent starts forgetting a constraint.

The fix is the same as anywhere in production ML: an evaluation set. Assemble representative tasks with known good outcomes, and re-run them every time you change how context is managed. Then you can say "compaction at phase boundaries held task success at 0.9 while cutting average tokens 40%" instead of "I think it's better now." Treating evaluation as first-class rather than an afterthought is what turns context engineering from a craft into engineering — a number that moves when you change something, not a vibe.

Putting it together

Context engineering isn't a framework you install; it's a posture you adopt toward the model's window. The whole discipline collapses to one principle applied relentlessly: spend the finite budget on the smallest set of high-signal tokens that does the job.

In practice, for a real agent, that means:

Compact the history at natural boundaries so a long run doesn't drown in its own transcript.
Offload durable state to external memory instead of carrying it forever.
Delegate noisy, independent work to sub-agents so the main window stays clean.
Retrieve just-in-time instead of preloading everything you might need.
Curate the tools hard — fewer, sharper, no overlap.
Measure with evals so every change is verified, not hoped.

None of these are exotic, and that's the point. The model is already capable. What makes the capability hold up over a twenty-step run isn't a cleverer prompt — it's disciplined management of everything the model reads along the way.

Conclusion

Prompt engineering taught us to write one good instruction. Context engineering is what you need the moment that instruction has to survive a long, tool-using, self-accumulating agent run — which is to say, the moment you build anything real. The failure you saw at step twelve was never the model getting dumber. It was the context getting noisier, and no one curating it.

Adopt the budget mindset, apply the five techniques, and put an eval set behind every change. Do that, and the same model that fell apart at step twelve will run to step fifty and still know exactly what it's doing — because you engineered what it was looking at the whole way.

The courses linked throughout are part of Cursuri-AI.ro, an AI-learning platform with hands-on, current tracks on context engineering, AI agents, LLM integration, and evaluating AI systems in production.

Sources & further reading:

Anthropic — Effective context engineering for AI agents (definition, the four context components, tool-set failure modes, compaction and memory)
Anthropic — Effective harnesses for long-running agents
Liu et al. — Lost in the Middle: How Language Models Use Long Contexts

This article is educational content. Model behavior, context limits, and tooling evolve quickly; validate approaches against your own workloads and current official documentation.

Run LLMs Locally with Ollama in 2026: The Practical Developer Guide

galian — Sun, 05 Jul 2026 21:11:24 +0000

For years, "run the model locally" was the option you mentioned and then didn't take: the models were too weak, the tooling too fiddly, and the cloud APIs too convenient. In 2026 that calculus has genuinely shifted. Open-weight models in the 12–35B range now handle real coding and agent workloads, Apple Silicon got a dedicated inference engine, and Ollama quietly became a drop-in backend for the same tools you already use against cloud APIs — including Claude Code.

I teach practical AI engineering at Cursuri-AI.ro, Eastern Europe's AI education platform, and local inference has gone from a curiosity module to one of the questions companies ask us most — usually spelled "how do we use LLMs without sending our data anywhere?" This guide is the answer I give developers: what changed, what hardware you actually need, which models are worth pulling, and how to plug it all into a real workflow.

As always with this space: versions and model rankings move monthly. Everything below is verified against Ollama's official blog and docs as of early July 2026 — re-check before you build a budget or an architecture on it.

Why local, and why now

Three arguments have survived contact with production; the rest is mostly vibes.

Privacy and data residency. With a local model, prompts and outputs never leave your machine (or your VPC, if you self-host on a server). For anyone dealing with client data, medical text, legal documents, or GDPR-sensitive workloads, this eliminates the entire "what does the provider do with my data" conversation instead of managing it through contracts. This is the single biggest adoption driver we see in Europe, and it's the backbone of our course on local LLMs, self-hosting, and privacy.

Cost shape. Cloud APIs bill per token; local inference bills you once, in hardware you may already own. For high-volume, latency-tolerant workloads — batch classification, summarization pipelines, internal tooling — a mid-range GPU that's already on someone's desk can absorb work that would otherwise be a real monthly line item. (For low-volume or frontier-quality work, cloud still wins. More on that below.)

No external dependency. A local model doesn't get deprecated, rate-limited, price-changed, or suspended out from under you. After the model-availability surprises of the last year, "at least one workload runs on weights we control" has become a reasonable line item in a resilience plan, not paranoia.

What actually changed in Ollama in 2026

If you last touched Ollama when it was "a nice wrapper around llama.cpp," the 2026 releases are the reason to look again. All of this is from Ollama's official blog:

Anthropic API compatibility (January, v0.14.0). Ollama now exposes a native Anthropic-style /v1/messages endpoint. This is the sleeper feature of the year: Anthropic-native tools — most notably Claude Code — can talk to a local model directly, with no proxy or translation layer. There's a matching OpenAI-compatible endpoint too, so Codex and OpenAI-SDK apps work the same way.
ollama launch (January). A single command that configures and starts a coding agent against a local model — ollama launch claude sets up Claude Code, prompts you to pick a model, and you're in.
Experimental image generation (January). Early days, but the scope of "local model" is no longer text-only.
MLX engine on Apple Silicon (March preview → June release). Ollama moved its Mac inference path to Apple's MLX framework, which exploits unified memory. Ollama's own framing for the June release: its highest performance on Apple Silicon yet — faster output with reduced memory usage.
Ollama 0.30 and 0.31 (June). Version 0.30 brought improved performance and broader GGUF model compatibility through llama.cpp; 0.31 made Gemma 4 significantly faster on Apple Silicon via multi-token prediction (MTP), enabled by default.

The theme is clear: Ollama is positioning itself less as a hobbyist toy and more as the standard local backend for agentic tooling.

Getting started in five minutes

Install (macOS and Windows have installers at ollama.com/download; on Linux):

curl -fsSL https://ollama.com/install.sh | sh

Pull and run a model:

ollama pull gemma4
ollama run gemma4

That's a working local chat. Ollama also starts a local server on port 11434, which is where the interesting part begins — every API-based tool you have can point at it.

Useful daily commands: ollama ls (installed models), ollama ps (what's loaded and where — CPU vs GPU), ollama rm <model> (free disk space; models are multi-gigabyte).

Hardware: the honest sizing guide

The rule of thumb that matters: a model quantized to 4 bits needs very roughly 0.5–0.7 GB of memory per billion parameters, plus overhead for context. Everything else follows from that.

Your hardware	What runs comfortably	Experience
8 GB RAM, no GPU	3–8B models, quantized	Fine for chat, drafting, classification. Slow but usable on CPU
16 GB RAM (Apple Silicon)	8–14B models	Good daily-driver territory; MLX made this tier notably faster in 2026
24 GB+ (M-series Pro/Max or a 24 GB GPU)	27–35B models	Where local coding models get genuinely useful
48 GB+ unified memory / multi-GPU	Large MoE models	Server-class local inference

Two nuances that save people disappointment:

Quantization is why any of this works. Models ship in compressed 4–8 bit variants (the GGUF ecosystem) that trade a small quality loss for a 2–4× memory reduction. Ollama's default tags are already quantized — you rarely need to think about it, but it explains why a "27B model" fits in 24 GB.
Mixture-of-experts (MoE) models need memory for their total parameters but compute like their active subset. NVIDIA's Nemotron-3-Super, for example, is a 120B model with 12B active parameters: it runs faster than its size suggests, but you still need the RAM to hold it.

Context length eats memory too — an agent session with 32K+ tokens of context adds real overhead on top of the weights. If you're sizing for coding agents, budget for that, not just the model file.

The mid-2026 open-weight lineup worth knowing

Rankings churn monthly, so treat this as a map, not a leaderboard. From Ollama's model library, the families that matter right now:

Gemma 4 (12B–31B) — Google's open family, currently the most-pulled model on Ollama. Multimodal, tuned for reasoning and agentic work, and the main beneficiary of the MLX/MTP speedups on Macs.
Qwen3.5 / Qwen3.6 (0.8B–122B) — the ecosystem's Swiss army knife. Qwen3.5 spans everything from edge-tiny to server-large; Qwen3.6 (27B–35B) focuses on agentic coding. qwen3-coder is Ollama's own recommendation for coding-agent use.
GLM-5 family — flagship-class open weights (GLM-5 is 744B total / 40B active); strong at coding and long-horizon tasks. Too big for most desktops locally, but available as :cloud variants (see below) and self-hostable on serious hardware.
Nemotron-3-Super (120B MoE, 12B active) — NVIDIA's entry, aimed at multi-agent applications.
MiniMax-M3 — notable for a 1M-token context window, if your workload is long-document analysis.
Specialists: GLM-OCR for document understanding, TranslateGemma (4B–27B, 55 languages) for translation, LFM2 (24B) for on-device deployment, Ornith (9B–35B) for agentic coding.

Sensible defaults: on a 16 GB Mac, start with gemma4:12b. On 24 GB+, try qwen3-coder for code and gemma4:27b for general work. Then run your tasks on them — a model's rank on someone's benchmark tells you little about your use case.

The part that changes your workflow: Ollama as a drop-in API

Ollama's server speaks both major API dialects, which means "switch to a local model" is now a base-URL change, not a rewrite.

OpenAI-compatible (/v1) — any OpenAI-SDK app works:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resp = client.chat.completions.create(
    model="gemma4",
    messages=[{"role": "user", "content": "Explain GGUF quantization in one paragraph."}],
)
print(resp.choices[0].message.content)

Anthropic-compatible (/v1/messages) — and this is the one with teeth, because it means Claude Code runs against local models. Per Ollama's official docs:

export ANTHROPIC_AUTH_TOKEN=ollama       # accepted but not validated
export ANTHROPIC_BASE_URL=http://localhost:11434

claude --model qwen3-coder

Or let Ollama do the wiring for you:

ollama launch claude

Honest caveats before you get excited:

Ollama recommends at least 32K tokens of context for Claude Code, and its model suggestions for coding are qwen3-coder locally (30B — you want 24 GB+ of VRAM/unified memory) or glm-4.7:cloud / minimax-m2.1:cloud via Ollama's cloud, which keeps the same API surface but runs the weights remotely.
The compatibility layer doesn't cover everything: no prompt caching, no token-counting endpoint, no forced tool selection, no batches API, no PDF inputs (images must be base64). If your workflow leans on those, you'll feel it.
A 30B local model is not Opus, and it isn't trying to be. It's "capable pair of hands on an airplane / on confidential code," not "frontier reasoning."

The pattern that actually works in practice is routing: local models for the private, high-volume, or offline work; frontier cloud models for the hard reasoning. Deciding which tier a task belongs to — and building the escalation path — is an architecture skill, and it's exactly the kind of decision we drill in our AI system architecture course.

When local is the wrong choice

Being a fan of local inference means knowing where it loses:

Frontier-quality reasoning. For the hardest tasks, top cloud models remain clearly ahead of anything you can run on a workstation. If wrong answers are expensive, don't fight this.
Low-volume workloads. If you make a few thousand API calls a month, per-token billing is cheaper than any GPU. Local pays off at volume, at privacy constraints, or at both.
Ops you don't want. A self-hosted model is a service you now run: updates, monitoring, capacity. ollama run on a laptop is trivial; a team-wide inference server is real infrastructure.
Multimodal breadth and long-tail capabilities. Cloud APIs still bundle more (native PDF understanding, larger tool ecosystems, batch APIs) than the local stack replicates.

One more thing people conflate: running a model locally is different from customizing one. If your actual goal is a model that speaks your domain language or follows your house style, that's a fine-tuning question — LoRA adapters on an open-weight base, then serving the result through Ollama. That pipeline (when to fine-tune vs when to just engineer the prompt) is its own discipline, covered in our fine-tuning course.

Frequently asked questions

Is Ollama free?

The tool itself is open source and free. The models each carry their own licenses — Gemma, Qwen, GLM and friends have different terms, some with restrictions on commercial use. Check the license tab on the model's Ollama page before you ship a product on it. Ollama's optional cloud models are a paid service.

What hardware do I need to run LLMs locally in 2026?

As a rule of thumb at 4-bit quantization: 8 GB of RAM runs 3–8B models, 16 GB runs 8–14B comfortably (especially on Apple Silicon with the MLX engine), and 24 GB+ opens up the 27–35B class where local coding models get genuinely useful. More context = more memory on top of the weights.

Can a local model replace GPT or Claude?

For a growing set of tasks — summarization, classification, drafting, routine coding on mid-size codebases — yes, credibly. For frontier reasoning and the highest-stakes accuracy, no. Production teams typically route: local for private/high-volume work, cloud for the hard 10%.

Can I really use Claude Code with Ollama?

Yes. Since Ollama v0.14.0 (January 2026) there's native Anthropic Messages API compatibility: set ANTHROPIC_BASE_URL=http://localhost:11434, run claude --model qwen3-coder — or just ollama launch claude. Expect a capable assistant, not Opus-level reasoning, and note that prompt caching and a few other API features aren't supported through the compatibility layer.

Ollama vs llama.cpp vs vLLM — which should I use?

Ollama for developer experience: one command, model management, dual API compatibility. llama.cpp (which powers Ollama's GGUF path) for maximum control and minimal footprint. vLLM for high-throughput multi-user serving on server GPUs. Most developers should start with Ollama and only move down the stack when they hit a concrete limit.

The skill underneath the tool

Here's the uncomfortable part: pulling a model is the easy 5%. The value shows up when you can answer the questions around it — which model for which task, how to measure whether the local model is good enough for your workload instead of guessing, how to build the routing and fallback so privacy-sensitive work stays local while hard problems escalate to a frontier model. That's engineering judgment, not tooling trivia.

That judgment is what we teach at our AI education platform — hands-on courses built around real repositories and an interactive AI instructor, covering the full local-to-cloud spectrum: self-hosting and privacy, fine-tuning, architecture, and the agentic workflow on top.

Conclusion

In 2026, local LLMs crossed the line from hobby to infrastructure option. Ollama's dual API compatibility means your existing tools — including Claude Code — can run against open weights with a base-URL change; the MLX engine made a 16 GB MacBook a legitimate inference machine; and the open-weight lineup in the 12–35B range is good enough for a real slice of production work.

The play isn't "cancel your API keys." It's knowing which slice of your workload belongs on weights you control — then running it there deliberately, measured, with an escalation path for everything else. Start with ollama run gemma4 tonight; you're one evening away from having an informed opinion.

Written by the team at Cursuri-AI.ro — practical, hands-on AI engineering courses for developers and professionals across Eastern Europe, from local LLMs and self-hosting to agentic coding, evals, and AI system architecture.

Sources: Ollama Blog · Anthropic API compatibility — Ollama Docs · Claude Code with Anthropic API compatibility — Ollama · Ollama Model Library · Download Ollama

Claude Sonnet 5 Just Made Running Agents Cheap — What Builders Actually Need to Know

galian — Tue, 30 Jun 2026 22:06:37 +0000

Anthropic shipped Claude Sonnet 5 on June 30, 2026, and the framing in the announcement is unusually blunt for a model launch: it's pitched as the most agentic Sonnet yet — a model built to make plans, drive tools like browsers and terminals, and run autonomously at a level that, a few months ago, took something bigger and more expensive.

For anyone building on top of these models — agents, pipelines, coding tools — that's the headline that matters. Not "it's smarter," but "near-frontier capability just got cheaper to run in a loop." I write and teach about agentic engineering at Cursuri-AI.ro, Eastern Europe's AI education platform, so I'll keep this grounded in what changes for people who actually ship on these APIs — not the launch-day benchmark theater.

One disclaimer up front: model pricing and availability in this space change almost monthly, and this is a day-one snapshot. Verify the current numbers on Anthropic's official pages before you wire anything to a budget. I'm deliberately not quoting benchmark scores here — the launch materials presented them in a way that's easy to misread, so for hard numbers go straight to the Sonnet 5 System Card.

The one-sentence version

Sonnet 5 moves "good enough to run agents autonomously" down a price tier — and ships a new tokenizer that can quietly inflate your token counts by up to 35%.

Both halves of that sentence matter, and the second one is the part nobody puts on a launch slide. Let's take them in order.

What's actually new

Stripping the marketing down to verifiable claims from Anthropic's own announcement, here's what Sonnet 5 is:

The most agentic Sonnet so far. It's described as able to "make plans, use tools like browsers and terminals, and run autonomously," with improvements specifically in multi-step tool use — the exact workload that defines an agent rather than a chatbot.
Close to Opus 4.8 — at a lower price. Anthropic's own phrasing is that its "performance is close to that of Opus 4.8, but at lower prices." That's the whole pitch: most of the capability, a fraction of the cost.
A real step up from Sonnet 4.6. Called a "substantial improvement over its predecessor, Sonnet 4.6, on important aspects of agentic performance like reasoning, tool use, coding, and knowledge work."
Safer in agentic contexts. Anthropic reports an "overall lower rate of undesirable behaviors than Sonnet 4.6," plus lower rates of hallucination and sycophancy — which matters more than it sounds when a model is acting in a loop without a human reading every step.
Deliberately weaker at offensive cyber. It shows "substantially poorer performance than models such as Opus 4.8" on dangerous cyber tasks and was "never able to develop a full working exploit." That's a safety design choice, not an oversight — worth knowing if security tooling is your domain.

Two things Anthropic did not publish that I'm not going to invent for you: an official context window and max output token figure for Sonnet 5 weren't stated in the launch materials at the time of writing. If you need those for capacity planning, pull them from the official API docs rather than trusting a blog (including this one). Guessing is how teams ship broken truncation logic.

The economics shift is the real story

Here's why builders should care more than end users.

When you chat with a model, price-per-token is almost noise — you send a few thousand tokens and read the answer. When you run an agent, the model is in a loop: read context, call a tool, read the result, reason, call another tool, repeat. A single "task" can burn hundreds of thousands of tokens across dozens of turns. At that volume, the price-per-million-tokens line is your unit economics.

So a model that lands near Opus-4.8 quality at Sonnet pricing doesn't just make chat cheaper — it changes which agent designs are economically viable at all. Workflows you'd previously gate behind Opus (multi-step research, autonomous refactors, long tool-using runs) become defensible on a Sonnet budget. That's the unlock.

Here's the day-one pricing picture, with the rest of the current Anthropic lineup for context:

Model	Input / 1M tokens	Output / 1M tokens	Notes
Sonnet 5 (intro, through Aug 31 2026)	$2	$10	Promotional launch pricing
Sonnet 5 (standard, from Sep 1 2026)	$3	$15	Same as Sonnet 4.6's tier
Opus 4.8	$5	$25	Top accuracy; default in Claude Code
Haiku 4.5	$1	$5	Cheapest / fastest tier

A few honest notes:

The introductory $2 / $10 runs through August 31, 2026, then settles to $3 / $15 — the same standard tier Sonnet has occupied. So the long-run story isn't "Sonnet got cheaper"; it's "the Sonnet tier got dramatically more capable for the same price."
Sonnet 5 is the default model on Free and Pro plans, and is available to Max, Team, and Enterprise users — in Claude Code, the Claude platform, and the API. So if you're on Claude Code, you may already be one model-switch away from it.
Against Opus 4.8 the price ratio is roughly 1.7× (output $25 vs $15). When you're running agents at scale, that multiple compounds fast — which is exactly why the "close to Opus" claim is worth pressure-testing on your workload, not taking on faith.

The tokenizer gotcha that will mess up your cost math

This is the part I most want builders to internalize, because it's the easiest way to get a nasty surprise on your next invoice.

Sonnet 5 ships with an updated tokenizer. Anthropic states that the same input text now maps to roughly 1.0–1.35× as many tokens as before, depending on content type. Read that again: identical prompts can cost up to 35% more tokens on Sonnet 5 than the token count you measured on an older model — before any change in per-token price.

Why it bites:

Your cost dashboards, budget alerts, and per-request estimates were calibrated on the old tokenizer. Swap the model without re-measuring and your "same" workload silently costs more.
Code, structured data (JSON/XML), and non-English text tend to sit at the higher end of that multiplier — and those are precisely the inputs agentic and coding workloads are made of.
It interacts with context windows and truncation: more tokens for the same text means you hit limits sooner than your old math predicts.

The fix is boring and non-negotiable: re-baseline. Before you flip production traffic to Sonnet 5, measure real token counts on a representative sample of your prompts with the new tokenizer, recompute cost per task, and update your budgets and alerts. The headline price drop is real — but the effective saving is (price delta) × (token inflation), and you can't know the second factor without measuring. Anyone who tells you "it's 33% cheaper" did half the arithmetic.

This is also where good evals earn their keep. A model swap isn't just a cost change; it's a behavior change. Run your task suite on Sonnet 5 against the model you're replacing before you commit — quality, tool-call success rate, and cost together. If you don't have an eval harness yet, this is the launch that should convince you to build one; it's a discipline we treat as core, not optional, in our course on building LLM evals for production.

When to still reach for Opus 4.8

"Close to Opus" is not "Opus." The honest read on where Sonnet 5 fits:

Reach for Sonnet 5 as your default agent workhorse: high-volume tool-using loops, coding assistance, research and summarization, anything where you're paying per turn and the marginal quality of Opus isn't worth ~1.7× the output cost.
Stay on Opus 4.8 for the hardest reasoning, the highest-stakes accuracy, and security-sensitive work where Sonnet 5 is intentionally weaker (offensive-cyber tasks). When a wrong answer is expensive, the price gap is cheap insurance.

The pattern most production teams land on isn't "pick one." It's a router: Sonnet 5 handles the bulk of turns, and you escalate to Opus 4.8 for the steps that genuinely need it — with a human in the loop on the consequential ones. Getting that routing logic right (and knowing which task belongs in which tier) is a real engineering skill, and it's the through-line of our model-comparison course, which treats "which model for which job" as a decision you make with data rather than vibes.

A pragmatic migration checklist

If you're considering moving an agent or pipeline to Sonnet 5, here's the order I'd do it in:

Re-baseline tokens. Run a representative sample through the new tokenizer. Recompute cost per task. Update budget alerts.
Run your evals. Quality, tool-call success, latency, and cost, head-to-head against the model you're replacing. No eval suite? Build a small one first — even 30 representative tasks beats a gut call.
Shadow, then canary. Route a slice of real traffic to Sonnet 5, compare outputs, then scale gradually. Don't flip 100% on day one.
Keep an escalation path. Wire Opus 4.8 as the fallback for tasks that fail Sonnet 5's quality bar. Routing beats an all-or-nothing bet.
Re-read your safety posture. Lower hallucination and sycophancy is good news for autonomous runs, but "safer" isn't "supervise nothing." Keep guardrails and human checkpoints where consequences are real.

None of this is exotic. It's the same discipline that separates teams who run agents in production from teams who demo them — and it's exactly the muscle we build in our hands-on track on AI agents and automation, taught around real repositories rather than toy notebooks.

Frequently asked questions

Is Claude Sonnet 5 better than Opus 4.8?

Not across the board. Anthropic positions Sonnet 5's performance as close to Opus 4.8 at a lower price — so for high-volume agentic and coding work it's often the better value, but Opus 4.8 still leads on the hardest reasoning, top-end accuracy, and (deliberately) on offensive-cyber capability. Match the tier to the task instead of picking a favorite.

How much does Claude Sonnet 5 cost?

It launched with introductory pricing of $2 per million input tokens and $10 per million output tokens through August 31, 2026, then moves to a standard $3 / $15 — the same tier Sonnet 4.6 occupied. Your effective cost also depends on the new tokenizer (see below), so measure before you budget.

Does the new tokenizer really change my costs?

Yes. Anthropic states the same input can map to roughly 1.0–1.35× as many tokens under Sonnet 5's updated tokenizer, depending on content type — code and structured data sit at the higher end. Re-measure your real prompts before assuming the headline price drop equals your actual saving.

Can I use Sonnet 5 in Claude Code?

Yes. It's available in Claude Code, the Claude platform, and the API, and it's the default model on Free and Pro plans (and available to Max, Team, and Enterprise). If you're already in Claude Code, switching is a model selection, not a migration.

Should I migrate my agents to Sonnet 5 immediately?

Don't flip production on day one. Re-baseline token counts, run your eval suite head-to-head against your current model, then canary a slice of traffic before scaling — and keep an escalation path to Opus 4.8 for tasks that need it.

The skill underneath the model

Here's the part the launch posts skip: a cheaper, more agentic model doesn't make anyone a better builder. It just makes the consequences of your design bigger — cheaper to be right at scale, and cheaper to be confidently wrong at scale. Point Sonnet 5's autonomy at a vague spec and you get a fast, plausible wall of actions you didn't design and can't fully audit.

The developers getting real leverage from this launch aren't the ones who memorized the new price-per-token. They're the ones who understand agent architecture, context engineering, evals, and cost modeling well enough to know when the cheap-and-autonomous option is the right call and when it's a trap. That foundation — taught around real repositories with an interactive AI instructor, not slide decks — is what we build at our Eastern European AI education platform, including a dedicated, hands-on track on agentic coding with Claude Code.

Conclusion

Claude Sonnet 5 is a genuinely significant release for builders, but not for the reason most coverage leads with. The story isn't a benchmark number — it's that near-frontier agentic capability just moved down a price tier, which changes which agent designs are economically worth shipping. The catch is the new tokenizer: the real saving is the price drop minus token inflation, and you only learn the second number by measuring.

So don't migrate on the headline. Re-baseline your tokens, run your evals, canary your traffic, and keep Opus 4.8 one route away for the work that needs it. Do that, and Sonnet 5 is one of the better deals in the 2026 model lineup. Skip it, and you'll find out the hard way — on your invoice.

Written by the team at Cursuri-AI.ro — practical, hands-on AI engineering courses for developers and professionals across Eastern Europe, from agentic coding and AI agents to evals, context engineering, and the modern AI-native workflow.

Sources: Introducing Claude Sonnet 5 — Anthropic · Claude Sonnet 5 System Card · Claude Platform — Pricing · Claude Pricing

DEV Community: galian

The Model in Your Config Has an Expiry Date — a Runbook for LLM Deprecations

The vocabulary you're missing two-thirds of

The dates are published before the deprecation, and almost nobody reads them

Aliases are a strategy, not a shield

It is never just the model string

The runbook

1. Inventory: find every model string you own

2. Centralize the strings

3. A canary that fails loudly and cheaply

4. The eval gate, built before you need it

5. Migrate the request, then the prompt, then the model

What the provider actually owes you, and what it doesn't

The checklist

Your LLM Stream Works on localhost and Dies in Production — Fixing SSE End to End

First: streaming is not a UX nicety anymore

Break #1: your proxy is holding the tokens hostage

Break #2: the load balancer kills you during the pause

Break #3: the errors arrive with HTTP 200

Break #4: the user left and you're still paying

Break #5: EventSource can't do what your chat app needs

Break #6: partial output is a state you have to design for

The checklist

Anthropic Filed to Go Public. Here's What Actually Changes for Developers Building on Claude

First, the numbers that are actually confirmed

Consequence #1: your vendor gets a quarterly clock

Consequence #2: model retirement becomes an operational risk, not a footnote

Consequence #3: the S-1 will be the best vendor due-diligence document ever published

Consequence #4: "which model" stops being a taste question

What I'd actually do this quarter

The bottom line

The EU AI Act's Transparency Rules Went Live — Here's What You Actually Have to Ship

First: does this apply to you?

Obligation 1 — tell people they're talking to an AI

Obligation 2 — mark synthetic output so machines can detect it

Obligation 3 — emotion recognition and biometric categorisation

Obligation 4 — deepfakes and public-interest text

The deadlines, which are not all the same

A ship checklist

Conclusion

Claude Opus 5 Is Out — Here Are the API Changes That Will Actually Break Your Code

The spec sheet, in one table

Change #1: thinking is on by default

Change #2: the actual breaking change — disabled thinking + high effort = 400

Change #3: effort is now the control lever that matters

Change #4: the model behaves differently, even if you change nothing

The smaller changes you'll actually be happy about

The migration checklist, in the order things break

The bottom line

Multi-Agent AI Systems: When One Agent Isn't Enough — and When Two Is Already Too Many

What a multi-agent system actually is

The three things fan-out actually buys you

What it costs you

The case against — and it's a strong one

The orchestration patterns that survive production

The failure modes nobody demos

A decision checklist

Conclusion

We just launched our AI course catalog in English — 20 hands-on courses with an AI professor in every lesson

The idea: learning AI should feel like working with AI

What's in the English catalog

IT & Engineering track (10 courses)

Business & Professionals track (10 courses)

Pricing — simple and honest

Who this is for (and who it isn't for)

The ask

Fine-Tuning vs RAG vs Prompting: How to Actually Decide in 2026

The one distinction that resolves most arguments

Always start with prompting (yes, even now)

Reach for RAG when the problem is knowledge

Fine-tune when the problem is behavior — and only then

The combinations are the real answer

The decision, in one pass

Conclusion

Vector Embeddings Explained: Build Semantic Search in Python

What an embedding actually is

Cosine similarity: the one operation you'll use everywhere

Build a semantic search engine from scratch

Step 1 — Embed a corpus

Step 2 — Cosine similarity in NumPy

Break #5: `EventSource` can't do what your chat app needs

Change #3: `effort` is now the control lever that matters