DEV Community: llmops

Building a permission-aware MCP server in Python

sam ros — Tue, 13 Oct 2026 15:59:16 +0000

Atlassian's rebuilt MCP server, now generally available, enforces three things on every call. It uses OAuth 2.1 with PKCE, so the agent only gets the user's existing permissions. It sorts tools into read, write and destructive, so clients can ask for confirmation. And it applies admin-level scopes and audit logging. This article shows how to build the same three controls into your own MCP server with the official Python SDK, following the current MCP specification (2026-07-28).

What you'll build: a "tickets" MCP server where:

every request carries a token issued for this server
each tool checks its own scope
read, write and destructive tools are labelled so the client can gate them

1. Wire up token verification and audience checks

The MCP server is an OAuth 2.1 resource server. It never logs users in itself. It validates tokens issued by your authorization server (Okta, Entra ID, Keycloak and the like). The spec makes two checks mandatory:

Audience. Servers MUST validate that access tokens were issued specifically for them as the intended audience (RFC 8707).
No foreign tokens. Servers MUST NOT accept or transit any other tokens.

from pydantic import AnyHttpUrl
from mcp.server import MCPServer
from mcp.server.auth.provider import AccessToken, TokenVerifier
from mcp.server.auth.settings import AuthSettings

RESOURCE = "https://mcp.example.com/mcp"  # canonical URI of this server, no trailing slash

class JwtVerifier(TokenVerifier):
    async def verify_token(self, token: str) -> AccessToken | None:
        claims = validate_jwt(token)  # YOUR code: JWKS signature, exp, iss checks
        if claims is None:
            return None
        return AccessToken(
            token=token,
            client_id=claims["client_id"],
            scopes=claims.get("scope", "").split(),
            expires_at=claims["exp"],
            resource=claims["aud"],   # RFC 8707 resource indicator / audience
            subject=claims["sub"],
        )

mcp = MCPServer(
    "tickets",
    token_verifier=JwtVerifier(),
    auth=AuthSettings(
        issuer_url=AnyHttpUrl("https://auth.example.com"),
        resource_server_url=AnyHttpUrl(RESOURCE),
        required_scopes=["tickets:read"],
        validate_token_resource=True,
    ),
)

Three details matter here.

Set validate_token_resource=True explicitly. In the current SDK source, leaving it unset while resource_server_url is set logs a warning and behaves as False, so tokens minted for other services would be accepted. The SDK notes that 3.0 will make True the default. If your verifier already checks the audience itself, set it to False on purpose rather than leaving it unset.
aud can be a list. If your IdP issues multi-audience tokens, the SDK's docstring says to set resource to the entry that equals resource_server_url.
resource_server_url also drives discovery. The SDK uses it as the base route for OAuth Protected Resource Metadata (RFC 9728). The spec requires servers to publish this metadata so clients can find your authorization server from the WWW-Authenticate header on a 401.

2. Enforce scopes per tool, not just at the door

required_scopes is a gate for the whole server. Real systems need finer control: read for search, write for updates, a separate scope for deletes. The SDK exposes the current token inside a tool. Raising ToolError returns an isError: true result the model can read and act on:

from mcp.server.auth.middleware.auth_context import get_access_token
from mcp.server.mcpserver.exceptions import ToolError

def require_scope(scope: str) -> AccessToken:
    token = get_access_token()
    if token is None or scope not in token.scopes:
        raise ToolError(f"This action needs the '{scope}' scope. Re-authorize to grant it.")
    return token

Using ToolError (not a generic exception) matters. In the SDK, any other exception is treated as a crash, and the model sees only Error executing tool <name>. That gives the agent nothing to recover from.

At the HTTP level, the spec describes the full step-up flow for insufficient scope:

respond 403 with WWW-Authenticate: Bearer error="insufficient_scope", scope="...", resource_metadata="..."
include all the scopes the operation needs in a single challenge, not one per retry, which would force several re-authorization round-trips for one action

If you front the server with a gateway that can inspect the tool being called, that is the place to issue the challenge. The in-tool check above is the minimum.

3. Label tools by risk with annotations

from mcp.types import ToolAnnotations

@mcp.tool(title="Search tickets",
          annotations=ToolAnnotations(read_only_hint=True, open_world_hint=False))
def search_tickets(query: str) -> list[dict]:
    require_scope("tickets:read")
    ...

@mcp.tool(title="Close ticket",
          annotations=ToolAnnotations(read_only_hint=False, destructive_hint=False,
                                      idempotent_hint=True))
def close_ticket(ticket_id: str) -> str:
    require_scope("tickets:write")
    ...

@mcp.tool(title="Delete ticket",
          annotations=ToolAnnotations(read_only_hint=False, destructive_hint=True))
def delete_ticket(ticket_id: str) -> str:
    require_scope("tickets:admin")
    ...

Annotations let clients show a confirmation prompt before delete_ticket runs, the same read/write/destructive split Atlassian describes. Remember what annotations are:

Hints, not security. The spec says clients MUST treat tool annotations as untrusted unless they come from trusted servers. The SDK docs say not to rely on clients honouring them. The scope check inside each tool is the real control.
Only meaningful for writes. destructive_hint and idempotent_hint only apply to tools that write, which is why the read-only tool leaves them out.

Spec details that affect cost and latency

Return only the tools a caller can use. The spec allows tools/list to vary by the authorization on the request, for example listing only the tools the caller's granted scopes permit. Every tool definition the client loads is context the model pays for on each turn. Hiding delete_ticket from read-only callers cuts prompt tokens as well as risk.

Keep tool order deterministic. The spec says servers SHOULD return tools in the same order when the set hasn't changed. It notes that this improves LLM prompt-cache hit rates when tools are included in model context. Shuffled tool lists quietly defeat prefix caching.

Never forward the user's token downstream. If close_ticket calls your ticketing API, call it with the server's own credentials, or with a token obtained through a proper exchange. Passing the incoming token through violates the spec's "MUST NOT ... transit any other tokens" rule.

Verify it before you ship

Run these four checks against the server:

A request with no token returns 401, and the WWW-Authenticate header contains resource_metadata="https://mcp.example.com/.well-known/oauth-protected-resource".
Protected resource metadata is served at that URL and lists your authorization server.
A valid token issued for a different audience is rejected with 401.
A tickets:read-only token calling close_ticket gets an isError: true result with a message naming the missing scope, not a generic crash.

Note: the code follows the official MCP Python SDK documentation and source (main branch, checked October 13, 2026). validate_jwt stands in for your identity provider's JWT validation. Pin your SDK version, because names like MCPServer replaced earlier ones.

Sources

Originally published on InferPrice, which tracks GPU cloud and LLM API prices daily.

Running LLM-as-a-judge on live traffic without blowing your budget

Russell Van Curen — Tue, 13 Oct 2026 14:59:55 +0000

Offline evals tell you a model did well on the examples you thought of. Live traffic tells you what it's actually doing. The obvious way to watch live traffic is to have a strong model grade every response. That's also the fastest way to make evals cost more than the feature they're watching.

This guide covers how to run an LLM judge on production traffic at a cost you won't notice. The principles apply to any setup. Where it helps, we show how it works in TokenRouter's monitors, which we build.

Principle 1: deterministic checks on everything, the judge on a sample

Some failures don't need judgment. A leaked API key matches a regex. A broken JSON response fails a parser. Those checks are cheap, so run them inline on every request and block when they match.

A judge model is for judgment calls. "Does this answer cite a document we don't have?" "Did it follow an instruction injected through a retrieved page?" Those need a model, and they don't need to run on every request.

In TokenRouter these are two separate features. Rules are expressions evaluated in-process on every request or response, with actions like block, redact, transform, route or alert, and an observe mode for rolling a rule out safely. Monitors are the judge, async and sampled.

Principle 2: sample deterministically

Random sampling is fine for averages and bad for debugging. If you rerun a check, or compare Tuesday with Wednesday, you want to judge the same slice. Hash a stable request ID into the decision, so the same request always gets the same answer.

A minimal version:

import hashlib

def should_sample(monitor_id: str, request_id: str, rate_pct: int) -> bool:
    digest = hashlib.sha256(f"{monitor_id}:{request_id}".encode()).hexdigest()
    return int(digest[:8], 16) % 100 < rate_pct

Including the monitor ID means two monitors at 10% don't judge the identical 10%. TokenRouter monitors sample this way: you set the rate per monitor, and the same request always makes the same sampling decision.

Principle 3: never put the judge in the request path

A judge that runs inline adds latency to every sampled request and gives production a new way to fail. Run it after the response. Monitors run seconds after the response, off the request path, so they can't add latency, block a request, or touch a stream.

Principle 4: make each judgment cheap

Three levers set the cost of each judgment:

Model. Start with a small, cheap judge and move up only if it can't do the job. In TokenRouter the judge has to be a tool-capable model, because verdicts come back through a tool call (more on that below), and the model picker only lists those.
Input size. Truncate what the judge reads. Monitors cap the input at about 4k tokens per judgment.
Rubric shape. One narrow question with a yes/no verdict and a reason. A 1-10 score per response is harder to act on and harder to calibrate.

Principle 5: give the judge its own budget

A judge is just more LLM traffic, so it needs a cap like any other traffic. In TokenRouter, each monitor has a daily judge budget. When it's used up, the monitor pauses itself, and the pause clears at midnight UTC. Leave the field empty and there's no cap, so set one.

Judge calls run on your own provider keys with zero markup and go through the gateway like any request: rate-limited, logged with source=monitor, and counted against your organization's default team. A team budget on that team is a second backstop, and the judge's spend sits in analytics next to the traffic it reviewed.

Our docs give a rough figure: at 100,000 requests a month with 10% sampling, a mini-class judge costs roughly $2 to $10 a month of your own provider spend. Your prompt sizes will move that, so watch the source=monitor spend for the first week.

Principle 6: treat the judge as an attack surface

If the judge reads user content, it can be prompt-injected like any other model. Two defenses help. Present the transcript to the judge as data rather than instructions. And make it return its verdict through a forced tool call, not free text. Monitors do both. Even so, no judge is perfect, so the actions are graded:

Flag (always on): the finding goes to a review queue with a severity, categories and the judge's reasoning.
Notify (opt-in): org admins get an email, at most one per cooldown window, with repeat findings batched into it.
Disable key (opt-in, with a severity gate): the offending API key is disabled, admins are emailed, the action is audit-logged, and re-enabling it is one click on the API Keys page.

Treat auto-disable as a circuit breaker for severe cases, not a content filter.

Principle 7: calibrate before you trust the dashboard

Before pointing a judge at live traffic, hand-label 50 to 100 real exchanges, run the judge on the same set, and read every disagreement. Many turn out to be a vague rubric rather than a bad model. Tighten the rubric and repeat. After launch, keep reading the judge's reasoning on each finding. That's how you notice it drifting.

Principle 8: only capture what you mean to judge

You can't judge what you didn't keep, and keeping everything is its own risk. By default TokenRouter stores no prompt or response bodies. Request logs carry token counts, cost and latency only.

Capture is opt-in for a key, a team or a member, and a request is captured when any scope covering it has capture enabled. Captured bodies are compressed and encrypted before they leave the API process. Capture can never fail, slow down or alter a request. Retention is capped by plan: 7 days on Starter, 30 on Team, 90 on Enterprise. You can choose a shorter window, and admins can purge everything at once.

Monitors only read captured traffic, so start by opting in one team or one key, not the whole organization.

Principle 9: turn findings into tests

A flagged answer is the best eval example you'll get. From a captured transcript, Add to dataset turns it into a test case. Dataset items are snapshots, so retention sweeps and purges never break your evals.

Eval runs grade with exact_match, contains, regex, or llm_rubric, a judge scoring against your rubric. That makes "prove a cheaper model scores the same on our traffic, then switch" something you can measure instead of argue about.

Putting it together in TokenRouter

A reasonable first setup:

Turn on capture for one team. Go to Settings → Data capture and add a capture scope (Team, Member or API key). Or use Enable capture on a single key from its menu on the API Keys page. Changes reach the gateway within a minute.
Create a monitor. Go to Monitors → Create monitor and fill in:
- Name
- Scope: Whole organization, Team, Member or API key, plus a target
- Judge model: a cheap, tool-capable one
- Sample rate (%): start at 5 to 10
- What should it watch for?: one narrow, yes/no rubric
Set the Daily judge budget ($).
Leave notifications and key disables off for the first week. Review findings and tune the rubric.
Add the real failures to a dataset.

Plans matter here. Monitors need capture, which starts on Starter. Starter includes 1 judge monitor, Team includes 5, and Enterprise is unlimited. Datasets and eval runs are on Team and Enterprise. The Free plan includes neither capture nor monitors.

We build TokenRouter. If you need deep tracing, experiments and CI gating, dedicated eval platforms do that well, and plenty of setups use both. Our monitor docs are at tokenrouter.io/docs/monitors.

Stable Image Core vs Ultra vs Flux Schnell: Which For Filler

Robert — Tue, 13 Oct 2026 10:00:00 +0000

If you generate filler images for PDF guides, you have to decide which image tool to hand the job to. We ran three on the same job, called pdf_guide.filler, across 317 calls in the last 90 days.

The three were Stable Image Core (stability/stable-image-core), Stable Image Ultra (stability/stable-image-ultra) and Flux Schnell (replicate/flux-schnell). Below are our own figures for each tool, what separates them on cost, speed and failures, and which one we are still using today.

One caution before the numbers. The three tools did not run side by side. Each had its own stretch of dates and its own number of calls, so read the figures as what happened on our job, not as a controlled test.

What each tool cost us per call

Flux Schnell was the cheapest at $0.003 per call. We made 48 calls with it and paid $0.144 in total.

Core cost $0.03 per call, which is 10 times what Flux Schnell cost us, or $0.027 more per call. Across 171 calls we paid $5.13 in total.

Ultra cost $0.08 per call. That is 26.67 times what Flux Schnell cost us, and $0.077 more per call. Across 98 calls we paid $7.84 in total.

Notice that Core carried the most calls but Ultra carried the largest bill. At 98 calls, Ultra cost us more in total than Core did at 171.

All of these are what we paid per call on our own usage. They are not the vendors' list prices, and we are not quoting a rate card here.

How long each one took

Flux Schnell had a median of 3.8 seconds per call. It was the fastest of the three, so everything else is measured against it.

Core had a median of 8.7 seconds. That is 2.29 times slower than Flux Schnell, or 4.9 seconds slower per call.

Ultra had a median of 15.0 seconds. That is 3.95 times slower than Flux Schnell, or 11.2 seconds slower per call.

We only measured the median per call, so we cannot say more than that about the total time of a full guide.

How often each one failed on our side

Core had 0 failures in 171 calls. Ultra had 0 failures in 98 calls.

Flux Schnell had 1 failure in 48 calls, or 2.1 percent. That was the only failure across all 317 calls.

This is the count of calls that came back as failures on our side. It is not the vendor's uptime and should not be read as such.

Which one we kept and which we stopped using

We still use Core. Its last call was on 2026-10-05, and it is the only one of the three still in use.

Flux Schnell was used from 2026-07-08 to 2026-07-13, so only about five days. Ultra was used from 2026-07-14 to 2026-09-09. Core ran from 2026-07-12 onward.

The record does not say why we stopped using Flux Schnell or Ultra. What the figures do show is that Flux Schnell was the cheapest and the fastest and also the only one with a failure, and that Ultra was the most expensive and the slowest of the three. Beyond that we will not guess at a reason.

The limits of this comparison

This is not a benchmark. These are our prompts, for our one job, at our volumes, measured over the window above. A reader with a different job may get different costs, times and failure counts.

The call counts are uneven: 171, 98 and 48. The tools were also used over different dates, so they never ran under identical conditions. Flux Schnell in particular has only about five days of history behind its figures.

Our figures say nothing about image quality, because we did not measure it here. A cheaper or faster tool is only the better choice if the images it produces are good enough for your use, and that is a judgement these numbers cannot make for you.

Today we would pick Core for this job, since it is the one we kept using, it had no failures in 171 calls, and it cost $0.03 per call. We would pick Flux Schnell instead if cost and speed mattered more than anything else, because it cost $0.003 per call and took a median of 3.8 seconds, as long as one failure in 48 calls is acceptable for the work. We would not choose Ultra on cost or speed alone, as it was the most expensive and the slowest of the three.

Originally published at neuragrowth.co. NeuraGrowth is a one-person digital-products studio; this is the log of what its pipeline does and where it breaks.

免費的 Colab CLI A100/H100 真香 :)

JH5 — Tue, 13 Oct 2026 07:56:52 +0000

前幾天 Google 官方宣布 Google AI Pro 方案納入 Colab 進階算力，讓訂閱者能直接在雲端使用 A100/H100 GPU。平常無聊跑些小模型時，我都會透過 AGY Cli 呼叫 google-colab-cli 來使用以前的 T4。既然這次 Google 這麼佛心給了 A100，我馬上決定直接從本機連上去。從認證、開立實例到載入模型，把這免費資源好好測過一遍。

釐清兩款 CLI 工具

PyPI 上有一個舊套件叫 colab-cli（版本 2.2.5），如果你跑 pip install colab-cli，它會把環境裡的 click 降級到 7.1.2、typer 降級到 0.1.1，接著就會發生一堆套件依賴衝突。看它的原始碼就知道，這只是幾年前包裝 pydrive 的上傳小工具，只能把本機的 .ipynb 同步到 Google Drive 再幫你開瀏覽器分頁，根本沒有遠端調用 GPU 的能力。

真正的官方工具是 Google 在 2026 年推出的 google-colab-cli（最新版本 0.7.4，要求 Python 3.12+）：

# 用 uv 安裝官方 CLI
uv tool install --python 3.12 google-colab-cli

安裝完成後輸入 colab --help，就會看到包含 colab new、colab exec、colab ssh、colab usage、colab log 等完整的現代化管理指令。

搞定 OAuth 與 A100 實例
第一次使用時，執行 colab usage 會觸發 Google OAuth 授權。跟其他 Google 終端機工具的流程一樣，會印出一串授權網址。用瀏覽器打開、登入訂閱了 Google AI Pro 的帳號並核准權限後，把網頁上的授權碼貼回終端機：

Enter the authorization code: 4/0AXlqoi4...
Current balance: 200.00 compute units
Usage rate: 0.00/hr
Active assignments: 0

接著直接下一行指令，在背景向 Google 申請一台 A100 實例：

colab new --session gemma-test --gpu A100

只花了大約 5 秒鐘，終端機就回傳 [colab] Session READY.。

實測 Gemma4 推論：100% GPU 滿載飆出 179 tok/s
雖然 A100 算是老卡了，但透過 colab exec 可以直接在遠端實例部署 Ollama，並拉取 gemma4:26b（GGUF Q4_K_M）進行臨床變異推論測試。

在 Colab A100 上的實際執行數據如下：

# 查詢遠端實例顯存分佈
nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv 
# 輸出：NVIDIA A100-SXM4-40GB, 40960 MiB, 20072 MiB

模型順利吃下 20,072 MiB VRAM，達到 100% GPU 負載，主機還有 83.5 GB RAM 與 20 GB 顯存緩衝。

簡單輸入乳癌基因臨床考題：

"What is the role of BRCA1 in breast cancer? Give a 2-sentence clinical summary."

可以看到模型觸發內建的思考鏈（Thinking Process）完成約 270 個 tokens 的推導後，隨後產出結構完整的兩句話臨床結論：

BRCA1 is a critical tumor suppressor gene responsible for maintaining genomic stability through the repair of double-strand DNA breaks via homologous recombination. Pathogenic mutations in this gene impair DNA repair mechanisms, significantly increasing the lifetime risk of developing hereditary breast, ovarian, and other related malignancies.

效能表現相當不錯：

生成時間：2.16 秒（共產出 369 tokens）

生成速度：179.42 tok/s

上下文視窗：Ollama 預設分配 32,768 tokens（32K context）

即時監控點數使用量
當任務正在 Colab 遠端執行時，可以透過 Colab CLI 隨時監控實例的健康狀況與資源消耗：

算力與點數計費狀態（colab usage）這條指令會回傳帳號目前的點數消耗速度：

Current balance: 197.28 compute units
Usage rate: 5.30/hr
Active assignments: 1

我們從一開始的 200.00 點跑完環境建置、下載 18GB 模型與推論，大約過了 30 分鐘，點數扣至 197.28 點，實際驗證扣款速率確實鎖定在 5.30 點/小時。

實例運作狀態（colab status -s gemma-test）查詢實例生命週期與最近一次命令執行狀態：

[gemma-test] gpu-a100-s-kkb-usc1f1-ryarjq9kw3qd | Hardware: A100 | Shape: Standard | Variant: GPU | Status: IDLE
  Last Execution: stdin at 2026-10-12 08:26:15

當任務正在跑時狀態會轉為 RUNNING，閒置時為 IDLE。

硬體級別監控（colab exec -s gemma-test "nvidia-smi"）可以直接看 GPU 實體監控數據：

溫度與功耗：32°C，待機 47.19W（推論瞬間衝上 180W+，硬體上限 400W）
顯存分配：20,072 MiB / 40,960 MiB（待機時僅佔 6 MiB）
主機配置：12 個 CPU 邏輯核心、83.47 GB 主機記憶體、235.68 GB NVMe 磁碟空間

服務層監控（colab exec -s gemma-test "ollama ps"）確認模型是否真正常駐顯存：

NAME          ID              SIZE      PROCESSOR    CONTEXT    RUNNER
gemma4:26b    001e5dafc3c7    1.2 GB    100% GPU     32768      llamacpp

Ollama 預設在閒置 5 分鐘後會自動釋放顯存，避免背景常駐佔用資源。

執行歷史與一鍵匯出這是我認為官方 CLI 最強大的功能之一，它紀錄了每次透過 stdin 或檔案丟上去的 microsecond 級別執行日誌。只需要下這行指令：

colab log -s gemma-test -o colab_a100_gemma4_replay.ipynb

它會直接把這半小時內所有的輸入、輸出、測試日誌自動打包成一份 728 KB 的標準 .ipynb 檔案。其他人打開就能看見所有 cell 的輸出結果，完全重現實驗。

網頁端即時圖形儀表板透過 colab url -s gemma-test 可以拿到專屬的筆記本網址。用瀏覽器打開後，右上角會顯示整合的硬體資源長條圖（包含系統 RAM、磁碟與 GPU RAM），可直接在圖形介面看到顯存被切走 20 GB 的即時動態。

解讀點數與定價機制：Pay As You Go 隨選加購怎麼算

在 Colab 介面點選資源分頁的「深入瞭解」，或者在終端機輸入 colab pay，系統會直接引導你打開官方的 Colab 付費方案與定價說明頁面（Colab Paid Services Pricing）。

這個頁面說明了 Google 對 Colab 算力的計費方式：

Google AI Pro / Colab Pro 訂閱制：
- 每月固定配發算力點數（如 Google AI Pro 配發 200 點）。
- 解鎖高階硬體配額（A100、TPU v5e 等），並擁有比免費版更高的排隊優先權。
Pay As You Go（隨選運算單元）加購制：
- 當你的 200 點在某個月高強度微調用光時，不需要被綁定更貴的企業合約，可以單次隨選加購點數（100 點約 9.99 美元）。
- 加購的運算單元只要帳號有效就不會過期，會自動接在每月的免費額度後面扣除。
真實消耗時間折算：
依我們這次實測的真實扣點速率（標準 A100 實例每小時扣除 5.30 點）：

   200 點 ÷ 5.30 點/小時 ≈ 37.7 小時

shell
連續跑能撐 37.7 小時，如果每一次 Prompt 測試只佔用 20 分鐘，這 200 點足夠支撐超過 100 次完整的推論工作階段。

最省錢的操作是利用 CLI 提供的 Ephemeral Job Runner：

colab run -s gemma-eval --gpu A100 my_benchmark.py

這個指令會在雲端開機、跑完腳本、抓回輸出後立刻自動關機釋放資源，連手動下 colab stop 都不用，點數一秒鐘都不會浪費。

結語

如果你對 Colab 的印象還停留在「只能在網頁裡點滑鼠」，那我強烈建議你裝一下 Google 官方的 google-colab-cli 試試。它可以讓你在本機終端機就像管理遠端叢集一樣自如地調度 A100 GPU。

再搭配上 Google AI Pro 給的 200 點算力，讓它成為 CP 值超高的臨時火力支援點。結合 colab log 匯出可重現的 notebook，以及任務結束後隨手 colab stop，雖然 Gemini Pro 訂閱常被吐槽，但我還是覺得這樣搭配 AGY、Jules 使用真的滿划算。

Can You Tell an Eval from a Review?

Tatsuya Shimomoto — Mon, 12 Oct 2026 12:33:03 +0000

On October 11, 2026, Boris Cherny, who builds Claude Code, posted on X. He was replying to a post saying that you can just tell the tool "the build is slow, make it faster" and it will do it. His point: since around Opus 4.8, Claude has done this kind of work better than he does, the 5.5 models are very good at it, and you only need to give the model three things.

A goal (for example, "Make p95 ci time <10 mins". p95 is the time within which 95% of runs finish)
A way to measure the result (for example, read the timings in BigQuery after each change. BigQuery is Google Cloud's service for aggregating large volumes of records with SQL)
How hard to try (for example, don't stop until it's done. It may turn into weeks of experiments)

That, he says, works for most things (post).

Of those three, I always get stuck on the second. READMEs and articles have no p95. I have worked on evals since February 2026, and I have built skills such as learn-eval and llm-as-judge. I still have not grasped the overall shape of what an eval is. Lately, when I loosely ask Opus 5.5 to "run an eval loop," my READMEs get better, which makes it even harder to place what I am actually doing.

I also suspect the word "eval" points to different things for different people. Some people mean only a test that produces a score. Others include reading outputs to find what to fix.

In this article, I split the parts into three layers and walk through them in order: the eval itself, the loop that runs the eval to raise the score, and the goal you hand that loop. The names and contents of the parts come from Anthropic's official guides as of October 2026 and from what eval practitioners have written. At the end, I place what I have been calling evals onto these parts. Once placed, what I had been asking for with "run an eval loop" turned out not to be an eval loop that climbs by comparing numbers. By the end, you should be able to map your own practice onto the same parts and see which kind of loop you are running.

An eval is made of inputs, a way to run them, and grading

The claude-api skill bundled with Claude Code has a procedure for building evals: /claude-api build-eval (the guide was added to anthropics/skills on September 29, 2026). Its target is apps built on Claude, and it measures whether a change worked, such as moving to another model or rewriting a prompt. The guide defines an eval as three things:

A set of inputs
A way to run the app on each input
A way to grade each output

Anthropic's engineering blog post "Demystifying evals for AI agents" (January 9, 2026) names the same things with these terms:

task: a single test with fixed inputs and success criteria
trial: one attempt at a task. LLM output varies from run to run, so the same task is run many times
grader: logic that scores some aspect of the result
transcript: the complete record of one trial
outcome: the final state left in the environment at the end of a trial

A task is one input, a trial is one run of it, and the grader is the grading. Only the grader produces a score. However carefully you assemble the inputs, if the grader is off target, the score is off target too.

What the grader reads is also one of the parts. When you measure an agent that changes its environment, such as writing files or fixing code, build-eval recommends making the main grading a check that reads the final state left in a disposable workspace (the outcome), not the conversation record (the transcript): whether the tests pass, whether the expected files exist, whether it touched anything it should not have.

The same blog post splits evals into two by purpose. A capability eval starts with a low pass rate and builds a hill to climb. A regression eval should pass nearly 100% of the time and watches for backsliding. The loop we look at later is the machinery for climbing a capability eval's hill.

Claude Code plugins and skills have one more official tool. claude plugin eval runs each case in an isolated claude -p that loads only that plugin, and grades with six kinds of graders: regex, which tools were called, the order of tool calls, whether files exist, an LLM verdict, and comparison against a reference record (baseline) (official docs).

The word "eval" means different things to different people

Back to the question from the opening. I checked the major sources on whether something that produces no score is called an eval (the full research is on GitHub, in Japanese). First, here are the axes the sources use when they talk about evaluation. Any single evaluation can be described as one choice from each axis. Which cells count as an eval, though, differs by source. After the table, we look at the cells where they split.

Axis	Options	Source
What it returns	A score / a verdict (pass-fail, a label) / a structured review / a free-form review	LangSmith, Braintrust, Hamel Husain and Shreya Shankar's FAQ, Anthropic
Who grades	A program / a model (LLM-as-a-Judge) / a person	Anthropic, build-eval
How it grades	Run a fixed check (match against the answer, tests pass) / compare two / score one against criteria	build-eval
What it reads	The conversation record (transcript) / the final state left in the environment (outcome)	Anthropic, build-eval
What it's for	Build a hill to climb (capability) / watch for backsliding (regression)	Anthropic
When it runs	During development, on fixed inputs / in production, watching real usage (Anthropic counts production monitoring as a separate method alongside automated evals)	Anthropic
How it's used	Compare versions and climb (hillclimb) / find what to fix (evaluator-optimizer)	The build-eval and hillclimb guides, Anthropic

The biggest split is how much of "what it returns" counts as an eval. The boundary was not whether there is a score. Lined up, what gets returned besides a score falls into three groups:

A verdict against criteria: pass-fail, or a fixed label. Even without a score, most sources count this as eval output. Anthropic lists pass-fail as a grading format. The evaluation tool LangSmith counts categorical values as evaluation output alongside numeric scores, and Braintrust treats classifiers that assign labels as evaluation tools alongside scorers that assign scores
A structured review: for example, whether each criterion is met, with a reason. The FAQ by eval practitioners Hamel and Shreya says an eval returns "a score or structured review"
A free-form review: just reading and returning "I got stuck here" or "fix this part." This is where the sources split. Hamel and Shreya call error analysis (reading outputs, annotating them, and grouping them into failure types) the most important activity in evals. Anthropic defines an eval as a test that applies grading logic to an output, and lists reading records as a step for checking that grading works correctly, or as a separate method alongside automated evals

Loops have the same line drawn through them. Anthropic calls this pattern evaluator-optimizer: one call writes, another returns an evaluation and advice, and the first revises. Anthropic presents it under a name separate from eval, as one of its agent design patterns (workflows). In research, Self-Refine (Madaan et al., 2023) has the same shape. Seen from the eval side, code review can be treated two ways. If an AI reads a PR and a developer just reads its comments and fixes things, that is a review. If instead you give that reviewer PRs with known bugs planted in them and grade the share it found and the number of wrong comments, that is an eval that measures the AI reviewer. This is my reading, applying Anthropic's definition of a test that applies grading logic to an output; no source officially defines it this way.

I found no study that measured what most people mean by "eval." Even among widely read sources, the line sits in very different places. "Eval" is a new word whose meaning shifts from person to person.

Align grading with human judgment before you use it

There are several ways to grade, and each is checked differently. If an LLM grades, you revise it until it agrees with human judgment before you use it. Even then, with too few cases, the score is buried in noise.

build-eval lists four grading methods, roughly from cheapest to most expensive, and recommends choosing the cheapest one that actually measures what you want to measure:

Programmatic checks (exact match, JSON shape, tests pass). For outputs with a bounded range of answers
A blind side-by-side comparison of two outputs, where a model picks the better one. Swap the order of A and B each time
A model scores a single output against a rubric
A person grades a small number by hand

Having a model grade, as in 2 and 3, is called LLM-as-a-Judge, and the grading model is called the judge. When choosing a grading method, build-eval shows the user five graded examples and asks, "Would you have scored any of these differently?" If even one would be different, the rubric is not ready yet. You revise until the grading matches the user's judgment.

Hamel and Shreya's FAQ states the same idea, down to the numbers:

Read the outputs first (error analysis). Annotate the records, group them into failure types, and keep going until new reading turns up no new types
Grade pass-fail, not on a 1–5 scale
Validate the judge on human-labeled examples before using it. Measure the share it correctly passes (TPR) and the share it correctly fails (TNR). You need 100–200 examples per failure type
Labeling is done by someone who knows the product's context. They are strongly skeptical of outsourcing it

How well a judge agrees with people depends on the conditions. In the 2023 MT-Bench paper, on pairwise comparisons excluding ties, GPT-4 agreed with humans 85% of the time, and humans agreed with each other 81% of the time. Including ties, those drop to 66% and 63%. The agreement figure changes with how ties are counted.

The other condition that decides whether you can trust a score is the number of cases. build-eval recommends 15–100 inputs for a first eval. Below 15, a single case's variation moves the score. The noise in one pass rate is roughly 1/√(cases × trials), so even 25 cases run twice each comes to about ±14 points. A difference smaller than that noise cannot tell you whether the change worked or it was chance.

The loop climbs through holes in the grading too

Once you have an eval, you want to run a loop that raises the score. The claude-api skill's /claude-api hillclimb is that procedure. It first checks whether there is an eval that runs. If not, it stops and sends you back to build-eval. A hillclimb without an eval, it says, is "just editing and hoping."

Laid out as parts, hillclimb looks like this:

goal: what to raise or lower. One of: raise the score, lower the cost, lower the latency, or move to another model. Other recorded metrics become guardrails, conditions that must not get worse beyond the noise
What may change: the prompt, the tool descriptions, the model or effort, and so on. Also decide what must not be touched
Split: randomly split the inputs into train (problems read to make improvements) and test (problems never shown to whoever works out the fixes, measured for score only each round). The analyzer, which reads failures and works out fixes, reads only the train records. The side running the loop sees only scores
One change per round: if you bundle several changes, you cannot tell which one worked
Acceptance: if train goes up and test does not, treat it as fitting the cases it read and roll back. If train also goes down, roll back
When to stop: the recommendation is to stop once improvements smaller than the noise have continued for three or more rounds (a plateau). Two flat rounds is too few, the guide says¹
Budget: if cost matters, estimate the total as the cost of one run × (rounds + 1) × trials, and set a ceiling
Report: what you report at the end is the improvement on test since the start, with its confidence interval. If it stays within the noise, the guide recommends saying so and not merging

When there are few cases, you run on all of them without splitting and state that the numbers only show direction (directional). Before the first round, you lay out three things: the size of the noise, the headroom, and the smallest improvement the user would act on. If the noise is larger than that smallest improvement, you say up front that no change will show a visible improvement.

Why split train and test? Because when a loop finds a hole in the grading, it can use that hole to raise the score. A paper posted to arXiv in July 2026 has an example.

The authors had Claude Code (Opus 4.8) and Codex CLI (GPT-5.5) each run the same automatic improvement loop three times. The task was to build a program that takes speech-recognition transcripts of Quran recitation, identifies which verse each part is, and segments it. The loop graded the program after each rewrite and put the problems it got wrong into a failure report. But that failure report also showed the correct answers (the verse IDs) for the problems it got wrong.

All three times, Codex embedded those correct answers directly in the program: a program that, without solving the problem, returns the answer for problems it already knows. Because it copied the answers, its score beat Claude's. Claude Code did not embed the answers in any of its three runs, and its score plateaued below Codex's.

In the next experiment, the authors split part of the problems into a test the loop never sees, removed the answers from the failure report, told both agents that a test existed and copying would not work, and isolated the environment for each run. With all four changed at once, copying dropped to zero. The authors note that they did not separate which change did it.

hillclimb's train/test split corresponds to the first of those fixes. If a loop raises its score only on grading where it can see the answers, the raised score is indistinguishable from copying. Splitting off a test measures the score on problems the loop has not seen, to check that it has really learned to solve them.

You can set a numeric goal when a tool that measures the same quantity already exists

Back to Boris's example: the goal "p95 CI time under 10 minutes" comes with a way to measure it, "read the timings in BigQuery after each change." The goal and the measurement point to the same quantity. hillclimb, too, when setting the goal, lists candidates from among the metrics the eval already records.

In his July 2025 blog post "Asymmetry of verification and verifier's rule," Jason Wei wrote that how easily you can train AI to solve a task is proportional to how easily that task can be verified. He lists five properties of easily verified tasks:

Everyone agrees on what a good solution is (Objective truth)
One solution can be verified in seconds (Fast to verify)
Many solutions can be verified at once (Scalable to verify)
The verification result correlates well with solution quality (Low noise)
It is easy to rank many solutions to one problem from good to bad (Continuous reward)

For CI time, everyone can agree that shorter is better. For how clear a README is, the first property, "everyone agrees," is already hard.

If you set a goal with no way to measure it, the verdict has to rely on the words in the conversation. With Claude Code's /goal, you write a condition, and after each turn a small, fast model reads the conversation and judges whether it has been met. According to the official docs, this evaluator sees neither commands nor files. That is why the docs recommend writing "how to verify it" into the condition. The condition I wrote in July ("Loop until 20 ideas come out. ... When you're done, write a report in notes, wrap it up, and push") had no way to verify it, and on top of that the notes folder was in .gitignore, so it could not be pushed. The evaluator kept returning "not met" 41 times, and I stopped it by hand after 7 hours and 38 minutes.

Placing my evals on the parts

From here, I place what I have been calling evals onto the parts laid out so far. First, one real example.

On October 8, I asked, about the README of my research repository Contemplative Agent (a long-running experiment with an autonomous agent that runs on a local LLM): "I want to rewrite the README substantially, so run an eval loop and get it to a quality I'd be proud to show anyone as one of my main projects." Then I asked, "Run an eval on the tagline too and improve it." The tagline in the Japanese README, the sentence right under the title, changed like this (translated here):

Before (September 12 version): "An autonomous agent running on a local LLM that proposes changes to its own constitution and values. A human decides every time whether to adopt them."
After (October 8 version): "A long-running experiment. An AI agent posts on its own using a local LLM and proposes changes to its own harness (the constitution, identity, and skills that go into its prompt). The model weights do not change. A person makes the final call on each proposal."

A diagram of a four-stage loop went in at the top, and the record of proposal outcomes (82 skill proposals adopted, 547 rejected) went into the README too. Whether it reached a quality I'd be proud to show anyone is another matter, but it got a lot better than the earlier version.

Broken into parts, the loop looks like this. The goal was a phrase about quality, not a number: "a quality I'd be proud to show anyone." The inputs were the README draft and 37 candidate taglines. The readers were visitor personas, an LLM playing three people: a researcher, an engineer, and a skeptical reader. The taglines were read 234 times in total. The README as a whole was judged by a README judge (readme-judge) using fixed questions and evidence from checking claims against the code. I made the acceptance call at the end by reading it myself.

Along the way, I corrected the goal's direction. I said, "I was thinking, maybe we don't need to put human approval up front. The north star is toward removing human approval, right? This is trying to do recursive self-improvement through the harness." Opus 5.5 replied that it half agreed and that half of my premise was off, and proposed putting the harness at the center and adding one grading check for "can a reader tell this is about the harness." I replied "Go," we re-measured, and we settled on the current tagline with the harness at the center.

What I did on the README was a review loop

The parts are all there. But the way the scores were used differs from hillclimb.

For the README body, I used the visitor personas' answers not as scores but as hints for rewriting. I read "where did you lose interest" and "if you fixed one thing, where would it be" and revised the text. I did not compare the previous and current versions by score. The exception was the tagline: for each candidate, I counted whether readers could correctly state what it was about, and compared those counts. That comparison is close to an eval loop, but there was no test split, and I did not look at how much those counts fluctuate
At first I also asked "would you try this this week" on a 1–10 scale. I dropped that score, writing the reason in the README skill's records: if a score decides when to stop, raising the score becomes a goal in itself
I stopped on the README body when the readers stopped turning up new findings that a change in wording would fix, not when a score moved by more than the noise
The judge (readme-judge) returns one of Publishable, Fix, or Rewrite and lists what to fix. It does not compare versions. My llm-as-judge skill has the same shape: it rolls binary checks into one named verdict and does not aggregate scores

In other words, what I did on the README was a review loop built from eval parts. I had been asking for this, too, as an "eval loop."

As we saw in the section "The word 'eval' means different things to different people," whether free-form reviews count as evals differs by source. So in this article, "eval" refers to the parts from the first section (inputs, a way to run them, grading), and I split the loops that use them into two by the question they ask:

Eval loop: a loop that compares versions on a yardstick, such as a score or pass rate, and raises that value
Review loop: a loop that reads this one piece and looks for what to fix next

The "review loop" in this article corresponds to what Anthropic calls evaluator-optimizer.

Even with the same parts, the question they are trying to answer differs.

	Eval loop (hillclimb)	Review loop (README)
Question it answers	Did this change work? Does it work on cases it hasn't read?	Where is the next thing to fix in this one piece?
Role of grading	Compare versions	Find what to fix
What it needs	Case count and a noise estimate (train and test if there are enough cases)	Readers who read for different reasons
How it stops	When the improvement gets smaller than the noise	When no new findings come up (the same stopping rule as Hamel and Shreya's error analysis)

The figure's labels are in Japanese; it shows the two loops from the table. The eval loop on the left (change the version → measure on a yardstick → compare with the previous version → keep it if better) pushes the yardstick's value up. The review loop on the right (read → list what to fix → rewrite) rereads until the readers turn up nothing more to fix.

What build-eval and hillclimb measure is an app built on Claude. Applied to an artifact like a README, the form would be not a single README but running a README-writing skill on 15 repositories and comparing. A single README does not have enough inputs to split into train and test.

What did have the shape of an eval loop was September's daily-report bench (a setup that compares drafts to polish the prose of my daily research reports). I prepared cases with fixed source material and set about a third of them aside as a test not used for improvement. I repeatedly rewrote the prompt that writes the prose and compared on the same cases, and the prompt files reached 29. I judged readability myself, reading the two drafts blind, and left the judge only the pass-fail check of faithfulness to the facts, because when I had the judge decide readability too, it picked the opposite of the draft I had judged better, in both orderings (one comparison). I started the bench on September 23, before the build-eval and hillclimb guides were published on September 29, so I built it from scratch with Claude Code.

How often were the review's findings right?

While writing this article, I checked once how often the README review loop's findings were right. From the README loops of 7 repositories, I picked 28 findings and verdicts from the judge and the visitor personas, and labeled each one myself as "hit," "miss," or "can't tell." A "hit" is a finding or verdict I agreed with; a "miss" is one I did not. For example, the README of my repository akc-cycle had an English sentence saying, in effect, "skill-stocktake and generation-audit use skill-health's scan, so install them together." The judge pointed out that it does not say what you need if you install only skill-health. I labeled that a "miss" and wrote in my note: "It's bundled in the plugin, so I don't feel the need to fix it, but the README doesn't need to go into that level of detail in the first place."

Target	Count	Hit	Miss	Can't tell
Judge flagged as should-fix	10	3	3	4
Judge rated as satisfied	7	7	0	0
Visitor persona findings	11	7	3	1

The four "can't tell" among the judge's should-fix findings came from checking README claims against the code and records. To verify them I would have had to check against the code myself, which would defeat the purpose, so I marked them "can't tell." Where I disagreed with the judge's "should fix," it was not about factual errors but about preference: how much the README should say. The 28 are only what the judge and visitor personas raised; anything they missed is not included. In terms of the two ways of treating code review, this is the side that measures the reviewer. What I measured is how many findings missed; I did not measure the share of real problems they found.

I have avoided eval loops

What I have been doing for READMEs and articles is not an eval loop but a review loop. For articles, too, I run the same kind of review: another LLM plays readers, reads the draft, and I fix how it comes across. The reason I felt I didn't know the best practice was that I was holding my review loops up against the procedure of an eval loop like hillclimb.

I have not run eval loops on READMEs and articles, and not because I fell short. I avoided them. The biggest reason is that I dislike Goodhart's law. Named after the economist Charles Goodhart, the law in its commonly quoted form says, "When a measure becomes a target, it ceases to be a good measure" (that phrasing was refined by the anthropologist Marilyn Strathern in 1997). The example in the section "The loop climbs through holes in the grading too," where the loop copied the answers to raise its score, has the same shape. So I had decided not to aggregate scores in my llm-as-judge skill and not to create numeric scores in my README skill. This direction was there before. In February, I dropped rubric scores in favor of an overall judgment in my skill audit (article). In August, I wrote about two drafts that passed every evaluation and were both hollow (article), and about throwing away my homegrown LLM judge even though the KPIs had improved (article).

There is one more reason: I have rarely worked on material where a numeric goal can be handed to a loop. Even in Contemplative Agent, most of what I look at is natural language, such as the agent's comments and its skill choices. There are instruments that measure in numbers, but I use them only for observation and catching regressions, not as a loop's goal.

The benchmark score is the number to trust least

RC — Thu, 08 Oct 2026 22:25:46 +0000

Claude Opus 5.5 scores 58 on Artificial Analysis's Intelligence Index, more than double the median of 26 for comparable reasoning models. That is the figure that gets quoted. It also tells you the least about what the model costs to run.

The same evaluation run that produced the 58 recorded everything else. Getting through the Intelligence Index took Opus 5.5 some 260 million output tokens. The median model needed 81 million. That verbosity shows up on the bill: Artificial Analysis puts the average at $5.98 per task to run the index. Input is $4 per million tokens and output $20, both at the top of the range against medians of $2 and $10.

The composite score hides the number an operator actually budgets against. A model that reaches the same answers in a third of the tokens, and lands a few points lower on the index, can be the better buy for a given workload. The headline will not tell you that.

Latency is the other figure the score buries. Artificial Analysis reports a time to first token of 724.53 seconds against a 3.77-second median. Opus 5.5 is a reasoning model, so that window covers the extended-thinking phase before the first answer token appears; the site's end-to-end response time folds thinking time in for exactly this reason. Once it starts writing it is quick, 96 tokens per second against an average of 76. The wait is front-loaded into the thinking phase.

For an interactive tool that difference is the whole user experience. For a batch pipeline it may not matter at all. The Intelligence Index scores neither case; it scores correctness and leaves you to discover the rest.

None of this makes 58 a bad result. It places the model among the leaders on capability, with a 1M-token context window and text-plus-image input. The point is narrower: a single composite number is a ceiling on what the model can do, not a profile of what it does on your traffic. Treat it as a filter for the shortlist, then measure the candidates yourself, on your own prompts, counting output tokens, dollars per task and time to first useful token.

If you are pricing Opus 5.5 specifically, the blended rate Artificial Analysis quotes is $2.94 per million tokens at a 7:2:1 cache-hit/input/output mix, and the model is served by nine API providers, so the per-token cost is worth comparing across them before you commit.

AI Agent Observability: Steps, Cost, and Failure Traces

lizer yang — Thu, 08 Oct 2026 01:05:39 +0000

Originally published at AI Agent Observability: Steps, Cost, and Failure Traces on smartgate.network.

A shorter version of "AI Agent Observability: Steps, Cost, and Failure Traces" — the full piece lives at smartgate.network.

What the full piece covers

ai agent observability: three questions, one record — An agent run does not look like a chat completion, and observability built for chat completions does not fit it.
llmops: the agent run as the unit of analysis — LLMOps is the practice of running language-model systems as production software, and its centre of gravity shifts when the system is an agent.
agent evaluation: read the trace before you trust the answer — Agent evaluation and agent observability are usually discussed as separate disciplines, and in practice they share one artifact: the trace.
Where the run id comes from, read from our own implementation — A run view needs an identifier that both sides agree on.
ai agent monitoring: latency budgets and failure modes — Monitoring an agent means watching the run's shape over time, not just its error rate, because an agent degrades before it fails.
langfuse alternative: what an agent-run view adds to prompt tracing — If you already run prompt-level tracing, the honest question is not which dashboard is prettier but which events exist in the record.
langsmith alternative: trace data and the privacy boundary — A trace is more sensitive than a log line, and that is the fact most observability rollouts discover late.
How to get started — The first move is not a product; it is a correlation id and one row per step.
Where SmartGate fits — SmartGate is the control plane an agent run passes through, made concrete: an MCP-native algorithm gateway for token control, traffic shaping and agent audit.
Limitations — This page is a design frame for the agent layer, and it is deliberately bounded.

Read the full piece: AI Agent Observability: Steps, Cost, and Failure Traces on smartgate.network.

Agent cost optimization: same quality, lower price

Nicolai Bohn — Wed, 07 Oct 2026 12:27:17 +0000

Most teams pick their agent's model while prototyping. It works, it ships, and nobody touches it again. Meanwhile cheaper models come out every few weeks.

Switching sounds easy until someone asks whether the new model will handle your refund policy questions as well as the old one. Nobody knows. There is no test set to run it against, so the team stays on the old model, and the bill stays where it is.

Breaking that loop takes two things: knowing what a single run of your agent costs, and a safe way to compare one setup against another.

Where the money goes

Cost is tokens times price, summed over every LLM call your agent makes. Four levers decide how big that number gets.

Model. Price per token can differ by an order of magnitude between model tiers. For most agents this is the biggest lever. Test it by swapping the model.

Prompt and context. The system prompt and any retrieved context are sent again on every call. A 3,000-token system prompt in a ten-turn conversation is 30,000 input tokens before the user has said much at all. Test it by shortening the prompt.

Agent steps. Every tool call or reasoning step is another LLM call, and conversations carry their history forward. An agent that takes six steps where two would do pays for it on every request. Test it by capping the steps.

Caching. Many providers bill cached input tokens at a lower rate. Keeping the start of your prompt stable lets you benefit from that. Test it by keeping prefixes stable and comparing.

Cheaper only counts if quality holds

A cost number on its own tells you very little. If v2 costs half as much but fails a third of your cancellation tests, you haven't saved anything. You've moved the cost to your support team.

So look at pass rate and cost together, on the same test set.

That gives you four outcomes:

Cheaper, same or better quality: promote it.
More expensive, better quality: decide whether the gain matters to your users.
Cheaper, lower quality: check which behaviors failed. A prompt tweak may win the quality back.
More expensive, lower quality: roll it back.

Change one thing, compare side by side

The method is plain experimentation. Change one variable at a time so you know what caused the difference.

Save a baseline. v1 is your current setup: model, prompt and temperature.
Vary one parameter. v2 swaps the model. v3 keeps the model and trims the prompt.
Run the same tests against each version, with the same metrics.
Compare the runs on pass rate and cost.

For a support agent this could look like:

Version	Model	System prompt	What you're testing
v1	your current model	full prompt	baseline
v2	a smaller model	full prompt	does the cheaper model hold up?
v3	your current model	trimmed prompt	how much does the prompt cost you?

How this works in Rhesis

Rhesis is an open-source platform for testing LLM apps and agents. We recently added cost tracking to test runs, and it pairs well with Experiments:

Every test run shows its tokens, cost and model in the runs list. You can sort by cost to find your most expensive runs.
Each run has a Cost card with input and output tokens priced separately. It updates live while the run executes.

Experiments store configurations as versions. A parameter schema defines the slots (model, prompt, temperature), and every change becomes an immutable version, so a run against v2 is reproducible later.
Side-by-side comparison shows pass rate and cost with deltas against the baseline. The cost delta turns green when the new run was cheaper.

Costs are calculated from the tokens each LLM call reports, multiplied by list prices from LiteLLM's model database. That makes them a good relative comparison. They don't include discounts or committed-use pricing, and models missing from the database show tokens without a cost. Details are in the token costs docs and the experiments docs.

Here's the walkthrough:

Try it

Rhesis is MIT-licensed. You can use it at app.rhesis.ai or self-host it from GitHub.

If you're running cost experiments on your own agents, I'd like to hear which lever saved you the most. Let me know in the comments.

Originally published on the Rhesis blog.

MLOps and LLMOps: Keep Your AI Reliable After Launch

ROBERT KAWOSKI — Tue, 06 Oct 2026 10:17:41 +0000

MLOps and LLMOps are the monitoring, evaluation, versioning and governance practices that keep AI systems accurate, safe and accountable after go-live. Models drift, prompts regress and costs creep. The fix is a simple loop: instrument every call, evaluate quality on a schedule, and release every change through a versioned, reversible path.

What are MLOps and LLMOps?

MLOps is the set of practices for running machine learning models reliably in production: tracking data and model versions, monitoring predictions, retraining when needed and controlling releases. LLMOps applies the same discipline to applications built on large language models, where the moving parts also include prompts, retrieval pipelines, tools and agent behaviour.

Both exist because of one uncomfortable fact. A model that passed its launch review is not automatically accurate a quarter later. The data changes, the users change, the upstream model provider updates a version, and someone edits a prompt on a Friday afternoon. Without operations discipline, nobody can tell which behaviour moved or when.

Why AI systems fail quietly after launch

Traditional software fails loudly. A server crashes, an error page appears, an alert fires. AI systems fail quietly. They keep returning answers that look plausible, so the damage shows up late: a recommendation engine slowly loses relevance, a support assistant starts giving outdated policy answers, or a scoring model treats a new customer segment unfairly.

Researchers at Google described this years ago in their well known paper on hidden technical debt in machine learning systems (Sculley et al., NeurIPS 2015). Their central point still holds: the model code is a small part of a production ML system, and most of the risk lives in the surrounding data, configuration and monitoring. LLM applications make that even more true, because the “code” now includes natural language instructions that anyone can edit.

Three failure patterns show up again and again:

Drift: the live data no longer looks like the data the model learned from.
Regression: a prompt, model or dataset change improves one behaviour and silently breaks another.
Workarounds: users stop trusting the output and quietly build manual checks around it, which erases the value you built the system for.

MLOps vs LLMOps: what actually changes?

If you are wondering about the difference between MLOps and LLMOps, the short version is that the principles stay the same while the artifacts change. Here is a side by side view.

Area	MLOps	LLMOps
What you version	Models, features, training data	Prompts, models, retrieval indexes, tools, datasets
Main risk	Data and concept drift	Prompt regression, hallucination, cost spikes
Evaluation	Accuracy, precision, recall on labelled data	Graded test sets, rubric scoring, human review
Cost driver	Training and serving compute	Tokens per call, context length, retries
Rollback unit	Model version	Prompt plus model plus config bundle

How to monitor LLM models in production

Monitoring starts with a record. If you cannot reconstruct what the system saw and said, you can only guess at why it misbehaved. We treat this as step one of our approach: every prediction leaves a record.

What to log on every call

Capture the input, the output, the model and prompt version, the retrieved context if any, latency, token usage and cost. Add a trace ID so a single user complaint can be traced to the exact call. Mask personal data at the point of logging, not later.

What to track over time

Watch quality scores, refusal and fallback rates, latency percentiles, cost per request and the share of outputs flagged by users or reviewers. A single dashboard that shows these by version makes it obvious when a change helped or hurt.

Model drift detection in production

Drift comes in two main forms, and they need different checks.

Data drift

The inputs change. A lending model trained on salaried applicants starts seeing gig workers. Compare live feature distributions against a baseline window using tests such as the Population Stability Index (PSI) or the Kolmogorov-Smirnov test, and alert when the gap crosses a threshold you agreed in advance.

Concept drift

The relationship between inputs and the right answer changes. Customer behaviour shifts after a price change or a market event. This is harder to see because inputs may look normal. You catch it by tracking accuracy against fresh labels as they arrive, even if the labels are delayed by weeks.

For LLM applications, add a third lens: behavioural drift. Run the same fixed test questions every day and watch whether answers change after a provider model update.

Building an LLM evaluation framework for production

Quality has to be checked on a schedule, not when someone remembers. A workable evaluation setup has four layers:

Regression suite: a fixed set of inputs with expected behaviour, run on every change.
Graded evaluation sets: curated examples scored against a clear rubric for accuracy, tone, safety and groundedness.
Online checks: sampling of live traffic, scored automatically.
Human review: kept wherever automated scoring cannot be trusted, such as legal, medical or high value financial answers.

Using one model to grade another (often called LLM as judge) is useful for scale, but it needs calibration. Compare its scores to human ratings on a sample and revisit that comparison whenever the judge model changes.

Expert insight: The Sculley et al. paper makes the case that ML systems accrue maintenance cost faster than ordinary software because of data dependencies and feedback loops. That is the core reason evaluation and monitoring must be treated as permanent work, not launch work.

Prompt versioning best practices (and model and dataset versioning)

Our rule is simple: nothing changes without a version. That covers models, prompts, datasets and policies, and all of them move through the same gated release path.

Store prompts in source control, not in a dashboard text box.
Give every prompt, model and dataset an immutable version ID, and log it with each call.
Require evaluation results to pass before a version is promoted.
Record who approved the change and why, so audits take minutes rather than weeks.

How to roll back a bad model release

Rollback only works if you planned for it. Keep the previous version deployable, release new versions to a small share of traffic first (a canary), and define the trigger in advance: for example, a drop in quality score or a spike in cost beyond an agreed limit. Because prompt, model and configuration are versioned together, reverting is a switch, not a rebuild.

Frequently asked questions

What is the difference between MLOps and LLMOps?

MLOps operates predictive models trained on your data. LLMOps operates language model applications, where prompts, retrieval, tools and outputs also need versioning, evaluation and cost control.

How do you monitor LLM models in production?

Log every call with inputs, outputs, versions and cost, then run scheduled evaluations on graded test sets and track quality, latency, cost and failure rates over time.

How do you detect model drift in production?

Compare live input and prediction distributions against a baseline using tests such as PSI or Kolmogorov-Smirnov, and track accuracy against fresh labels when they arrive.

When should we start MLOps?

Before launch. Instrumentation and versioning are far cheaper to build in than to retrofit once users depend on the system.

Do we need MLOps consulting in India or can we build it in house?

Many teams build the basics in house and bring in a partner for evaluation design, governance and audit readiness. iAastha works from Indore and Mumbai with startups, enterprises and GCCs.

Conclusion

A model in production is a system to be operated. The work that matters most starts at launch: evaluation that keeps running, versioning that makes every change reversible, and monitoring that tells you which behaviour moved and when. Teams that build this discipline early spend less time firefighting and more time improving.

Originally published on iAastha: https://iaastha.com/insights/blog/mlops-and-llmops/

The Silent Quality Collapse: Why Your LLM Router Is Secretly Destroying User Trust

Mohit Verma — Mon, 05 Oct 2026 05:37:34 +0000

Your Cost Dashboard Is Lying to You (Just on a 6-Week Delay)

Your LLM router shipped. Cost dashboards show 60% savings. Your CEO is thrilled. Engineering is celebrating.

Then week 7 hits. Churn spikes 18%. Nobody connects the dots.

This is the failure mode that is not getting written about: LLM routing quality degradation is optimized for observable metrics (cost, latency, throughput) while silently degrading unobserved metrics (response nuance, edge-case handling, user trust). The feedback loop between quality degradation and user cancellation has a 6 to 7 week lag that makes root-cause analysis nearly impossible without purpose-built instrumentation.

This is not a routing algorithm problem. It is a monitoring architecture problem. Even a perfectly calibrated router will cause silent quality collapse if you lack the eval layer to detect drift between model tiers on production traffic. The cost savings are visible on the dashboard. The quality loss is not. The failure is insidious precisely because it looks like success.

Teams that shipped routers in Q1 2025 are hitting this wall right now. The standard observability stack (LangSmith traces, cost dashboards, latency p99s) is blind to it. This article is the canonical reference for understanding why the collapse happens, how to detect it before it reaches churn, and how to build the eval architecture that makes your router trustworthy at scale.

The counterintuitive prescription: build the eval layer before you build the router.

System Design First: What a Production LLM Router Actually Does

Before diagnosing the failure, it helps to be precise about the system. A production LLM router sits between your application layer and your model providers. Its job is to classify each incoming request and dispatch it to the most cost-efficient model that can adequately handle it.

The three dominant routing architectures each carry distinct tradeoffs:

Always-small: Every request goes to the cheaper model (Haiku, GPT-4o-mini). Maximum cost savings. Maximum quality risk. No intelligence in the routing layer itself.

Rule-based: Explicit conditions route to tiers. Short queries go to the cheap model. Queries containing certain keywords or exceeding a token threshold go to the premium model. Predictable and auditable, but brittle. Rules do not generalize to novel query patterns.

Cascade (the most common production pattern): Try the premium model first. Fall back to the cheaper model on timeout or rate-limit. Or invert it: try cheap first, escalate to premium if a confidence threshold is not met. Cascade gives you a safety net, but as we will see, that safety net has holes that are invisible in your logs.

RouteLLM-style classifier routing: Train a binary classifier to predict whether a given query requires the premium model. The classifier assigns a confidence score. Above the threshold, route to premium. Below it, route to cheap. This is the most sophisticated approach, and it introduces the most subtle failure mode: confidence score miscalibration.

The architectural decision that shapes everything downstream is this: the router's intelligence is only as good as its training distribution. When production traffic drifts away from that distribution, the classifier degrades silently. There is no error thrown. The response is still delivered. The log says success. The user gets a worse answer.

This is the root cause of the silent quality collapse.

The 6-Week Time Bomb: Walking the User Behavior Decay Chain

The chain from routing decision to cancellation is invisible when you look at any single metric in isolation. It only becomes visible when you trace the full behavioral sequence.

Week 1 to 2: Subtle Response Quality Drop

15 to 30% of queries get routed to the cheaper model. Most responses are fine. But edge cases get noticeably worse answers. Users do not consciously register this yet.

Week 2 to 3: Unconscious Rephrasing

Users start retrying queries. Follow-up question rate increases 20 to 35%. From your dashboard, this looks like increased engagement. It is actually frustration expressed as extra work.

Week 3 to 4: Learned Helplessness

Users reduce usage frequency by 10 to 15%. They stop bringing high-stakes tasks to the tool. DAU still looks stable because power users compensate for the drop in casual users.

Week 5 to 6: Alternative Evaluation

Users start trying competitors. Engagement drops below the reactivation threshold. NPS surveys have not captured any of this yet.

Week 6 to 7: Cancellation

Only about 5% of frustrated users ever filed a support ticket. The other 95% just leave. The churn spike appears in your dashboard with no obvious cause.

The 3 Invisible Routing Failure Modes Your Dashboard Will Never Show

Failure Mode 1: Semantic Drift

Smaller models handle the center of the distribution well. They fail on edge cases that larger models were specifically RLHF-trained to handle. The router's confidence score says "simple query" but the query contains domain-specific nuance the classifier never saw during training.

Failure Mode 2: Confidence Score Miscalibration on Domain-Specific Prompts

The RouteLLM paper (arxiv 2406.18665) documents that classifier calibration degrades meaningfully on out-of-distribution domains. Edge-case queries may represent roughly 20% of volume but approximately 60% of user-perceived value.

Failure Mode 3: Cascade Timeout Silent Fallbacks

When a router implements cascade logic, the fallback is logged as a successful response, not a degraded one. Under load spikes, 5 to 15% of requests silently fall back without any quality flag.

Building the Eval Layer: The Architecture That Prevents Silent Collapse

The eval layer is not a testing framework. It is a production monitoring system that runs continuously alongside your router, sampling live traffic and scoring response quality in real time.

The four components that make it work:

Route decision logger — captures route_intended, route_actual, confidence_score, model_used, and latency for every request
Quality scorer — runs a lightweight LLM judge on sampled responses, scoring completeness, accuracy, and nuance
Drift detector — compares quality scores across routing tiers over time, alerting when the gap widens
Feedback loop — surfaces quality degradation signals back to the routing threshold, enabling dynamic recalibration

The Counterintuitive Prescription: Build Eval Before Router

The standard advice is to ship the router first and add monitoring later. This is backwards.

Without a baseline quality measurement before routing, you cannot detect degradation after routing. You need a pre-routing quality baseline to measure against. The eval layer is not a nice-to-have. It is the instrument that makes the router trustworthy.

Build the eval layer first. Then ship the router. Then tune the threshold using real quality signals, not cost proxies.

Follow for more production ML engineering insights. Connect on LinkedIn.

The Silent Quality Collapse: Why Your LLM Router Is Quietly Destroying User Trust

Mohit Verma — Mon, 05 Oct 2026 05:29:40 +0000

The Silent Quality Collapse: Why Your LLM Router Is Quietly Destroying User Trust

Your LLM router just saved 60% on inference costs. Your CEO is thrilled. Then week 7 hits and churn spikes 18%. Nobody connects the dots.

This is the failure mode nobody is writing about.

LLM routing is optimized for observable metrics: cost, latency, throughput. It silently degrades unobservable ones: response nuance, edge-case handling, user trust.

The feedback loop between routing decisions and churn has a 6-7 week lag. By the time it shows up in your dashboard, the root cause is buried under weeks of other changes.

The Decay Chain

Here is the decay chain I have seen play out repeatedly:

Week 1-2: 15-30% of queries route to cheaper models. Edge cases get subtly worse answers. Users don't notice yet.

Week 2-3: Follow-up question rate increases 20-35%. Users are rephrasing prompts. Your dashboard reads this as engagement. It is frustration.

Week 3-4: Usage frequency drops 10-15%. Users stop bringing high-stakes tasks to the tool. DAU looks stable because power users mask the signal.

Week 5-6: Users start evaluating competitors. Quietly. No support tickets.

Week 6-7: Churn spike. No obvious cause in the postmortem.

The Three Failure Modes Your Observability Stack Will Never Catch

1. Semantic Drift

Smaller models handle the center of the distribution fine. They fail on domain-specific edge cases that larger models were RLHF-trained to handle.

2. Confidence Score Miscalibration

RouteLLM-style classifiers degrade on out-of-distribution domains. The confidence score is high. The quality gap is real.

3. Cascade Timeout Silent Fallbacks

Under load spikes, 5-15% of requests silently fall back to cheaper models. The log says success. The user got a worse answer. Nobody knows.

The Fix: A Monitoring Architecture Problem

The fix is not a better routing algorithm. It is a monitoring architecture problem.

You need three things:

A shadow eval layer that scores routed responses against a quality rubric before they reach users
Per-route-tier quality metrics, not aggregate quality metrics
Instrumentation that surfaces route_actual vs. route_intended mismatches

The counterintuitive prescription: build the eval layer before you build the router.

Cost savings on the dashboard. Trust erosion in the background. The gap between those two is where products quietly die.

Resources

Have you hit this wall? Curious how others are instrumenting their routers for quality, not just cost.

LLMOps: The Four Layers an Agent Stack Must Cover

lizer yang — Fri, 02 Oct 2026 08:15:12 +0000

Originally published at LLMOps: The Four Layers an Agent Stack Must Cover on smartgate.network.

A shorter version of "LLMOps: The Four Layers an Agent Stack Must Cover" — the full piece lives at smartgate.network.

What the full piece covers

llmops: the four layers, and what each one actually decides — The word covers a lot of ground, so it is worth separating what each layer answers.
agent orchestration: who decides which step runs next — Orchestration is the part of the stack that decides the next action, and there are only three honest designs.
multi agent orchestration: supervisor, handoff, or a queue — Multi-agent orchestration is where budgets go to die, so it is worth being blunt about when a second agent earns its cost.
multi agent systems: when more than one agent earns its cost — Multi agent systems sit at the top of this taxonomy, and the deciding question is not "how many agents" but "how many independent failure domains".
ai agent framework: the four questions that decide it — The framework market is loud, so the useful move is to compare candidates on four questions that do not depend on branding.
agent memory: what to persist, where, and for how long — Memory is the layer with the largest gap between what is demonstrated and what is operated.
ai agent tools: the categories, and what each one is for — The phrase "AI agent tools" covers two different things, and confusing them produces bad architecture.
agent evaluation: what to measure before and after shipping — Evaluation is the layer teams postpone longest, because it looks like extra work before the first release and like insurance only after the first regression.
A buying checklist, in the order it matters — Written as a sequence, because each step makes the next one cheaper to decide.
Where SmartGate fits — SmartGate is the place in this stack where the record and the enforcement point are the same object.
Limitations — This page is a selection guide, not a benchmark: no tool is ranked, because the ranking depends on where your state, your record and your credentials already live.

Read the full piece: LLMOps: The Four Layers an Agent Stack Must Cover on smartgate.network.

DEV Community: llmops

Building a permission-aware MCP server in Python

1. Wire up token verification and audience checks

2. Enforce scopes per tool, not just at the door

3. Label tools by risk with annotations

Spec details that affect cost and latency

Verify it before you ship

Sources

Running LLM-as-a-judge on live traffic without blowing your budget

Principle 1: deterministic checks on everything, the judge on a sample

Principle 2: sample deterministically

Principle 3: never put the judge in the request path

Principle 4: make each judgment cheap

Principle 5: give the judge its own budget

Principle 6: treat the judge as an attack surface

Principle 7: calibrate before you trust the dashboard

Principle 8: only capture what you mean to judge

Principle 9: turn findings into tests

Putting it together in TokenRouter

Stable Image Core vs Ultra vs Flux Schnell: Which For Filler

What each tool cost us per call

How long each one took

How often each one failed on our side

Which one we kept and which we stopped using

The limits of this comparison

免費的 Colab CLI A100/H100 真香 :)

釐清兩款 CLI 工具

解讀點數與定價機制：Pay As You Go 隨選加購怎麼算

Can You Tell an Eval from a Review?

An eval is made of inputs, a way to run them, and grading

The word "eval" means different things to different people

Align grading with human judgment before you use it

The loop climbs through holes in the grading too

You can set a numeric goal when a tool that measures the same quantity already exists

Placing my evals on the parts

What I did on the README was a review loop

How often were the review's findings right?

I have avoided eval loops

Related links

The benchmark score is the number to trust least

AI Agent Observability: Steps, Cost, and Failure Traces

Agent cost optimization: same quality, lower price

Where the money goes

Cheaper only counts if quality holds

Change one thing, compare side by side

How this works in Rhesis

Try it

MLOps and LLMOps: Keep Your AI Reliable After Launch

What are MLOps and LLMOps?

Why AI systems fail quietly after launch

MLOps vs LLMOps: what actually changes?

How to monitor LLM models in production

What to log on every call

What to track over time

Model drift detection in production

Data drift

Concept drift

Building an LLM evaluation framework for production

Prompt versioning best practices (and model and dataset versioning)

How to roll back a bad model release

Frequently asked questions

What is the difference between MLOps and LLMOps?

How do you monitor LLM models in production?

How do you detect model drift in production?

When should we start MLOps?

Do we need MLOps consulting in India or can we build it in house?

Conclusion

The Silent Quality Collapse: Why Your LLM Router Is Secretly Destroying User Trust

Your Cost Dashboard Is Lying to You (Just on a 6-Week Delay)

System Design First: What a Production LLM Router Actually Does

The 6-Week Time Bomb: Walking the User Behavior Decay Chain

Week 1 to 2: Subtle Response Quality Drop

Week 2 to 3: Unconscious Rephrasing

Week 3 to 4: Learned Helplessness

Week 5 to 6: Alternative Evaluation

Week 6 to 7: Cancellation

The 3 Invisible Routing Failure Modes Your Dashboard Will Never Show

Failure Mode 1: Semantic Drift

Failure Mode 2: Confidence Score Miscalibration on Domain-Specific Prompts

Failure Mode 3: Cascade Timeout Silent Fallbacks