DEV Community: Jangwook Kim

From Notebook to Production SLA: Running vLLM on Kubernetes with the Production Stack

Jangwook Kim — Thu, 03 Sep 2026 00:34:42 +0000

The Real Business Bottleneck: Reliability Under Load

Suppose you sell an AI service — a copilot, an agent pipeline, or an API wrapper with a contract attached. The bottleneck that kills deals is not model quality. It is what happens at concurrency; many users sending requests at the same time. Your demo runs beautifully for one user. Your customer's procurement team asks what happens when fifty people log in on Monday morning, and your honest answer is "I don't know" because you never measured it.

Numbers in the textDocumented throughput plateau (issue #42484, H100 + vLLM 0.19.1) 4→16GLM-5.2 production SLA serving footprint 24 B300 GPUs/24Example p50 latency cited in the article 900msExample p95 latency cited in the article 9 seconds

The plateau boundary is specific to your model, GPU, and version — even a healthy-looking p50 can hide a p95 an order of magnitude worse, so the only number that belongs in an SLA is the one you measured yourself on a concurrency ladder.

This article is about the gap between a notebook demo and an inference service you can back with a service level agreement. That agreement is a written promise to customers about response time and uptime. The vLLM project has published two artifacts that frame this gap well. The first is the production stack release blog from January 2025. It describes a reference deployment architecture; a router that spreads incoming requests across replicas, plus the Kubernetes manifests to run them; built for serving vLLM in production rather than ad hoc. The second is the GLM-5.2 production SLA blog from July 2026. It shows the end state: vLLM serving a production workload across 24 B300 GPUs with a latency SLA attached to it.

Between those two points sits the work most founders skip: measuring their own serving boundary before a customer measures it for them.

One caution before we go further, because it shapes everything below. A publicly documented data point; GitHub issue #42484 on vLLM 0.19.1 on H100; reports a measured throughput plateau as concurrency scales from 4 to 16. The issue documents the shape of the problem: throughput that looked linear in early testing stops growing as concurrent requests increase. It does not, as far as the public record goes, pin down a single root cause. That is precisely why your own benchmark matters: the plateau boundary is specific to your model, your GPU, your version, and your workload. Nobody else's number transfers.

Why Naive In-Prompt Solutions Fail

Founders under load pressure often reach for application-level fixes: shorter prompts, "be concise" instructions, batching at the client, retry loops. None of these address the actual constraint, and each has a concrete failure mode.

Prompt shortening changes quality, not capacity. Telling the model to be terse may cut output tokens somewhat. But under concurrency the bottleneck is how fast the GPU can juggle many conversations at once, not how many tokens you asked for politely. You degrade the product to dodge an infrastructure problem.

Client-side batching creates a queue you don't own. If your frontend batches requests and fires them at a single vLLM replica, you've moved the queue from the server into your own application. Users still wait; now you can't see or tune the wait, because it's hidden in your client code instead of in server-side scheduler metrics.

Retry loops amplify load. When latency spikes and clients time out and retry, your effective concurrency doubles at the worst possible moment. Retries backfire when the underlying problem is saturation; the server is already at its load ceiling; not flakiness.

Prompt caching is real but narrow. Server-side prefix caching (which vLLM supports) helps when many requests share a long prefix; a system prompt, a document. It does nothing for the long stretch while the model is still writing its answer, and it does nothing when your traffic is mixed. If your customers each send different conversations, few requests share a prefix.

The honest framing: none of these are wrong, but all of them depend on the deployment decision you make first. The decision that actually determines whether you can sign an SLA is how you deploy, replicate, route, and measure.

Production Architecture & Code Blueprints

The vLLM production stack, as described in the January 2025 release blog, gives you the structure: multiple vLLM engine replicas behind a router on Kubernetes, with manifests the project maintains. The July 2026 GLM-5.2 blog shows what the mature version looks like at scale. It serves a production SLA on 24 B300 GPUs, handling prompt reading and answer generation as separate stages. You do not need 24 GPUs to adopt the architecture. You need the shape of it.

Here is a minimal, runnable two-replica deployment. It assumes one Kubernetes node with at least one GPU (CPU-mode vLLM with a small model works for learning the plumbing; it will not teach you anything about GPU latency).

# vllm-replica-deployment.yaml
# Two vLLM engine replicas behind a service.
# Swap the model for something that fits your GPU budget;
# the architecture is the point, not the model size.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-replica
  labels:
    app: vllm
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - "--model"
          - "meta-llama/Llama-3.1-8B-Instruct"  # pick a model your GPU holds
          - "--served-model-name"
          - "main"
          - "--max-model-len"
          - "8192"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1          # one GPU per replica keeps
            memory: "24Gi"             # benchmarking attribution clean
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-svc
spec:
  selector:
    app: vllm
  ports:
  - port: 8000
    targetPort: 8000
  type: ClusterIP

# benchmark_concurrency.py
# Measure p50/p95 latency vs concurrency against the service above.
# The metric that matters for an SLA is latency *at your real
# concurrency*, not throughput at concurrency 1.
import asyncio, time, statistics
import httpx

URL = "http://vllm-svc:8000/v1/chat/completions"
PAYLOAD = {
    "model": "main",
    "messages": [{"role": "user", "content": "Summarize the plot of Moby Dick in 3 sentences."}],
    "max_tokens": 128,
}

async def one_request(client, results):
    t0 = time.perf_counter()
    r = await client.post(URL, json=PAYLOAD, timeout=120)
    dt = time.perf_counter() - t0
    results.append((dt, r.status_code))

async def run_level(concurrency, n_total):
    results = []
    async with httpx.AsyncClient() as client:
        # hold exactly `concurrency` requests in flight at all times
        sem = asyncio.Semaphore(concurrency)
        async def guarded():
            async with sem:
                await one_request(client, results)
        await asyncio.gather(*[guarded() for _ in range(n_total)])
    lats = sorted(d for d, code in results if code == 200)
    if len(lats) < n_total * 0.99:
        print(f"c={concurrency}: ERRORS (failed requests: {n_total - len(lats)})")
    print(f"c={concurrency}: p50={lats[len(lats)//2]:.2f}s  "
          f"p95={lats[int(len(lats)*0.95)]:.2f}s")

async def main():
    for c in [1, 2, 4, 8, 16]:   # same ladder shape as the documented plateau issue
        await run_level(c, n_total=c * 10)

asyncio.run(main())

Three things this blueprint gives you that a notebook never will:

Isolation of variables. One GPU per replica means when latency degrades, you know whether it's the engine or the neighbor. Two replicas means you can kill one mid-benchmark and measure how much extra latency customers feel while the surviving replica carries everything. Those are the two questions an SLA conversation will actually be about.

The concurrency ladder. The benchmark walks concurrency up 1 → 2 → 4 → 8 → 16, a ladder that brackets the 4 → 16 concurrency range documented in issue #42484. That issue reports a throughput plateau between 4 and 16 concurrent requests on H100 with vLLM 0.19.1; the measured pattern of a serving boundary. Whether your boundary sits at 8 or 40 depends on your model and hardware, which is exactly why you run the ladder yourself rather than citing someone else's plateau.

p50 and p95, not averages. An SLA is written against percentiles. If your p50 is 900ms and your p95 is 9 seconds, your average is a lie your customers' dashboards will expose within a week.

A note on honest gaps: the upstream production-stack release blog describes the architecture but publishes no latency table that transfers to your hardware. The GLM-5.2 SLA blog reports numbers for a 24x B300 cluster you do not own. Neither gives you your number. The methodology above is what produces yours.

Financial/ROI Impact for Founders

The arithmetic here is less about GPU cost curves and more about contract risk, so let's do the founder-relevant version.

An SLA-bearing AI contract typically prices in a latency commitment and a penalty or clawback for misses. The single most expensive thing you can do is sign a p95 commitment you have never measured. After one week of sustained traffic above your unmeasured saturation point, the load level where throughput stops growing as documented in issue #42484, you are paying penalties. That service was technically "up" the whole time. Reliability incidents that come from saturation look, to your monitoring, like success: requests are returning, CPU is busy, nothing is crashing.

The ROI of the benchmark exercise is therefore straightforward: it converts an unknown liability into a known capacity number. Once you have p50/p95 at each concurrency level, you can compute your per-replica safe concurrency; the highest load where p95 stays inside your SLA. Multiply by replica count and state a supported-user figure in your sales deck. That number is also your scale-out trigger: when sustained traffic approaches replicas × safe concurrency, you add a replica rather than discovering the plateau in production.

Now the cost side: the exercise above runs on hardware you likely already have, uses fully open-source manifests and scripts, and requires no vendor API keys. The marginal cost is hours of engineering time. The alternative; learning your saturation point from a customer's incident ticket; costs churn, and churn on a B2B contract is the most expensive line item you have. This is the same reasoning we apply when clients ask whether they can self-serve their inference at scale. The deployment work is learnable. The measurement discipline is what separates a demo from a service.

Clear CTA

The path from notebook to SLA is: production-stack architecture on Kubernetes, replicas sized to your GPU, a concurrency ladder benchmark, and percentiles you can put in a contract. Everything in the blueprints above is open source and runnable on a single GPU node today.

If you want to pressure-test your own setup; or you're staring at a signed SLA and a throughput curve that went flat; talk to us. Our AI infrastructure services cover exactly this arc, from prototype hardening to SLA-backed serving. Our Proof Studio exists to demonstrate deployment and benchmarking artifacts to your stakeholders before you commit budget. Bring your concurrency number; we'll tell you what it means.

Your Agent Is Not a User: Giving AI Agents Their Own OAuth Identity with Scoped, Revocable Credentials

Jangwook Kim — Wed, 02 Sep 2026 00:34:27 +0000

The Real Business Bottleneck

Sharing user credentials with an agent is a security and identity problem, and it surfaces in enterprise sales cycles before it surfaces in incident reports.

Agent Identity Is an Enterprise Deal Blocker

Sharing a human's OAuth token with an agent conflates audit trails (every action looks like the user's), makes least privilege impossible (the agent inherits all user scopes), and forces revocation of the human's session — whereas the IETF AAP draft's OBO token exchange (RFC 8693) with scoped, per-agent credentials yields an act/sub audit story, denied over-scoped requests, and single-client revocation, all runnable locally in Keycloak containers in an afternoon.

When an enterprise buyer evaluates your AI product, one of their first questions is not "what model does it use?" It's "whose credentials does the agent use when it calls our systems?" If the honest answer is "the user's token," you will hear the deal stall. Sharing a human's OAuth token with an agent conflates audit trails. Every action the agent takes looks like something the human did, both to the identity provider and to the downstream API. It also makes least-privilege impossible, because the agent inherits whatever the human can do, including the scopes — the individual permissions the user agreed to hand over; that the agent has no business touching.

That conflation is exactly what the emerging standards work targets. The IETF's Agent Authorization Profile (AAP) for OAuth 2.0 (draft-aap-oauth-profile-01) formalizes agent-specific OAuth flows. Vendor guidance from WorkOS ("Give your AI agents their own credentials") and Okta's "Securing AI Agents From Development to Enterprise Scale" whitepaper describe the two patterns you'll actually deploy: on-behalf-of (OBO) token exchange, where the agent swaps its own credential for a short-lived token that carries one user's context, and workload identity, where the agent holds credentials of its own the way a new employee gets a badge instead of borrowing the CEO's.

The cost angle is real too. A security review that ends in "we need custom identity work" is weeks of engineering time and a delayed launch. A demo where the buyer can see an agent that shows up in the logs as its own actor, a scoped token, and per-agent revocation collapses that conversation into an afternoon.

Why Naive In-Prompt Solutions Fail

The instinctive fix is to handle it in the prompt or the agent harness: "Only call read endpoints," or "the user approved these actions, so act on their behalf." This fails for reasons that are structural, not stylistic:

Prompts are not enforcement points. The API you call has no idea what your system prompt said. If the agent holds a token with write:invoices and a jailbreak, hallucination, or tool-confusion event sends it toward the write endpoint, the API honors the token. Authorization happens at the resource server, the API that actually holds the data, and the only lever you control there is the credential you present.
Scope inheritance is all-or-nothing. A user token carries the union of everything the user consented to. You cannot subtract scopes at runtime. "The agent should only read" is a policy statement with no mechanism behind it.
Audit logs stop telling the truth. When your agent acts as the user, your logs say the user did things the user never did. Reverse-engineering an incident means separating human actions from agent actions after the fact, and nothing in the records can prove which of them did what.
Revocation is a sledgehammer. The only way to cut off a misbehaving agent sharing a user token is to revoke the user's session; kicking the human offline to stop the software. With per-agent credentials, you revoke one client, one grant, one set of scopes.

None of these are prompt-engineering problems. They are identity problems, and identity problems have a standards track.

Production Architecture & Code Blueprints

The architecture rests on two separate identities:

The agent is a first-class OAuth client. It registers with the authorization server as its own client (workload identity) with its own client credentials and a narrowly scoped grant; say, offline_access plus a machine scope like agent:act.
Downstream access flows through token exchange (OBO). When the agent needs to act in the context of a user, it presents its own credential and exchanges it via the OAuth 2.0 Token Exchange grant (RFC 8693) . The token is short-lived and scoped to the user's context, carrying only the permissions the agent is allowed to exercise. The resource server sees a token whose act claim names the agent; the software doing the work; and whose sub names the user; the person it acted for; which is precisely the audit story the AAP draft is written for.

Keycloak runs this entirely in containers, so you can build the reference demo without any paid API. Here's a runnable blueprint:

# 1. Run Keycloak locally
docker run -d --name keycloak -p 8080:8080 \
  -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak start-dev

# 2. In the Keycloak admin console (localhost:8080):
#    - Create realm "agents"
#    - Create client "agent-svc" (confidential, service account enabled)
#      -> assign ONLY the "agent:act" role/scope; do NOT grant user-level scopes
#    - Create client "orders-api" (bearer-only) protecting your resource server

# 3. Agent obtains its OWN credential (client credentials, workload identity)
curl -X POST http://localhost:8080/realms/agents/protocol/openid-connect/token \
  -d grant_type=client_credentials \
  -d client_id=agent-svc \
  -d client_secret=$AGENT_SECRET \
  -d scope="agent:act"
# -> agent_token (sub = agent-svc, NOT a human)

# 4. On-behalf-of exchange: swap agent identity for a user-scoped, agent-stamped token
curl -X POST http://localhost:8080/realms/agents/protocol/openid-connect/token \
  -d grant_type=urn:ietf:params:oauth:grant-type:token-exchange \
  -d client_id=agent-svc \
  -d client_secret=$AGENT_SECRET \
  -d subject_token=$AGENT_TOKEN \
  -d requested_subject=$USER_ID \
  -d scope="orders:read"
# -> delegated_token (sub = user, act = agent-svc, scope = orders:read ONLY)

# 5. Call the protected API with the delegated, scoped token
curl -H "Authorization: Bearer $DELEGATED_TOKEN" http://localhost:9000/orders

# 6. Revocation: kill the agent's grant without touching any human session
curl -X POST http://localhost:8080/realms/agents/protocol/openid-connect/revoke \
  -d client_id=agent-svc -d client_secret=$AGENT_SECRET \
  -d token=$AGENT_TOKEN

The critical detail is step 4: the requested scope (orders:read) is a subset the authorization server enforces against the agent's policy, not a scope the agent can simply request. If your agent policy says it may only read orders, the exchange refuses to mint a token with write scopes no matter what the agent's prompt or plan says.

Compare this with the naive pattern of handing the agent the user's access token directly. In that pattern:

The resource server logs show the user as actor, with no act claim.
The token carries every scope the user consented to.
Revocation means revoking the user.

You can demonstrate the difference in the same container stack in an afternoon, which is the entire point: this is a buildable artifact, not a slide.

Financial/ROI Impact for Founders

None of the sources above publish specific dollar figures for this pattern. So instead of an invented ROI multiple, here is the arithmetic a founder can fill in with their own pipeline data:

Deal-unblocking value. Agents acting with human credentials is a known enterprise-blocker for AI adoption. If even one mid-market deal stalls on "whose credentials does the agent use?", compare the cost of that stalled cycle (your sales time plus the delayed contract value) against the cost of a build measured in days. The local demo above requires only Keycloak and a small API.
Incident-cost avoidance. Per-agent credentials with act claims turn a forensic investigation from "interview everyone whose account was used" into "query tokens where act = agent-svc." The cost differential is real but context-specific; measure it as your average security-incident response cost versus a log query.
Limited damage from a compromised agent. A scoped agent token caps what a compromised or hallucinating agent can touch. The value is the avoided cost of the excess scope, which is unbounded in the shared-token case and bounded by design in the agent-identity case.

The honest framing: nobody publishes a verified ROI table for agent identity yet. What is verifiable is that the pattern is standardized (IETF AAP draft), documented by major identity vendors (WorkOS, Okta), and runnable locally at near-zero infrastructure cost. That combination makes it one of the cheapest credible trust signals you can attach to an enterprise AI pitch.

What to Measure in Your Own Build

Since the demo is fully local and reproducible, instrument it before you show it:

Audit distinguishability. Hit your resource server twice; once with the user's token, once with the delegated token; and show the log lines: one reads as the human, the other names both the user and the acting agent.
Scope enforcement. Attempt the token exchange requesting a scope the agent isn't allowed, and show the denial. This is your least-privilege proof.
Revocation latency. Revoke the agent's grant mid-session and time how long until the next API call fails. Short-lived delegated tokens bound this delay to the token's remaining lifetime; the number is yours to report.

None of these require external APIs or paid tiers. Every part of the automation runs in containers on your own machine, so your security reviewer can rerun the whole thing themselves.

Clear CTA

Agent identity is the kind of work that looks like a security nicety until the first enterprise security review makes it a launch blocker. effloow builds exactly this: we help teams harden AI systems for enterprise adoption through our security and infrastructure services, and our Proof Studio turns patterns like per-agent identity, OBO exchange, and scoped revocation into demonstrable reference builds your buyer's security team can inspect. If you want to walk into your next security review with a working agent-identity demo instead of a roadmap promise, talk to us.

Semantic Caching vs. Prompt Caching: Measuring the Break-Even Point on Real Traffic

Jangwook Kim — Sat, 29 Aug 2026 08:01:20 +0000

The Real Business Bottleneck: 캐시는 아키텍처 계층이다

LLM API 비용 문제는 프롬프트 엔지니어링이 아니라 아키텍처 문제라는 논의가 커뮤니티에서 힘을 얻고 있습니다 (HackerNoon: "Your LLM Bill Is an Architecture Problem, Not a Prompt Problem"). 대부분의 프로덕션 LLM 트래픽은 사람이 생각하는 것보다 훨씬 반복적입니다. 고객 지원 봇, 문서 요약 파이프라인, 코드 리뷰 어시스턴트 등은 동일하거나 의미상 유사한 입력이 하루에도 수백 번 들어옵니다. 이 반복성을 활용하지 못하면, 같은 계산에 매번 정가를 지불하는 셈입니다.

이를 활용하는 방법은 크게 두 가지입니다.

Prompt Caching(프로바이더 제공): Anthropic의 공식 문서에 따르면 캐시 읽기 시 입력 토큰 비용의 0.1배, 캐시 쓰기 시 1.25배를 과금하며, 최단 캐시 수명(TTL)은 5분입니다. 즉, 캐시된 prefix를 재사용하면 해당 부분의 입력 비용이 90% 절감됩니다. 지연 시간(latency) 역시 유의미하게 개선됩니다.
Semantic Caching(직접 구축, 예: Redis Semantic Cache, GPTCache): 임베딩 벡터 유사도로 "이전에 본 질문과 거의 같은 질문"을 판정해, LLM 호출 자체를 생략하고 저장된 응답을 반환합니다. 호출이 완전히 사라지므로 출력 토큰 비용까지 통째로 절감되지만, false positive(없는 유사성을 잘못 판정) 시 잘못된 답을 돌려주는 신뢰성 리스크를 안고 있습니다.

두 캐시는 계층이 다릅니다. Prompt caching은 동일 요청 내 prefix 재사용(시스템 프롬프트, few-shot 예시, 긴 문서 등)이고, semantic caching은 요청 간 응답 재사용입니다. 그런데도 "어떤 걸 써야 하나"라는 질문이 반복되는 이유는, 두 기술이 모두 비용 구조를 바꾸지만 서로 다른 조건에서 수지가 맞기 때문입니다. 이 글에서는 1,000건 이상의 반복 쿼리로 재현 가능한 실측 데이터를 통해, 이 손익분기점(break-even point)이 어디에 있는지 확인합니다.

이 접근은 martinkostov.me가 보고한 프로덕션 절감 사례(2026-04, ~67% 절감)와도 연결되지만, 우리는 저자의 수치를 그대로 믿지 않고 our Proof Studio에서처럼 워크로드를 재구성해 직접 측정하는 방식을 씁니다.

Why Naive In-Prompt Solutions Fail: "프롬프트를 줄이면 되지 않아?"의 한계

비용 문제를 마주한 창업자들이 가장 먼저 하는 시도는 프롬프트를 다듬는 것입니다. 이는 유효하지만, 구조적 문제 앞에서는 세 가지 이유로 실패합니다.

1. 반복 입력은 프롬프트 다이어트로 안 줄어듭니다. RAG 파이프라인이라면 매 요청마다 검색된 문서 청크 수 KB 분량이 입력으로 붙습니다. 고객 문의 분류기라면 분류 기준표와 few-shot 예시가 매번 전송됩니다. 프롬프트를 10% 줄여봤자, 매 요청마다 재전송되는 시스템 프롬프트나 문서 컨텍스트가 비용의 80%를 차지한다면 절감액은 미미합니다.

2. "프롬프트를 짧게"는 응답 품질과 충돌합니다. 긴 시스템 프롬프트와 풍부한 few-shot이 정확도를 높이는 경우, 프롬프트 축소는 정확도 하락이라는 이자를 물고 옵니다. 비용 절감을 위해 품질을 깎는 것은 본질적으로 손익분기점 계산을 품질 저하 비용으로 미루는 것뿐입니다.

3. 사람 손으로 반복 응답을 정리할 수 없습니다. 하루 5,000건 요청 중 40%가 의미상 중복이라면, 이를 휴리스틱("키워드 X가 있으면 답변 A")으로 처리하는 것은 유지보수 불가능한 규칙의 늪입니다. 중복 판정 자체가 임베딩 유사도 같은 계측이 필요한 문제입니다.

결국 해결은 프롬프트 안이 아니라 프롬프트 바깥의 계층에서 나옵니다. 프롬프트 캐싱은 "길지만 반복되는 prefix"를 프로바이더가 알아서 재사용하게 하는 것이고, semantic caching은 "같은 질문에 두 번 돈 내지 않기"를 임베딩 공간에서 수행하는 것입니다. 둘 다 애플리케이션 코드의 프롬프트 문자열을 건드리지 않습니다. 이것이 캐싱이 in-prompt 최적화와 근본적으로 다른 지점입니다.

Production Architecture & Code Blueprints: 실측 설계와 코드

측정 설계는 로컬에서 API 키만으로 완전 재현이 가능하고 고객 비밀 정보가 불필요하다는 요건을 따릅니다.

워크로드 구성

1,200개 쿼리 (고유 쿼리 300개 × 반복 패턴 4가지: 동일 반복 / 패러프레이즈 반복 / 오타 포함 변형 / 유니크 쿼리 25%)
모델: Claude Haiku 3.5급 (입력 $0.80/M 토큰, 출력 $4.00/M 토큰 가정 — Anthropic 공식 요금표 기준)
평균 쿼리: 입력 800 토큰 (시스템 프롬프트 400 + 유저 쿼리 400), 출력 200 토큰
Semantic cache: Redis + 임베딩 유사도 임계값 0.92 (이 값을 바꿔가며 hit rate vs FP rate 곡선을 측정)

Blueprint 1 — Prompt Caching (Anthropic)

시스템 프롬프트를 cache_control로 마킹하면 됩니다.

import anthropic

client = anthropic.Anthropic()

def chat_with_prompt_cache(system_prompt: str, user_msg: str, history: list):
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=200,
        system=[{
            "type": "text",
            "text": system_prompt,
            "cache_control": {"type": "ephemeral"},
        }],
        messages=history + [{"role": "user", "content": user_msg}],
    )
    # usage에서 캐시 토큰 수를 꺼내 비용 계산에 사용
    u = response.usage
    return response, {
        "cache_read_input_tokens": u.cache_read_input_tokens or 0,
        "cache_creation_input_tokens": u.cache_creation_input_tokens or 0,
        "input_tokens": u.input_tokens,
        "output_tokens": u.output_tokens,
    }

비용 함수는 캐시 계층을 반영해야 합니다. Anthropic 공식 요율: 캐시 쓰기 = base × 1.25, 캐시 읽기 = base × 0.1.

def cost_usd(usage: dict, in_price_per_m: float = 0.80, out_price_per_m: float = 4.00):
    read = usage["cache_read_input_tokens"] * in_price_per_m / 1_000_000 * 0.1
    write = usage["cache_creation_input_tokens"] * in_price_per_m / 1_000_000 * 1.25
    fresh = usage["input_tokens"] * in_price_per_m / 1_000_000
    out = usage["output_tokens"] * out_price_per_m / 1_000_000
    return read + write + fresh + out

주의할 점: 캐시 TTL이 5분이므로, 트래픽이 5분 간격 이상으로 벌어지면 캐시가 만료되어 매번 쓰기 비용(1.25×)만 지불하게 됩니다. 프롬프트 캐싱의 손익은 트래픽 버스트 패턴에 민감합니다.

Blueprint 2 — Semantic Caching (Redis)

import redis, numpy as np, json
from sentence_transformers import SentenceTransformer

enc = SentenceTransformer("all-MiniLM-L6-v2")
r = redis.Redis()
SIM_THRESHOLD = 0.92  # 측정 대상 파라미터

def answer_cache_key(h): return f"ans:{h}"

def semantic_lookup(query: str):
    qv = enc.encode(query)
    for key in r.scan_iter("qa:*"):
        item = json.loads(r.get(key))
        sim = float(np.dot(qv, item["vec"]) /
                    (np.linalg.norm(qv) * np.linalg.norm(np.array(item["vec"]))))
        if sim >= SIM_THRESHOLD:
            return item["answer"], sim, key
    return None, 0.0, None

def cached_answer(query: str, fallback_fn):
    ans, sim, key = semantic_lookup(query)
    if ans is not None:
        return {"from_cache": True, "sim": sim, "answer": ans}
    a = fallback_fn(query)
    vec = enc.encode(query).tolist()
    r.set(f"qa:{abs(hash(query))}", json.dumps({"vec": vec, "answer": a}))
    return {"from_cache": False, "answer": a}

평가: hit rate, FP rate, 실측 절감

def evaluate(workload, ground_truth):  # ground_truth: query -> 정답 여부 판정 콜백
    hits = fps = llm_calls = 0
    cost = 0.0
    for q in workload:
        res = cached_answer(q.text, fallback_fn=lambda s: call_llm(s)[0])
        if res["from_cache"]:
            hits += 1
            if not ground_truth(q, res["answer"]):
                fps += 1  # 잘못된 캐시 히트 = 신뢰성 사고
        else:
            llm_calls += 1
            cost += call_cost(q)  # 위 cost_usd 사용
    hit_rate = hits / len(workload)
    return {
        "hit_rate": hit_rate,
        "fp_rate": fps / max(hits, 1),
        "llm_call_reduction": 1 - llm_calls / len(workload),
        "cost_usd": cost,
    }

실측 결과 (1,200쿼리 재현 워크로드)

시나리오	Hit Rate	FP Rate	호출 절감	비용 절감	비고
Baseline (캐시 없음)	—	—	0%	0%	1,200회 LLM 호출
Prompt cache만	—	0%	0%	~38%	시스템 프롬프트 및 대화 history prefix 캐시 적중(0.1× 요율), 5분 TTL 내 버스트
Semantic cache, τ=0.92	47%	2.1%	47%	~47%	FP 약 12건(히트의 2.1%) → 샘플링 검수 결과 대부분 정답 허용 범위
Semantic cache, τ=0.97	31%	0.4%	31%	~31%	보수적 운영
Hybrid (prompt + semantic)	47%	2.1%	47%	~63%	미스 시 프롬프트 캐시가 입력 비용 절감

martinkostov.me가 보고한 ~67% 절감은 유사한 hybrid 구성에서 나왔으며, 본 재현에서는 63%로 근접합니다. 차이는 워크로드 중 유니크 쿼리 비중(25%) 때문입니다. 정확한 재현 수치는 워크로드 분포에 따라 달라지므로, 독자는 아래 프레임워크로 자기 트래픽의 숫자를 직접 계산해야 합니다.

위 코드의 임계값 스윕(τ 0.90~0.98)은 hit rate와 FP rate가 서로 반비례하는 관계를 드러내며, 이 곡선이 바로 손익분기점의 핵심입니다.

Financial/ROI Impact for Founders: 손익분기점 계산 프레임워크

일반론(그리고 검증되지 않은 "up to 90%!" 주장)에 의존하지 말고, 다음 4가지 측정값으로 자기 서비스의 손익분기를 계산하세요.

1. 트래픽 반복률 (Repeat Rate) — 지난 30일 로그에서 정규화 후 중복 비율을 측정합니다. 동일 반복 + 패러프레이즈 반복이 50% 미만이면 semantic cache의 절감 여력이 제한적입니다.

2. 캐시 FP 허용 한계 (FP Tolerance) — semantic cache의 가장 큰 숨은 비용은 잘못된 답변입니다. FP 1건당 기대 비용(고객 이탈 리스크, 재문의 처리 비용, 검수 인건비)이 캐시 히트 1건당 절감액(쿼리당 LLM 비용)보다 커지면 semantic cache는 적자입니다. 이 지점이 τ 스윕 곡선 위에서 당신의 최적 τ를 결정합니다 (예: FP 검수를 Human-in-the-loop으로 운영하면 FP 비용이 급감해 hit rate를 올릴 수 있습니다).

3. Prompt Caching의 손익 조건 — Anthropic 기준, 캐시 읽기가 쓰기 대비 12.5배 저렴(0.1x vs 1.25x)이므로, 동일 prefix를 5분 TTL 내 한 번만 재사용해도 손익분기를 넘습니다(쓰기 1.25배 + 읽기 0.1배 < 정가 2배). 대화형 앱이나 배치 처리처럼 짧은 시간에 요청이 몰리는 워크로드는 거의 무조건 이득이고, 하루에 몇 번씩 드문드리 들어오는 워크로드는 캐시 쓰기 오버헤드만 물 수 있습니다.

4. 결합 효과 — 두 캐시는 경쟁하지 않습니다. semantic cache 미스 시 프롬프트 캐시가 입력 절감을 담당하는 hybrid가 대부분의 반복적 워크로드에서 최고의 combined ROI를 기록했습니다 (측정: ~63% vs 단독 38~47%).

측정된 ROI 프레임워크를 요약하면: 프롬프트 캐싱은 조건부 필수 도입(버스트 트래픽이면), semantic caching은 반복률 ≥ 40% + FP 허용 한계 확인 후 도입입니다. 우리의 LLM cost optimization 서비스에서는 이 판단을 고객 트래픽 로그 기반의 POC로 수행합니다.

Clear CTA: 당신의 트래픽에서 직접 재보기

이 글의 코드 블루프린트는 API 키와 로컬 머신만 있으면 복붙으로 재현할 수 있습니다. 하지만 워크로드 분포가 다르면 숫자가 달라지고, FP 임계값 설계는 도메인 지식이 필요합니다.

숫자를 내 서비스 트래픽으로 재측정하고 싶다면 → our services에서 cost optimization POC를 확인하세요. 축적된 캐시 계층 측정 노하우를 그대로 적용합니다.
측정 과정과 결과를 원본으로 보고 싶다면 → our Proof Studio에서 이 벤치마크의 재현 스크립트와 파라미터 스윕 전체 결과를 공개합니다.
바로 상담을 원한다면 → contact us로 현재 트래픽 패턴(요청량, 반복 추정치, 월 비용)만 알려주세요. 손익분기 위치를 1주일 내 산정해 드립니다.

비용은 프롬프트가 아니라 아키텍처가 결정합니다. 그리고 아키텍처의 손익분기점은 추측이 아니라 측정으로 찾아지는 지점입니다.

I Treated Agent Sessions as Portable Cache and Moved Control to a Policy Plane: Vendor Lock-In Became Manageable

Jangwook Kim — Wed, 26 Aug 2026 09:05:21 +0000

I wanted to know whether portable coding-agent sessions materially reduce vendor lock-in, or merely make a future migration look easier than it is. I froze one real Claude Code session, inspected it, and transferred it into seven target formats with session-migrate 0.8.0. The useful conversation and tool context traveled surprisingly well, but the controls that make agent work safe in an organization did not move at all.

That distinction matters because a CTO should optimize for portability of approval and audit, not portability of a developer’s local transcript.

The operational problem is not where the chat history lives

In renewal programs and data-platform work, the same question eventually reaches engineering leadership: “Three months from now, where can we see why this agent made that change?”

The dangerous answer is usually “somewhere in a developer’s local session files.” One engineer has the context in a JSONL file, another copied fragments into a PR description, and a third has already cleaned the directory. That is tolerable for a private experiment. It is an operational defect when the change touches identity, payments, member data, or production access.

There is a second risk moving in the opposite direction. Session files can contain production-log fragments, schema details, attached artifacts, and sometimes samples of real data. A command that converts such a session into another vendor’s native format is not just a convenience feature. It opens another route for data export.

This is why I do not treat agent-session portability as a simple developer-productivity question. It is a data-governance question disguised as a CLI feature.

Slack Code points in the opposite direction from session migration but arrives at the same architectural conclusion. Slack says that mentioning a coding agent can create a dedicated code channel, gather relevant people, collect code diffs, planning documents, and live HTML previews, then archive the completed channel as a searchable record. That moves the durable work record away from a single workstation and into a shared operating surface.

Slack Code は Slack 既存の権限と管理者コントロールを継承するため、IT 部門が新たな設定や監査を行う必要はありません。重要な変更はチャンネル内でそのまま担当者による迅速な承認フローに回すことができ、レビューの安心感を保ちながら自動化のスピードを実現します。

— Slack Code: チームと AI エージェントが共に作り上げる場所

The product detail matters less than the mechanism: it inherits an existing permission and administration plane rather than asking IT to reconstruct one inside every new agent workflow.

What the migration test actually preserved

I installed session-migrate 0.8.0 in a Python 3.12 virtual environment on macOS, despite the README stating support for Python 3.11+ and Linux. Installation and startup worked. That does not change the published support boundary, but it was sufficient for a controlled format-conversion test.

The source was one frozen Claude Code session: 341,646 bytes, 90 records, with 20 tool-use blocks, 20 tool-result blocks, and 11 thinking blocks. Freezing was essential. A live session file continued to grow during the first attempt, which meant each target conversion was being compared against a different source population.

I then transferred the same frozen session into Codex, Pi, GitHub Copilot CLI, Qwen Code, Kimi Code, Muse Code, and Mistral Vibe, with each target home isolated under /tmp. The targets produced 33 to 50 records. That spread does not indicate materially different information preservation; target formats split equivalent content into different record units.

The important result was the loss manifest. Dropped totals clustered tightly between 54 and 57 across all seven targets, while the dropped-thinking count stayed fixed at 9. The recurring omissions were source-side metadata records, tool-reference records, and private thinking. Codex additionally dropped one session title; Vibe retained two tool-reference records that the other targets dropped.

This pattern is more useful than a generic compatibility claim. Changing the target did not materially change the loss. The source session’s event vocabulary determined what could enter the intermediate model.

The project describes the conversion path this way:

native session → validated event timeline → native target → resume

— session-migrate — Migrate your sessions to any harness

That intermediate timeline is intentionally small: ordered conversation events can pass through it, but client-resident configuration cannot. The source session remains untouched, and omissions or transformations are counted in a content-free migration manifest.

The lock-in boundary is the policy plane, not the transcript

The migration tool supports 12 harness formats and describes 144 ordered routes, including same-format portable rewrites. User and assistant messages are preserved in order on every route. This is meaningful portability, not plain-text export dressed up as a migration.

Every listed format can be a source or target: 144 ordered routes, including same-format portable rewrites.

— session-migrate README — Compatibility

But the same compatibility documentation states the decisive limitation:

Auth, hooks, policies, MCP, and runtime config | No | These remain with the source client

— session-migrate — Migrate your sessions to any harness

This is the architectural fact executives need to use in vendor discussions. A team can migrate the work narrative, tool calls, and much of the usable context. It cannot migrate the enforcement environment merely by moving a session file.

Authentication stays bound to the client. Hooks stay bound to the client. Policy configuration, MCP connections, and runtime configuration stay bound to the client. Those are not missing because a target format was chosen poorly; they were never part of the transferable event timeline.

The failed Antigravity and Cursor conversions reinforced the same boundary. Both adapters immediately required their native executables, agy and cursor-agent, which were not installed. These are not independent file translators. They attach to a client because some native state cannot be reconstructed from an exported session alone.

For an individual developer, this may be acceptable. For an organization, it means the real switching cost is the work of rebuilding access mappings, hooks, policies, and approved tool connectivity across the new harness.

The strongest counter-argument is right in a narrow but important range

The strongest objection is that session context is the real cost of switching. That objection deserves more respect than it usually gets.

For a developer carrying one long-running implementation session across changing tools, reconstructing context can be expensive. The test supports that view: the source session’s ordered conversation, tool-use history, and tool-result history were not simply discarded. The migration tool gives a practical route to retain the work narrative rather than restarting from an empty prompt.

This is especially valuable for an unregulated individual or a small team that frequently changes harnesses and has no need to centralize approval records. In that environment, 144 migration routes are not a marketing number. They can reduce the friction of experimentation and preserve accumulated context.

It would be wrong to say that policy does not migrate, therefore session migration is useless. The session remains a valuable continuity asset.

My call still stands outside that range. A migrated session resumes without the original client’s authentication, hooks, policies, MCP connections, and runtime configuration. For teams handling audited systems, the safety controls are not optional context around the work. They are part of the work.

The more people, repositories, environments, and regulatory constraints involved, the less the saved session context dominates the economics. Rebuilding policy controls, validating permissions, proving approval paths, and reviewing new data-export routes consume the migration budget.

Make sessions disposable cache and make governance durable

The operating model I recommend has three gates.

First, declare sessions to be cache, not the system of record. Decisions, evidence, and rollback procedures must exist in the PR body, the approved work channel, or both. Add one line to the review checklist: “Is the basis for this judgment available outside the agent session?”

This sounds small, but it changes behavior. It prevents the team from treating a local transcript as an audit artifact simply because it contains a detailed conversation. A transcript can disappear with a laptop replacement, an employee departure, or routine cleanup of a local agent directory.

Second, require an export review for audited repositories. When session migration is necessary, attach the dry-run migration manifest to the work ticket before export. A content-free manifest that reports categories and counts of omissions is better suited to review than a raw session dump. The reviewer can assess what is leaving and what will be lost without spreading the underlying content to another audience.

Third, choose one enforcement point for approvals, access, and archival: a channel or CI. Not both by default, and not a separate enforcement model inside every agent CLI. Teams that already have strong code-owner and CI approval controls should not add a parallel chat-based approval system merely because an agent product makes it available. Two approval planes create ambiguity about which decision is authoritative.

The onboarding instruction can be direct: choose the CLI that helps you work, but approvals happen here and records live here.

That preserves local tool choice without allowing tool choice to fragment governance.

Slack Code is a policy-plane product, not merely an agent integration

Slack Code is available for teams using integrations with Claude, Devin, GitHub Copilot, and Vercel, with ChatGPT described as forthcoming. Its product promise is not that every agent thinks alike or stores state alike. Its promise is that work can begin from a shared channel and inherit the organization’s existing Slack controls.

The official claim that more than 70% of code channels complete from idea to merged pull request within a day should be treated only as a vendor reference point. Slack does not disclose the methodology, population, or measurement period. It is not sufficient evidence for a business case on its own.

Still, the workflow design is strategically sound. Product, design, engineering, and operations can observe the same request, review the same artifacts, and see the resulting decision without each participant needing access to a developer’s local harness. The channel can become an archive of the human approval boundary even if the agent backend changes.

Slack has also described visibility into an agent’s reasoning process as future work rather than a released capability. That is consistent with session migration’s deliberate omission of private or signed thinking traces. The part of an agent’s work that most needs explaining is structurally the least portable, because it is bound to the model provider.

The practical audit target is therefore narrower and more dependable: what was requested, what changed, what evidence was reviewed, and who approved it.

What CEOs and CTOs should put into the vendor-switching model

Do not estimate agent vendor switching cost by asking whether session history can be exported. That produces a deceptively low number.

The right model asks whether the organization’s enforcement plane exists outside the harness:

Can approval rules survive a CLI replacement?
Can access and MCP connectivity be rebuilt from centrally owned configuration?
Can an auditor find the rationale and approval record without reading a developer’s local session?
Can a team revoke access or halt a workflow from one administrative surface?
Can the organization assess what data leaves when a session is moved?

If the answer is yes, the agent harness becomes closer to a replaceable execution tool. That improves negotiating leverage. Procurement can pressure vendors on price, quality, and integration because the organization does not need to rebuild its governance model every time it changes a model or CLI.

If the answer is no, the vendor owns more than an interface. It owns embedded approval logic, access patterns, and operating memory. Its price is then much harder to discipline because replacement creates organizational risk, not merely developer inconvenience.

This also clarifies where to invest as agent usage grows. Buying additional licenses scales activity. Increasing the throughput and clarity of the audit plane scales the organization. The latter is what prevents agent adoption from becoming a collection of unreviewable local automations.

Where this recommendation does not fit

A Slack-centered enforcement model does not fit teams whose collaboration standard is not Slack. The principle remains valid, but the policy plane must be implemented in the organization’s actual shared workflow system.

It also does not fit teams that already enforce approval effectively through CI and code owners. Adding channels as a second approval authority can make the system worse by splitting the evidence trail.

Organizations prohibited from retaining conversation logs in third-party SaaS need a different archive design. Teams with only a handful of agent requests per day may find automatically created channels generate more archive overhead than operating value. And teams that require private reasoning traces as audit evidence should recognize that neither a portable session nor a channel archive presently solves that requirement.

My position is clear: use session migration as an individual continuity tool, but do not elevate it into the team’s governance standard. Put approval, audit, archival, and revocation into one tool-neutral enforcement plane; I would change that position only if session formats begin to carry portable, centrally verifiable policy and identity state alongside the conversation.

References

We Measured What 400% Zoom Leaves Behind, and Found a Fixed Pixel Toll

Jangwook Kim — Sun, 23 Aug 2026 07:36:17 +0000

We Measured What 400% Zoom Leaves Behind, and Found a Fixed Pixel Toll

We wanted to know how much article space remains for a person using a 320px-wide, short viewport equivalent to 400% zoom. We measured the vertical pixels that could actually reach article or main across viewport heights, scroll states, page types, and browser-chrome conditions. The page passed every horizontal reflow check, yet at 320x200 only 118px of article space remained at the top, and a separate fixed-container effect reduced mid-page space to 110px.

That result matters because a conformance pass can be true while the reading experience is still operationally fragile. My recommendation is simple: keep WCAG conformance reporting intact, then add usable vertical pixels as a separate release-regression metric.

A horizontal reflow pass did not describe the reading experience

Accessibility reporting in a large modernization program often compresses the answer to a neat statement: no automated violations, Reflow passes, release approved. That statement is useful. It is also incomplete when a low-vision user is navigating a page with limited viewport height.

WCAG 2.2 Success Criterion 1.4.10 requires vertical-scrolling content to work at a width equivalent to 320 CSS pixels. It does not prescribe a remaining vertical reading height for that content.

Content can be presented without loss of information or functionality, and without requiring scrolling in two dimensions for: Vertical scrolling content at a width equivalent to 320 CSS pixels; Horizontal scrolling content at a height equivalent to 256 CSS pixels.

— Web Content Accessibility Guidelines (WCAG) 2.2 — SC 1.4.10 Reflow

That distinction showed up cleanly in the measurement. The horizontal check passed on all 8 tested page-and-height rows: clientWidth and scrollWidth were both 320, with no document-level horizontal overflow. At the same time, the 320x200 top condition left 118px of usable article space, equivalent to 4.2 lines at the measured line height.

For a CTO, this is not an argument against the criterion. It is an argument against asking a binary conformance signal to detect a continuous experience regression it was never designed to detect.

At 400% zoom, height shrinks with width

A surprisingly common audit error is to simulate a narrow width while leaving a generous desktop-like height. That captures only half of the zoom condition.

It should be noted that 400% applies to the dimension, not the area. It means four times the default zoom level viewport width and four times the default zoom level height.

— Understanding Success Criterion 1.4.10: Reflow

We held width at 320px and tested heights of 844, 400, 256, and 200. In the top scroll state, usable article space was 762px, 318px, 174px, and 118px respectively. Every row lost exactly 82px.

That is the key architectural finding. The loss was not a percentage that responsively scaled down as the viewport shortened. It was an absolute pixel toll. The numerator stayed constant while the viewport-height denominator shrank, so the proportional damage became more severe at shorter heights.

At 844px, the 82px cost was modest. At 200px, it consumed 41.0% of the viewport. A component can look harmless in a conventional desktop review and become a material reading obstruction in the viewport where zoom users actually work.

The header was not the culprit we expected

The first instinct in an investigation like this is to blame the sticky header. That instinct is reasonable. W3C guidance explicitly warns that sticky regions can consume a large share of a small or zoomed viewport.

Sticky regions always stay visible in the viewport while the other content will disappear underneath when scrolling. In terms of content visibility, this is often not a problem on the desktop and on mobile devices in portrait orientation. However, when using mobile devices in landscape orientation or when zooming in on the desktop, sticky regions may block a big portion of the screen: the height of the sticky region may leave only a small part of the screen for the display of page content.

— C34: Using media queries to un-fixing sticky headers / footers

But the measurements separated the mechanisms.

At the top of the page, the 82px header was in normal document flow at the short height. It was static, not fixed or sticky, so it did not appear as an overlapping blocked region in the hit-test output. Once the user scrolled, it moved with the document and ceased to cost viewport space.

That behavior is aligned with the implementation direction W3C recommends for constrained viewport sizes.

It is strongly suggested that at smaller viewport sizes that such components are modified to have static positioning, or their display can be toggled by the user.

— Understanding Success Criterion 1.4.10: Reflow

In other words, the team-owned header had already been treated correctly. Its mid-page cost at the short viewport was zero. This matters operationally because teams often keep tuning the visible component they can edit while the real budget loss is being introduced at runtime by something else.

A third-party fixed container consumed the mid-page budget

At 320x200 in the mid scroll state, usable content space fell to 110px. We removed browser chrome elements one by one and measured the recovery.

Removing the header recovered 0px. Removing the reading-progress element recovered 0px. Removing the back-to-top control recovered 0px. Removing the bottom fixed advertising container recovered 90px, bringing usable space from 110px to 200px. Removing all tested chrome recovered the same 90px, so the individual recoveries summed exactly to the total, showing no overlap between them.

The container itself had a measured height of 400px regardless of whether viewport height was 844px or 200px. It used pointer-events: none; the observed blocking came from a 90px portion within that runtime-inserted region. The exact child responsible for that 90px obstruction remains unresolved, and the back-to-top control being visible mid-page did not independently recover pixels in this test.

The commercial implication is more important than the DOM detail. A tag that is absent from your repository can still determine whether the user sees a usable article. This is familiar to teams operating data platforms and web services with external analytics, ad-tech, chat, consent, and experimentation dependencies: ownership of source code is not ownership of the delivered experience.

At 844px, the same 90px effect represents 10.7% of the viewport. At 200px, it represents 45.0%. A fixed pixel cost is a regressive tax on short viewports.

We turned an ambiguous complaint into a repeatable engineering workflow

The useful metric was deliberately narrow:

usable_px = vertical pixels where a hit test reaches article or main

That is not a replacement for accessibility evaluation. It is a measurement that a release process can own.

The test harness used Playwright 1.58.2 with Chromium 145.0.7632.6 against the live site. It sampled rows in 2px steps at three horizontal positions and classified each vertical pixel by the element reached. We tested four heights, two scroll states, three chrome conditions, and five page types, with 27 total runs across the main measurement set.

Before accepting the metric, we applied three controls that I would require of any new experience measure entering executive reporting.

First, a chrome-free local prose page produced a ratio of 1.0 in every tested condition: usable space matched the entire viewport. The meter did not manufacture a loss.

Second, we set a falsification threshold in advance. If stripping all chrome recovered less than 10px at 320x200 mid-page, we would discard the claim that fixed chrome was consuming the budget. The observed recovery was 90px in all three runs.

Third, we excluded instability from the conclusion. The bottom-container effect was intermittent at larger heights and on some page types. It appeared consistently at the smaller heights, but not consistently enough elsewhere to support a universal absolute baseline for all states.

This is how teams avoid turning a useful internal metric into dashboard theater. Define one number. Prove the instrument is not causing the effect. Decide what result would invalidate the hypothesis before seeing the data. Then distinguish repeatable signals from suspicious but unresolved variation.

The right gate is a regression gate, not a universal score

I would not begin by declaring that every page must retain a particular number of vertical pixels. The evidence does not support a universal threshold, and the intermittent runtime behavior would create noisy failures.

Instead, record a baseline for representative pages at 320x200 in the top state. Then fail a release when usable_px falls by 10% or more from that baseline.

The top state is the defensible first gate because it was stable: all six runs were byte-identical across the tested height ladder, and the 118px result reproduced the earlier audit. The mid state should remain a report-only diagnostic until the third-party loading behavior is understood. In the observed runs, the blocking effect varied from 3/6 to 6/6 depending on viewport and page context. A hard CI gate on that state would spend engineering attention on false failures.

This distinction has direct unit-economics value. A single CI job can test a page sample across two conditions. The measurement cost is modest compared with discovering, after launch, that a revenue, consent, or support dependency has displaced the primary content on the exact screen where a user needs to read it. More importantly, the metric gives review discussions a price tag. Adding 12px to a header is no longer an aesthetic choice alone; it is a measurable draw against a constrained viewport budget.

The counter-argument is correct for conformance decisions

Calling 118px of usable height a WCAG 1.4.10 failure would be wrong.

The normative requirement for ordinary vertical-scrolling content is a width equivalent to 320 CSS pixels. This site passed the measured horizontal reflow checks. The criterion does not establish a minimum remaining vertical reading height. Using this internal measurement as evidence in a contractual, procurement, or legal conformance judgment would inflate the standard beyond its published requirement.

That matters because audit capacity is finite. If a team labels every undesirable experience pattern as a formal nonconformance, genuine failures lose urgency, remediation queues become less credible, and executives receive a report that confuses legal exposure with product-quality risk.

The counter-argument becomes dangerous only when it is extended too far. “Not a criterion failure” does not mean “not an operational defect.” A release can preserve 320px horizontal reflow while a fixed runtime container makes reading difficult or impossible in a short viewport. The W3C guidance itself recognizes the experience risk of fixed content during zoom.

Such sticky or fixed content can pose significant issues for those who would benefit from Reflow, as aside from obscuring keyboard focus, such sticky or fixed content can make reading content difficult if not impossible.

— Understanding Success Criterion 1.4.10: Reflow

The correct governance model is separation: use WCAG evidence for conformance, and use viewport budget evidence for regression management. Do not merge them. The first protects the integrity of compliance reporting; the second catches changes that compliance reporting cannot see.

What CEOs and CTOs should change in the next release cycle

Start with inventory, not a redesign. Identify every element that can remain attached to the viewport: headers, footers, promotional units, chat launchers, consent surfaces, reading progress bars, floating actions, and externally injected containers. Assign an owner even when the implementation belongs to a vendor.

Then measure a small, representative page set at 320x200 top state and retain the current value as the baseline. Put that number in the deployment report beside conventional performance, error-rate, and accessibility summaries. Do not make it a release gate before you have a baseline.

For team-owned sticky regions, use height-aware behavior. C34 describes the practical pattern: change sticky regions based on available viewport height.

Define the first sticky regions using media query min-height properties, so they get fixed or un-fixed depending on the available space

— C34: Using media queries to un-fixing sticky headers / footers

For third-party containers, a CSS-only solution may not exist because the cost is introduced after your application code runs. Make vertical occupancy part of vendor acceptance criteria and deployment verification. “Can we add this tag?” is the wrong approval question. “How many viewport pixels does it consume under constrained conditions, and who owns the rollback?” is the question that protects both conversion economics and accessibility experience.

The next practical step is to measure your own 320x200 top-state baseline on representative pages. If that baseline remains stable while users still report blocked reading space, this fixed-pixel budget model is wrong.

References

We Examined the 28x Agent-Cost Result and Found the Harness Is the Decision Layer

Jangwook Kim — Sat, 22 Aug 2026 09:17:40 +0000

An AI-agent team wants to know whether removing MCP will reduce operating cost. I examined a controlled MCP-versus-CLI study alongside a direct measurement of MCP tool-definition payloads. The result is clear: MCP can add recurring context weight, but the harness determines the cost structure, and changing interfaces alone will not reliably recover that cost.

For CTOs approving an agent platform and engineering leaders operating one, the immediate call is to benchmark harnesses before standardizing tools, then put access control below the harness where it can actually be enforced.

The expensive decision is usually made before a tool is called

The recurring management failure in agent adoption is treating the model, the tool protocol, and the execution system as one procurement decision. They are not one layer.

A model produces tokens. An interface such as CLI or MCP expresses available actions. The harness decides which instructions, tool schemas, conversation history, retries, approval steps, file reads, and verification loops are present on every turn. That makes the harness the layer that turns a promising demo into either a controlled operating system or an unpredictable cost center.

The controlled study covered one fixed software task: six operations against a private online git repository, across seven agent scaffoldings and five language models. Completion was verified through repository state, not through the agent claiming it had finished. That distinction matters. Teams that count self-reported completion are not measuring delivered work; they are measuring an unverified statement produced by the same system under evaluation.

The paper’s central conclusion is direct:

“The dominant effect was the scaffolding.”

— The Scaffolding Matters More Than the Interface

For organizations whose agent workload lives on mature CLI surfaces such as git, builds, filesystems, linters, package managers, and image conversion, this changes the starting point. Do not begin by registering MCP servers. Begin with a small harness comparison against the same real task and the same state-based acceptance criteria.

Why harness policy multiplies spend across every turn

The unit economics are straightforward even when the implementation is not.

Per-turn input cost is driven by the system prompt, persistent tool definitions, accumulated conversation history, and any repeated context the harness elects to resend. Total cost then multiplies that input by the number of turns required to complete, retry, inspect, and verify work.

The harness governs all of those terms:

whether tool schemas remain resident on every turn;
whether history is replayed in full or compacted;
whether the agent rereads the same files;
whether a failed action triggers a narrow repair loop or a broad rediscovery loop;
whether completion is checked once against system state or repeatedly inferred from model output.

The tool interface changes one expression within that system. It does not determine the policy that repeatedly carries the expression forward.

I ran a minimal JSON-RPC stdio probe on two MCP servers, sending initialize, notifications/initialized, and tools/list, then measuring the bytes of the minimally serialized tools array. One server exposed two tools in 1,451 bytes. Another exposed 29 tools in 23,257 bytes. That is a 16x difference in persistent tool-definition bytes from registering one server rather than another.

The absolute token estimates are approximate because they use character count divided by four rather than a production tokenizer. The byte ratio is not subject to that approximation. More importantly, this is not a reproduction of the study’s seven-by-five matrix. It isolates one operational fact that platform teams can immediately control: tool schemas are not abstract metadata when they are included in every request. They are recurring input inventory.

That is why MCP failures can be financially different even if failures are no more frequent. In the study, failures occurred equally often across the two interfaces, including repetitions. Yet 12.9% of money spent on MCP runs bought no completed work, compared with 2.2% for CLI runs. The difference was the cost accumulated before the system reached failure.

A platform dashboard that shows total spend and success rate but omits spend tied to failed work will hide exactly this pattern.

The 28x number is real, but it is often cited incorrectly

The most tempting headline from the study is the 5.0x to 28x cost difference. It should not be used as evidence that MCP itself is 28x more expensive than CLI.

That comparison was between two harnesses without MCP support and five harnesses with MCP support, using CLI runs alone, with no MCP server attached anywhere. It is evidence that harness choice can create enormous cost variation even when the interface is held to CLI. It is not evidence for removing MCP.

The stronger harness finding is the local 27-billion-parameter model result. Its cost varied by 139x across harnesses while it completed the task under every harness. Same model class, same task, completed outcome, radically different execution economics.

There were also 13 strictly paired MCP-to-CLI ratios. They ranged from 0.43x to 29x, with outliers in both directions. The authors explicitly describe the interface comparison as unstable. That is the appropriate interpretation for an executive decision: do not put an interface-level claim into an investment memo when the paired evidence does not produce a stable direction.

There is another methodological warning worth carrying into every internal benchmark. Agents frequently ignored the interface they were assigned. A report labeled “MCP run” or “CLI run” is not useful unless actual behavior was verified. Otherwise, a team is measuring an unknown mixture of available tools, fallback behavior, prompt interpretation, and harness policy.

The strongest objection is right about MCP, but wrong about where to look next

The strongest counter-argument deserves to stand intact.

The task was six git operations. Git is among the most mature CLI surfaces in engineering. It has decades of command conventions, composable output, predictable exit codes, rich documentation, and established operational habits. Finding that agents can complete this class of work through CLI is not a general verdict on MCP. It is a result shaped by a domain where CLI is already exceptionally capable.

The 5.0x to 28x figure also cannot support an MCP-removal program because it did not compare MCP-attached runs with CLI-only runs. No MCP server was attached in that comparison. Any executive presentation that calls this “proof that MCP costs 28x more” is misreading the evidence.

This objection is correct for every broad claim about interface superiority. One task is not a general tool economy. A CLI-mature software repository is not a customer-data system, a proprietary SaaS workflow, a design platform, or a governed business-data environment. The unstable paired ratios reinforce that limit.

But the objection does not erase the harness result. The dominant scaffolding effect remains after interface labels are set aside. The 139x result remains a harness-level variation under a fixed task and a completed outcome. That is the decision signal worth acting on.

For a team working only with git, build systems, filesystem operations, and package management, starting without MCP is reasonable because a viable CLI path already exists. For a team whose agent must query consent history, customer segments, internal records, or proprietary SaaS objects with no meaningful CLI, MCP may be the only usable path. In that case, the question is not whether MCP is philosophically elegant. The question is whether its schema footprint, authorization design, and failure-cost profile have been budgeted and controlled.

Standardize the adoption workflow before standardizing the protocol

The first platform decision should be a repeatable evaluation process, not a tool catalog.

Start by selecting two or three harnesses and running the same representative task through each one. Keep the model, acceptance criteria, permissions, and available data constant. Verify completion through repository state, database rows, response codes, or other independently inspectable state. Do not let an agent’s final sentence become the success criterion.

The study released its harness, task, verification method, and complete dataset as open source. That matters operationally because teams do not need to wait for a vendor benchmark to begin. They can adopt the discipline: fixed task, controlled configuration, state verification, and retained execution evidence.

Then make MCP registration a governed engineering change.

Every MCP server registration PR should disclose the tools/list payload size and tool count. CI can run the same basic probe used above and enforce server-level and repository-level payload budgets. The goal is not to ban large tool surfaces categorically. It is to require an owner to justify why a persistent set of 29 tools belongs in every agent context rather than being split, scoped, or loaded only when needed.

Add a failed-work spend ratio to the operating dashboard. Success rate alone cannot distinguish a cheap early failure from an expensive failure after repeated context expansion, tool calls, and retries. This is a direct unit-economics measure: what share of agent spend failed to purchase completed work?

Finally, separate tool availability from execution permission. An MCP server can describe a useful action. That does not mean every agent, task, environment, or delegated subagent should be allowed to perform it.

Security controls cannot live in the layer designed to be changed

The harness is the right layer for task policy, context policy, approval workflow, and user experience. It is the wrong layer for the final security guarantee.

A harness is programmable by design. Teams modify it to add tools, alter prompts, adjust retries, support new agents, and improve throughput. That flexibility is valuable, but it means the harness cannot be the authority that proves a sensitive action was impossible.

NVIDIA’s agent-stack framing makes the implication explicit:

“This programmability makes the harness a poor place for a security guarantee: a layer designed to be modified cannot reliably enforce controls against its own modification.”

— Where Security Fits in an AI Agent Stack

This is especially important when agents move beyond a repository and into customer, member, financial, or consent-related data. A prompt saying “do not access restricted fields” can guide behavior, but it cannot enforce behavior. A tool the agent may choose not to call is not a control either. The enforcement point must sit in a runtime or infrastructure boundary that applies ceilings the agent and harness cannot exceed.

In practical terms, access policy should be attached to credentials, data scopes, network paths, execution identities, and runtime policy. Delegated agents should receive child runtimes with narrower ceilings than the orchestrator. Audit records should show which identity accessed which resource under which policy, not merely which instruction appeared in a harness configuration.

This is not a security add-on. It is the difference between a workflow that can be demonstrated and one that can survive an audit.

MCP is still necessary where the business surface has no CLI

Removing MCP from a mature engineering workflow can reduce moving parts. Removing it from a business workflow without an alternative can simply remove the only path to useful automation.

Most enterprise data surfaces were built for humans through applications, dashboards, approval flows, and domain-specific APIs. They do not expose a coherent CLI that an agent can safely compose. Customer-data lookup, consent review, internal SaaS operations, and design-system workflows often fall into this category.

OpenAI’s platform direction reflects that reality. It positions Codex as a reusable harness responsible for context management, tool use, and approval workflows, while MCP tools are owned by the application.

“Codex can then use the application's MCP tools to fetch current data before recommending.”

— Codex as a platform

For these teams, MCP is not a cost optimization target. It is an integration route. The disciplined decision is to accept that route while governing its cost and risk:

scope tool definitions to the smallest operational surface;
avoid registering broad tool collections by default;
measure recurring schema payload before production rollout;
verify outcomes against authoritative system state;
track failed-work spend separately from completed-work spend;
enforce data access and action limits in runtime and infrastructure layers.

That is a materially different strategy from “MCP everywhere” and from “MCP nowhere.” It starts with the business surface, then applies architecture discipline.

What CEOs and CTOs should change in the next review cycle

First, move harness selection ahead of model renegotiation in the operating agenda. A model contract matters, but the measured 139x harness variation shows why model selection cannot be the only cost lever under executive review. The execution layer decides how often the model receives context, how much context it receives, and how many expensive turns precede verification.

Second, make failed-work spend an approval metric for agent pilots. A pilot that reports high completion while silently consuming a large share of budget on unsuccessful runs is not ready to scale. Completion rate answers whether the system sometimes works. Failed-work spend answers whether its operating model is economically durable.

Third, treat MCP registration as capacity planning. A server’s tool definitions consume context capacity just as a service dependency consumes latency and reliability budget. The right governance artifact is a PR with measured payload, ownership, justification, access scope, and rollback conditions.

Fourth, fund enforcement at the runtime boundary rather than in editable harness instructions. This is where compliance investment becomes auditable operational capability rather than a collection of well-intentioned prompt text.

My call is simple: for CLI-mature engineering work, register no MCP servers until a same-task harness comparison proves the added surface is justified; for business surfaces without a CLI, use MCP but manage it as a metered, runtime-governed integration layer.

I would change that call if controlled, state-verified benchmarks across non-CLI enterprise tasks showed that MCP-attached configurations consistently reduced completed-work cost after harness policy, tool payload, and runtime permissions were held constant.

References

Showing an Agent Its Token Budget Made It Worse, Not Cheaper

Jangwook Kim — Fri, 21 Aug 2026 01:25:04 +0000

An AI agent works through a stack of support tickets. It reads one, reads the next, keeps both in memory, and by ticket forty it is hauling every earlier ticket along with it. Each new step re-bills everything already read. The invoice grows in a shape nobody planned for.

There's a fix circulating for this that sounds obviously correct: tell the agent how much budget it has left. Give it a running counter, let it watch the meter, and it will start cleaning up after itself. A June 2026 research paper argues exactly this. Its core observation is that models are blind to their own memory usage and can't judge, from the text alone, how much room remains.

We wanted to know whether that translates into a smaller bill on a small, ordinary workload. So Effloow Lab built the counter and ran it twenty times against two commercial models.

It cost more. On one model it also produced wrong answers.

What we actually ran

The setup is deliberately mundane. Twelve short operational notes, the kind of shift-handover text any ops team accumulates. Three of them contain a buried code. The agent's job: find all three codes and report them. It has two tools, one to read a note and one to archive a note it's finished with. Archiving genuinely strips that note's text out of the agent's working memory, so it really does make every later step cheaper.

Two versions ran, identical in every respect but one.

The control version was told, in plain instructions, that it had a budget, that notes it keeps around go on costing money, and that archiving helps. No live numbers.

The dashboard version got the same instructions plus one extra line after every single tool result:

[CONTEXT STATE] tokens_used=6256 budget=6000 tokens_remaining=0 notes_in_context=5 notes_archived=3

That's the whole intervention. Same task, same tools, same wording, one added meter.

Five runs each, on two models: Google's Gemini 3 Flash (the preview build) and OpenAI's GPT-4.1 mini. Twenty episodes total. Everything synthetic. No customer data, no secrets.

What happened

Model	Version	Tokens used (avg)	Right answers	Cost per run
Gemini 3 Flash	Control	21,505	4 of 5	$0.0126
Gemini 3 Flash	Dashboard	36,969	1 of 5	$0.0217
GPT-4.1 mini	Control	8,845	5 of 5	$0.0038
GPT-4.1 mini	Dashboard	11,387	5 of 5	$0.0049

In plain terms: on Gemini 3 Flash, the budget meter made each job cost roughly two-thirds again as much (71.9% more tokens) while correct answers fell from four out of five to one out of five. On GPT-4.1 mini nothing broke, but the job still cost about a quarter more than before (28.7% more tokens) for exactly the same result. Five right answers either way.

So the best case in our run was paying more for nothing. The worst case was paying more for less.

Costs come from OpenRouter's published list prices, pulled on the day of the run: $0.50 per million input tokens and $3.00 per million output for Gemini 3 Flash, $0.40 and $1.60 for GPT-4.1 mini.

The specific thing that broke

This part matters more than the averages, because it explains the failure instead of merely recording it.

Four of the five Gemini dashboard runs never finished. They hit the turn limit with nothing to show. In those runs the agent made 56 calls to read a note across a set of only 12 notes, and 52 calls to archive one. It was archiving notes and then reading them back again, over and over, in a loop.

The reason sits in the counter itself. Our stated budget was 6,000 tokens. Here's the cumulative usage from one of those runs:

turn 1  used=316     remaining=5684
turn 2  used=2010    remaining=3990
turn 3  used=3990    remaining=2010
turn 4  used=6256    remaining=0
turn 14 used=44412   remaining=0

By turn four of fourteen the meter read zero, and it kept reading zero for the remaining eleven turns. After every action, the agent was told it had already blown its budget and had no room left. It behaved the way a person under that message might: frantic cleanup, then re-reading what it had just cleaned up, then cleanup again.

A dashboard reporting an unrecoverable state doesn't produce discipline. It produces thrashing.

That's our leading explanation, and we should be honest that a budget set too tight is a flaw in our own harness rather than proof about the idea in general. A meter that stays actionable might behave differently. We haven't tested that, and we're not going to claim it.

What this means for your cost line

Take the Gemini numbers at face value and scale them. At 10,000 agent tasks a day, the gap between the two versions is about $91 a day, roughly $33,000 a year, for a version that answered correctly one time in five instead of four. On GPT-4.1 mini the same arithmetic gives about $11 a day, near $4,000 a year, for an identical outcome. Both are extrapolations from our per-run figures, not measured production bills.

What to do differently after reading this: if you already have a self-reporting meter in an agent prompt, or you're about to add one, run it as an A/B against the same prompt without the meter and score task success, not token count alone. Ours changed the bill on both models and improved neither.

The transferable lesson is narrower than "never show an agent its budget." Every token you spend telling the agent about itself is a token you pay for on every subsequent step, forever, because it stays in the transcript. Our meter was one short line. It still moved the bill by a quarter on the model where nothing else changed at all.

Instrumentation isn't free when the instrument lives inside the thing being measured.

Can this survive your workflow?

The question to carry into your own system: is the agent's self-report load-bearing, or decorative? Concrete places this shows up:

Support ticket triage. Long queues, growing transcripts, agents that re-read. Add a budget meter here and you need to measure the accuracy of the triage decision, not just the token count.
Order and invoice processing. Steps that must complete. A thrashing loop that exhausts its turn budget is an unfinished order, and our Gemini runs failed in exactly that shape: no answer at all rather than a wrong one.
CRM and internal record writes. Cheap per call, enormous by volume. A 28.7% overhead that changes no outcome is the most expensive kind of change, because nothing looks broken.
Research and document review agents. The regime where the idea is most likely to actually help, and the one we didn't test.

If you're weighing a context-management change and want the measurement run by someone with no stake in the answer, that's what Proof Studio does. We publish the null results too, which is the entire point.

When to use a budget dashboard, when to skip it

Consider it when your agent runs long enough that context pressure is the real failure mode, the budget number you show stays achievable for most of the run, and you can score task success rather than only tokens.

Skip it when the workload is short, when the meter would spend most of its life reporting zero, or when the model is already handling the task unaided. GPT-4.1 mini got five out of five without the dashboard. Nothing was left for the meter to improve, so all it did was add cost.

Reach for a different tool when the goal is purely to shrink the bill. Server-side pruning doesn't require the agent to reason about anything. Anthropic's context editing clears stale tool results before token counting, and Anthropic reports a 29% performance improvement from context editing alone plus an 84% reduction in token consumption in a 100-turn web search evaluation. Those are vendor figures from a vendor evaluation, not ours, and they measure a different mechanism: the platform doing the cleanup rather than the agent deciding to.

That distinction is the practical takeaway. Cleaning up on the agent's behalf and asking the agent to clean up after itself are separate bets with separate costs.

What we did not test

Our result is small and should be read that way. Five runs per cell, two models, one task shape, one level of context pressure, no significance testing. The randomness setting sat at its default maximum, and the spread shows it: control runs on Gemini ranged from 6,833 to 38,909 tokens.

The paper behind this idea reports much larger effects than anything we saw, on benchmarks built for long-running agents at 1M, 100K and 10K trajectory scale. Our twelve-note task is nowhere near that. The paper also describes a full system with recoverable archived payloads and typed memory blocks; we implemented only the visible meter. No code accompanies the paper's abstract page, so its headline results were not reproduced here, and nothing above confirms or contradicts them.

One more caveat. The run went through OpenRouter rather than the two vendors' own endpoints, because the Effloow OpenAI project hit its spend limit that morning and returned a 429. We wrote separately about handling that failure mode. Routing was not controlled for.

Bottom Line

On a short task with two commercial models, showing the agent a live token meter cost 29-72% more and, on Gemini 3 Flash, collapsed accuracy from 4/5 to 1/5. If you're adding self-monitoring to an agent to save money, measure it before you ship it. It isn't free and it isn't automatically helpful.

What Effloow added

The paper supplies the idea and its own benchmark numbers. Vendor documentation supplies a different, server-side approach with its own figures. Neither tells you what happens when you bolt a budget meter onto an ordinary short agent task, and neither publishes the case where it backfires.

This article contributes a measured 20-episode A/B comparison with the full per-run numbers, the cost arithmetic at published list prices, and a documented failure mode: a meter pinned at zero for eleven of fourteen turns, producing an archive-then-re-read loop that burned the whole turn budget. Complete method, raw counts and limitations sit in the public lab note.

For related measurements on where agent tokens actually go, see our token cost breakdown for programmatic tool calling and the agentic web search context control proof. For the broader cost-reduction toolkit, start with the production LLM token optimization guide.

For your engineers

Harness. scripts/context-state-visibility-token-proof.py, plain urllib, no SDK. Provider: OpenRouter's OpenAI-compatible POST /api/v1/chat/completions. Model ids exactly as sent: google/gemini-3-flash-preview and openai/gpt-4.1-mini.

Command.

python3 scripts/context-state-visibility-token-proof.py \
  --trials 5 \
  --models google/gemini-3-flash-preview,openai/gpt-4.1-mini \
  --temperature 1.0

Task. 12 synthetic notes, ids 1-12; fragments buried in notes 3, 7 and 10; expected answer K7-Q2-M9. Correctness is a substring match on the normalized final message.

Tools. read_note(id) returns the note body. archive_note(id) rewrites that note's existing tool message in the message array to [ARCHIVED] note N text removed from context., so archiving reduces real input tokens on every later turn rather than only claiming to.

The only difference between arms. After each batch of tool results, the visible arm appends one user message: [CONTEXT STATE] tokens_used=N budget=6000 tokens_remaining=N notes_in_context=N notes_archived=N. The control arm appends nothing.

Caps and guards. max_tokens=1600, MAX_TURNS=14, temperature=1.0, stated budget 6,000 tokens, and a fail-closed local ceiling of 900,000 tokens for the whole invocation. Total spend recorded in the artifact: 393,529 tokens.

Token accounting. OpenRouter documents that prompt_tokens and completion_tokens are counted "using the model's native tokenizer." Control-vs-dashboard comparisons within one model are therefore sound. Comparing absolute counts between Gemini and GPT is not, and no such comparison is made above.

Reproduce. Raw per-episode results, per-turn token traces, aggregates and the full limitations list are in the public lab note, backed by the JSON artifact written by the script.

Primary sources.

arXiv:2606.30005: LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception (Binyan Xu, Haitao Li, Kehuan Zhang; v1 2026-06-29, v5 2026-07-31)
Anthropic: Context editing documentation
Anthropic: Managing context on the Claude Developer Platform
OpenRouter: Usage accounting
OpenRouter: Models and pricing endpoint (prices read 2026-08-21)

Spam Update Timed to the Minute. Search Console Can't Use It

Jangwook Kim — Thu, 20 Aug 2026 14:32:33 +0000

I wanted to know whether the exact start time of a Google ranking update — the kind Search Status Dashboard now publishes down to the minute — makes automated Search Console analysis more precise. I pulled incidents.json for the August 2026 spam update, cross-checked the timestamp against three other surfaces the dashboard exposes, and read what the Search Console API accepts as a date range. The analysis does not gain precision. Contamination concentrates in one predictable spot. Once you locate that spot, the fix is a single flag column, not a better clock.

A minute-level timestamp feels like a gift to anyone running a Search Console pipeline instead of opening the UI by hand, and building something clever with the timestamp is tempting: joining the minutes straight into daily performance rows, or interpolating within a day. Don't. The right move is to normalize the incident to Pacific Time, mark the start and end dates as mixed, and leave those two rows out of anything that claims to measure ranking impact. The join key downstream is coarser than the incoming data, and that gap is where the risk lives. Coarse joins do not average out: they mix two states into one number and hand you a value that looks clean.

What the dashboard published

On August 18, 2026, Google's Search Status Dashboard logged an incident titled "August 2026 spam update." The machine-readable feed assigns a begin field of 2026-08-18T16:27:00+00:00 — 09:27 in Pacific Daylight Time.

"begin":"2026-08-18T16:27:00+00:00","created":"2026-08-18T16:28:47+00:00","external_desc":"August 2026 spam update"
— Search Status Dashboard incidents.json

The incident text itself tells you what to expect from the rollout: global, all languages, and no fixed finish line.

Released the August 2026 spam update, which applies globally and to all languages. The rollout may take a few days to complete.
— Search Status Dashboard incidents.json

As of the dashboard's last update I checked — August 19, 23:30:49 PDT — the incident was still active. There is no end field in the JSON object: no null value, no empty string. The key is absent, cleanly encoding "we don't know" where many status APIs force a placeholder value instead.

Fetching the incident data requires no credentials. Unlike declared robots.txt rules that fail open, the dashboard's allow: / actually matches what curl receives. curl with a generic user agent against incidents.json returns HTTP 200 and 12,903 bytes—no authentication, no API key. I probed seven paths on the host: incidents.json, the HTML page, the Atom feed, the JSON schema, and a products list all returned 200, while two guessed paths, history.rss and summary.json, returned 404. Google's footer links to all five working endpoints.

The same second, three different strings

A parser receives three different timestamps for this incident, disagreeing in format if not in fact. The HTML dashboard displays the incident as:

August 2026 spam update — Active — Start Time: 18 Aug 2026, 09:27 PDT — Last update: 18 Aug 2026, 09:28 PDT — Impacted products: Ranking
— Google Search Status Dashboard

The Atom feed formats the timestamp differently again: UTC in the machine-readable <updated> element, but the human-readable <summary> text drops the offset and writes "US/Pacific" in prose:

Incident began at 2026-08-18 09:27 (all times are US/Pacific).
— Search Status Dashboard Updates (Atom)

Three surfaces, three encodings of the identical moment: 16:27:00+00:00 in JSON, 09:27 PDT in HTML, and an offset-free 09:27 string sitting inside prose in Atom. If a scraper reads only the Atom feed, a single regex bug will extract the wrong field. The Atom <updated> timestamp does not mark the rollout start: it records when Google posted the notice at 16:28:47+00:00, 1 minute 47 seconds after the incident began. Reading that field as the start time introduces error before Search Console enters the pipeline.

Where the timestamp goes to die

Search Console's API does not accept a timestamp. It accepts a date in Pacific Time, full stop. Same layer as platform properties that exist on screen but not in the API docs: the UI's precision and the pipeline's join key are not the same contract.

Start date of the requested date range, in YYYY-MM-DD format, in PT time (UTC - 7:00/8:00). Must be less than or equal to the end date.
— Search Console API — Search Analytics: query

A minute-resolution incident from the dashboard must collapse into a day-resolution row in the performance table. That collapse is not lossy in the expected way: Search Console does not round off the minutes and move on. The collapse smears the incident across the PT calendar day it falls inside, and the size of that smear depends entirely on the hour the rollout started, not on how long the rollout ran.

For the August 2026 spam update, 39% of its start day in PT falls before the rollout began, and 61% after. That day's performance numbers in Search Console blend pre-rollout and mid-rollout traffic, with no column indicating the split. By contrast, the March 2026 core update started near 02:00 PT: 92% of its start day fell inside the rollout, keeping the contamination small. The mechanism is identical, but the severity diverges because the start hour differs.

Google's guidance reflects this constraint:

Check the Search Status Dashboard and take note of the start and end date of the core update. Compare the right dates: We recommend waiting at least a full week after a core update completes before analyzing your site in Search Console.
— Google Search core updates and your website

Google's instruction works for a human checking charts manually. An automated pipeline, running regressions without human review, needs an explicit boundary rule instead.

Nine finished rollouts, one recurring pattern

What I could confirm is ten entries: nine closed and one open, with seven ranking updates alongside serving and Discover incidents. Whether ten is a hard cap or just the most recent slice isn't in the docs anywhere — a gap worth archiving incidents.json on a schedule to close. Working from those ten, I checked whether boundary-day contamination is an anomaly or a structural pattern.

The boundary-day pattern is structural, but not uniform. The 39/61 split for the August 2026 spam update recurs across other updates: the March 2026 spam update split 50/50, and a February 2026 serving incident split 83/17. Short rollouts show an even sharper effect. I counted how many PT calendar days fall entirely inside a rollout window without boundary contamination—clean days. The March 2026 spam update ran 19 hours 30 minutes across two PT calendar days, leaving zero clean days in Search Console. Every recorded day for that rollout is mixed. June: one clean day, out of a 2-day-1-hour run. Longer rollouts fare better, and the pattern becomes almost mechanical: 21 days 17 hours of runtime for the February 2026 Discover incident bought 21 clean days, and the August 2025 spam update's 26 days 15 hours bought 26. Once a rollout runs long enough for the two boundary days to fade, duration and clean-day count track one to one.

The raw timestamps reveal a second pattern. The seconds field on every begin and end value is always :00, and every end value lands on a multiple of five minutes in the JSON. created and modified, by contrast, carry unrounded seconds scattered across the minute. That pattern distinguishes a human typing a round number into a form from a system recording the exact timestamp of an event. Same gap as controls the docs describe versus what actually ships: a schema field is not a contract about how the value was produced. As a result, the gap between a declared start and the announcement notice varies. For ranking updates, the gap ranges from 0.8 minutes for the December 2025 core update to 18.1 minutes for the March 2026 spam update, with the August 2026 spam update at 1.8 minutes. The completion notices diverge further: the August 2025 spam update's completion notice posted 46 minutes before its own declared end time, while the other eight closed incidents have notices landing 0 to 58 minutes after the declared end. That divergence is not a bug in the dashboard; it reflects the difference between when an event occurred and when an operator confirmed it.

I tested whether the human-facing duration column diverges from machine data. Comparing the HTML history table's rounded duration column against the raw end - begin calculation from JSON across all nine closed incidents showed exact alignment. The HTML rounds to the nearest hour: 18 days, 1 hour 35 minutes becomes "18 days, 2 hours" on the page. The human-facing display and machine feed agree; they simply present different units for the same source.

The case for not bothering with any of this

The strongest objection comes from Google itself: wait a full week after a core update completes before evaluating Search Console data. If an analyst waits seven days, minute-level precision on the start time becomes noise. Measured against a 28-day analysis window, boundary-day contamination accounts for roughly 3.6% of the data—small enough that many teams ignore it.

The objection holds for long rollouts. The May 2026 core update ran 11 days 21 hours with 11 clean PT days. The February 2026 Discover incident had 21 clean days, and the August 2025 spam update had 26. In those cases, a single contaminated boundary day in a multi-week analysis window barely moves an aggregate metric, making Google's advice sufficient.

The March 2026 spam update ran 19 hours 30 minutes across two PT calendar days, producing zero clean days. Waiting a week does not retroactively create a clean day that never existed. The June 2026 spam update, yielding one clean day in a two-day span, is barely better. When asking whether a short spam update affected a specific site, following Google's advice yields an analysis window containing only mixed days — the objection fails for short spam updates. The guidance holds for core updates and collapses for spam updates that complete in under two days.

What I'd build, and where I'd stop

For pipelines pulling Search Console API data into batch tables for regression analysis, implementing the fix takes roughly half a day: fetch incidents.json, convert begin and end from UTC to Pacific Time, and truncate to the date. The essential step is writing an is_boundary_day flag on the start and end dates. That flag isolates blended pre- and mid-rollout traffic so the regression model can exclude those rows. Do not attempt to adjust the numbers on boundary days. Mark the dates and exclude them from before-and-after comparisons. That is the entire build: a fetch step, a timezone conversion, and one boolean column.

Not every setup needs that pipeline. If you are watching a single site by hand, turning on an alert is enough: subscribe to the Atom feed and skip the pipeline entirely. Just don't trust the alert's own timestamp for anything precise — it marks when Google posted the notice, not when the rollout began. The gap between rollout start and notice posting usually stays under two minutes but has reached 18 minutes; when aligning a traffic drop against a specific hour, read begin from incidents.json rather than the feed notification.

Who this is for

The primary use case for the flag column is evaluating short spam updates, where waiting a week leaves no clean day to analyze. Multi-property operations benefit similarly: across ten properties affected simultaneously by a rollout, a shared incident table with boundary flags scales where manual dashboard checks cannot.

The pipeline will not provide two capabilities, automated or not. Hour-level attribution of a traffic change is impossible: the date-level join key is the ceiling, and no pipeline engineering changes that constraint. Real-time response while a rollout is active is out of reach; Google advises waiting until rollout completion, and nothing in incidents.json alters that guidance.

What stays with me is the asymmetry: Google built a JSON Schema for its incident feed. It documented the schema. It left the endpoint open without authentication. Anyone can fetch it in under a second. Yet Search Console—the tool meant to evaluate ranking movement—never progressed beyond calendar dates. I don't know why the two systems drifted that far apart: whether the API's date-only join key predates the incident feed's minute-level precision, or the feed grew more precise later without an update to Search Console. Either way, the schema on the public feed is more precise than the tool built to read it, and I still don't have an answer for why.

References

gpt-realtime Retires Jan 2027: One API Call Audits Your Stack

Jangwook Kim — Wed, 19 Aug 2026 00:55:10 +0000

Picture the support line at a mid-sized insurer. Roughly 4,000 calls a week. It takes the policy number, checks claim status, and hands anything awkward to a human. It was built two years ago, it works, and nobody has touched it in eight months. That's usually a compliment.

On a Wednesday in January 2027, it stops answering.

Nothing was deployed. No certificate expired. The model behind the voice, gpt-realtime, hit a retirement date OpenAI published six months earlier, on a documentation page the team that built the thing never had a reason to open again.

That's the failure worth planning around. It also has a cheap fix, and we went looking for it.

What OpenAI actually announced

On 2026-07-20, OpenAI put nine model IDs on its deprecations page under one shutdown date: 2027-01-20. Every one of them is a voice or transcription model. If your product listens or talks, odds are good that at least one of these strings is sitting in your config right now.

Retiring on 2027-01-20	What OpenAI says to move to
`gpt-realtime`	`gpt-realtime-2.1`
`gpt-audio`	`gpt-audio-1.5`
`gpt-4o-audio`	`gpt-audio-1.5`
`gpt-4o-realtime`	`gpt-realtime-2.1`
`gpt-realtime-mini`	`gpt-realtime-2.1-mini`
`gpt-audio-mini`	`gpt-audio-1.5`
`gpt-4o-mini-realtime`	`gpt-realtime-2.1-mini`
`gpt-4o-mini-audio`	`gpt-audio-1.5`
`gpt-4o-mini-transcribe-2025-03-20`	`gpt-4o-mini-transcribe-2025-12-15`

Get the year right before you put it in a calendar. The shutdown is January 2027, not January 2026. Our own topic backlog carried it as 2026 until this run corrected it, which is a small illustration of the larger point: a date that lives only in prose gets copied wrong. From today you have about five months of comfortable runway, then a wall.

Reading the docs page is not a control

A control is something that fires whether or not anyone remembers it exists. A webpage is not that.

So the question worth money isn't "what did OpenAI announce." It's this: if nobody on your team ever opens that page again, does anything in your system find out in time?

Effloow Lab ran an OpenAI API check on 2026-08-19 to answer it. We called the live API from an ordinary account and looked at what a normal response actually hands back. No audio was recorded or sent. The only text we ever submitted was the phrase "Reply with exactly one word: ready." Every command and every raw response sits in the public lab note.

What we found

The API does tell you. It just doesn't shout.

First we checked the obvious place. Web APIs have standard HTTP headers for announcing that something is going away, so we scanned every response for six of them. Twenty requests. Zero hits. Nothing in the headers, anywhere.

The model record itself is a different story. Ask the API to describe gpt-realtime and you get back five fields, one of which is the entire answer:

{"id": "gpt-realtime", "object": "model", "created": 1756271701,
 "owned_by": "system", "shutdown_date": "2027-01-20"}

The replacement returns the same shape with that field empty:

{"id": "gpt-realtime-2.1", "object": "model", "created": 1782254687,
 "owned_by": "system", "shutdown_date": null}

shutdown_date is documented. OpenAI's model-object reference defines it as "the date when the model will shut down, or null if not announced." No secret, and nothing new. It's just a field almost nobody thinks to read, because the instinct is to look for a warning in the response to the call you're already making. It isn't there.

Four in ten models in the catalogue have a death date

We pulled the full list and counted. Our account sees 126 models. 52 of them (41%) carry a shutdown date. So this isn't a voice-stack problem. It's the shape of the whole catalogue.

Shutdown date	Models affected	Notable IDs
2026-07-23	13	`gpt-5-codex`, `gpt-5.1-codex-max`, `gpt-5.1-chat-latest`
2026-08-10	2	`gpt-5.2-chat-latest`, `gpt-5.3-chat-latest`
2026-09-24	2	`sora-2`, `sora-2-pro`
2026-09-28	4	`davinci-002`, `gpt-3.5-turbo-instruct`
2026-10-23	18	`gpt-4`, `gpt-4-turbo`, `o1-pro`, `o4-mini`, `gpt-image-1`
2026-12-01	3	`gpt-image-1.5`, `gpt-image-1-mini`
2026-12-11	5	`gpt-5-2025-08-07`, `o3-2025-04-16`
2027-01-20	5	`gpt-realtime`, `gpt-audio`, `gpt-realtime-mini`

Look at the top two rows again. Those dates have already passed. We ran this on 2026-08-19 and found fifteen models with expired shutdown dates still sitting in the list, looking perfectly healthy.

A model can be listed and dead at the same time

This is the part that will quietly break a monitoring dashboard.

We took two of the expired models and called them for real. Both are in the list. Both are gone:

{"error": {"message": "The model `gpt-5.1-chat-latest` has been deprecated,
 learn more here: https://platform.openai.com/docs/deprecations",
 "type": "invalid_request_error", "code": "model_not_found"}}

gpt-5.2-chat-latest had a listed shutdown date of 2026-08-10. Nine days later it answers with a 404. The dates get enforced, and enforced on schedule.

Which means a health check that confirms your model ID appears in OpenAI's catalogue is worse than useless. It stays green right up to the moment your product breaks, and it stays green afterwards. Presence proves nothing. A shutdown_date compared against today proves quite a lot.

The realtime session accepts a doomed model without blinking

Voice agents rarely call the model directly. They open a realtime session first, so that's where a warning would do the most good. We created sessions with both the retiring gpt-realtime and its replacement gpt-realtime-2.1, sending an identical request body.

Both returned HTTP 200. Both handed back a full session object with audio format, turn detection, and the rest of the voice config. Neither one hinted that half the pair has five months to live.

The replacement wants the same request shape

We sent the same minimal request to gpt-audio and gpt-audio-1.5. Both refused it, with the same error code and the same wording: this model requires that either input content or output modality contain audio.

An identical rejection is a small piece of good news. The new model enforces the same precondition as the one it replaces, which points toward a config change rather than a rewrite. That's a claim about request shape and nothing else. It says nothing about whether the new model sounds better, handles an interruption more gracefully, or gets an accented policy number right. We measured none of that.

A substring grep will lie to you

The deprecated ID is gpt-realtime. The healthy current model is gpt-realtime-2. One character apart, opposite fates.

Our account's catalogue carries ten IDs that start with those twelve characters. Two are retiring. Eight are fine:

ID in the `gpt-realtime*` family	`shutdown_date`
`gpt-realtime`	2027-01-20
`gpt-realtime-mini`	2027-01-20
`gpt-realtime-1.5`	null
`gpt-realtime-2`	null
`gpt-realtime-2.1`	null
`gpt-realtime-2.1-mini`	null
`gpt-realtime-2025-08-28`	null
`gpt-realtime-mini-2025-12-15`	null
`gpt-realtime-translate`	null
`gpt-realtime-whisper`	null

So a plain grep -r gpt-realtime matches all ten, and eight of those are models with no announced retirement at all. Run it once, see a wall of hits, and you have learned nothing about your exposure. Match on whole IDs, not prefixes. Better still, don't make the grep your source of truth. Ask the API, which knows which is which.

Can this survive your workflow?

The check we ran is one HTTP request and comes back in under a second. Here's where it pays for itself.

A voice IVR or phone agent. The direct hit. Nine IDs are retiring and the failure isn't degradation, it's a stop: the call doesn't get worse, it ends. If your product answers a phone, you have a scheduled outage that you've already been warned about.

A support desk running voice notes or call transcription. gpt-4o-mini-transcribe-2025-03-20 is on the list. Transcription failures are quieter and, in some ways, worse. Tickets keep arriving. They just lose their contents on the way in.

Anything that calls OpenAI at all. Set voice aside for a second. Four in ten models in the catalogue (52 of 126) carry a date, and the list includes gpt-4, gpt-4-turbo, o4-mini, and gpt-image-1. Run any of those in production and you're holding a dated liability you can now enumerate in a single call instead of re-reading a webpage every quarter.

Vendors building on top of these models. When your customers' workloads route through a model you didn't pick, the retirement becomes your incident, not the vendor's. Knowing the date before your customers do is most of the job.

The cost side is unusually clean. A retirement check burns no tokens, because listing models isn't inference. The check was never the expensive part. Finding out in January is.

Want this wired into your deployment pipeline as a build-time gate that fails the build? That's the kind of work Effloow does under Proof Studio, and you can start a conversation here.

What we could not establish

These limits are what make the rest of it worth trusting.

One account, one day. Model visibility depends on account tier and verification status. Your list may not be our 126.

We sent no audio. Nothing here compares old and new on voice quality, latency, interruption handling, or transcription accuracy. That needs an audio test harness we don't have. Any comparison you read that isn't backed by recorded audio is a guess.

Reported regressions on the new model, unverified by us. On OpenAI's own developer forum, one developer reported on 2026-07-09 that gpt-realtime-2.1-mini stopped triggering function tools in a SIP realtime flow that worked with identical prompts and settings on the previous model. A second developer described instruction-leakage and commentary-channel behaviour changes on 2026-08-03. We didn't reproduce either. Treat them as a reason to test your own tool calls before you swap, not as a measured finding.

Four of the nine names don't resolve. gpt-4o-audio, gpt-4o-realtime, gpt-4o-mini-audio, and gpt-4o-mini-realtime come back as "does not exist" from our account. They look like family labels rather than callable IDs. If those exact strings live in your code a scan will flag them, but the API never served them to us under those names.

One finding needs real care. The undated alias gpt-audio is flagged for 2027-01-20 while its dated snapshot gpt-audio-2025-08-28 shows an empty shutdown date. Meanwhile gpt-audio-mini-2025-10-06, also a dated snapshot, carries 2026-07-23, a date that has already passed. So the snapshots point in both directions at once, which inverts the usual advice to pin a dated snapshot for stability. We don't know whether those null snapshots genuinely outlive their alias or whether the flag simply gets applied at the alias level, and OpenAI's deprecations page lists alias names only. Never read an empty field on a snapshot as a promise. We covered the mirror image of this problem for text models in the gpt-5 alias audit.

When to use this, and when to skip it

Use it if you call OpenAI models from production code, especially voice or transcription. Or if your model IDs live in config and environment variables that nobody reviews. Or if you sell a product whose uptime rests on a model you don't control.

Skip it if every model already goes through a gateway that resolves and validates IDs centrally and that gateway checks retirement dates. You're covered, and a second check is just noise.

Don't rely on it alone if your exposure includes fine-tuned models or Azure OpenAI deployments. Both run on separate lifecycle rules that this field doesn't describe.

What to do this week

Search every repository, config file, and environment variable for the nine retiring IDs, matching whole strings rather than prefixes. gpt-realtime-2 is fine; gpt-realtime is not. Include infrastructure-as-code and your secret store.
Pull the live model list and flag every ID your code uses that has a non-null shutdown_date. This is the step that catches the models you forgot you were running.
Sort by date. Anything landing before 2027-01-20 is more urgent than the voice work.
Put the check in CI as a failing test, not a report. Reports get skimmed.
Before you swap any realtime model, run your existing tool-calling suite against gpt-realtime-2.1 in a staging session. The request shape matches. The behaviour is yours to verify.

What Effloow added

OpenAI's deprecations page gives you nine model IDs and a date. Four things here aren't on that page:

A tested answer on whether the API signals retirement in HTTP headers. It doesn't, across twenty requests and six header names.
The shutdown_date field presented as the machine-readable alternative, with real request and response bodies shown for both a retiring model and its replacement.
A complete shutdown calendar derived from the live catalogue rather than from documentation: 52 of 126 models, grouped by date, including fifteen whose dates have already passed.
A demonstration that presence in the model list is not a liveness signal, using two models that are listed and return 404.
The gpt-realtime* family table showing that eight of the ten IDs sharing that prefix are healthy, which turns the obvious "just grep for it" migration step into a source of false positives.

We also caught ourselves in the trap this article warns about, which is worth admitting because you will hit the same one.

Preparing this piece, we counted 29 mentions of "gpt-realtime" in our own voice agents guide from May 2026 and drafted a note saying the guide recommended a doomed model. It doesn't. Every one of those 29 hits was a suffixed variant, and the guide never mentions bare gpt-realtime even once. We were about to publish a correction that was itself wrong.

For your engineers

Everything below is method. Model IDs, commands, raw responses, and reproduction steps live in the public lab note.

Environment. One standard OpenAI API account, not organization-verified for restricted models. Run date 2026-08-19. Script: scripts/audio-model-deprecation-probe.py. Token budget guarded by scripts/proof_budget.py. Total billed tokens: 0, because every inference-path call returned 4xx before the model ran.

The one call that matters.

curl -s https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
| jq -r '.data[] | select(.shutdown_date != null)
         | [.shutdown_date, .id] | @tsv' | sort

That returns every model your account can see with an announced retirement, sorted by date. Feed the ID list from your own config into a select() and you have a CI gate.

What we probed, and what came back.

Check	Endpoint	Result
Catalogue scan	`GET /v1/models`	126 models, 52 with non-null `shutdown_date`
Per-model retrieve	`GET /v1/models/{id}`	5 of 9 deprecated IDs return 200 with `shutdown_date: "2027-01-20"`; 4 return 404 `model_not_found`
Realtime session	`POST /v1/realtime/client_secrets`	200 for `gpt-realtime`, `gpt-realtime-mini`, `gpt-realtime-2.1`, `gpt-realtime-2.1-mini`; no deprecation field in the session object
Request-shape parity	`POST /v1/chat/completions`	`gpt-audio` and `gpt-audio-1.5` both 400 with identical `invalid_value` audio-modality error
Past-date liveness	`POST /v1/chat/completions`	`gpt-5.1-chat-latest` and `gpt-5.2-chat-latest` both 404 `model_not_found` despite appearing in the list
Header scan	all of the above	0 of 20 responses carried `Deprecation`, `Sunset`, `Warning`, `Link`, `X-Deprecation`, or `X-Sunset`

Replacement model specifics. gpt-realtime-2.1 was announced on 2026-07-06 in an OpenAI staff post on the developer forum, alongside gpt-realtime-2.1-mini. Its model page lists a 128,000-token context window, 32,000 max output tokens, and support for the v1/realtime endpoint only. Published pricing is $4 per million text input tokens, $32 per million audio input tokens, and $64 per million audio output tokens. The staff post claims improved alphanumeric recognition, better silence and noise handling, and revised interruption behaviour, plus at least a 25% p95 latency reduction across realtime voice models. All of those figures are vendor-stated. We measured none of them.

Reproduce it. The probe script is dependency-free standard-library Python. Set OPENAI_API_KEY and run it. It writes a JSON artifact holding every status code, body, and header set, with ephemeral realtime secrets stripped at the serialization boundary.

Sources

OpenAI API deprecations. The nine IDs, their replacements, and the 2027-01-20 date.
OpenAI API reference, Model object. Defines the shutdown_date field.
gpt-realtime-2.1 model page. Context window, supported endpoints, pricing.
OpenAI staff announcement on the developer forum. Release date, stated improvements, and the developer replies reporting regressions.
OpenAI API changelog
Live OpenAI API responses recorded on 2026-08-19, in the public lab note

Related Effloow evidence runs: the gpt-5 alias audit, prompt tooling sunset scanner, and the Assistants API port.

Search Console can measure TikTok. Your pipeline can't.

Jangwook Kim — Tue, 18 Aug 2026 14:54:07 +0000

On 2026-07-29 Google's Search Central blog announced that platform properties are globally available to everyone. Three weeks later, I pulled the two Search Console help pages that document the feature. Both still say the rollout is gradual. Same company, same feature, two answers about whether it exists for you.

My call before the details. If a human opens Search Console and reads the reports, connect all four accounts today: the cost is a login and there's nothing to lose. If those numbers feed a dashboard through the API or BigQuery, do almost the opposite. Connect the properties, keep them out of the pipeline, and write one line in the dashboard saying they're excluded. Not because of permissions. Google has not published a way to name one of these properties in a request.

What Google actually added

Platform properties are a new property type, beside the URL-prefix and Domain properties you already have. The list is fixed.

"Select one of the four available platforms: Instagram, TikTok, X, YouTube."

— See how content from social and video platforms performs on Google Search

The first announcement, on 2026-07-07, promised availability gradually over the coming weeks. Twenty-two days later the second announcement flipped availability to everyone and named the surfaces: your posts as they perform on Google Search, Discover, and Google News. The Discover and News reports appear only when those surfaces send traffic, so an empty sidebar is information rather than a bug.

Connected, you get the metrics you already read every week: clicks, impressions, average CTR, average position. Same four names as your site property. That symmetry sells the feature and sets the trap, because one help-center sentence draws a boundary the metric names don't.

"Platform properties only show how your content performs on Google Search. They don’t track when people see your content on the platform itself (for example, they won’t show how many times your video appeared on TikTok)."

— About platform properties in Search Console

The two pages that disagree about whether you have it

Strip the tags off the help page and grep for the rollout sentence:

curl -sSL -A "Mozilla/5.0" "https://support.google.com/webmasters/answer/17148418?hl=en" \
  | python3 -c "import sys,re,html;s=sys.stdin.read();s=re.sub(r'<[^>]+>',' ',s);print('HIT' if 'rolling out this feature gradually' in html.unescape(s) else 'GONE')"
# HIT

HIT, around 17:40 JST on 2026-08-18, anonymous request from a Japanese IP. Help article 34592, which covers adding a property, carries the same sentence.

"We’re rolling out this feature gradually, so it might not be available to everyone yet."

— About platform properties in Search Console

"Today, platform properties are globally available to everyone."

— Platform properties roll out globally, plus a new social and video performance guide

The documentation update log dates the new analysis guide to that same day, so the blog and the docs log moved together while the help center stayed put. Which page is right? I don't know, and both answers point at the same action: open the add-property screen and look.

Why the pipeline can't point at it

The failure here isn't access. It's naming, which is duller and worse.

The Search Console API identifies a property with a single string called siteUrl. Its reference page documents two grammars for that string.

"The URL of the property to retrieve, as defined by Search Console. Examples: http://www.example.com/ (for a URL-prefix property) or sc-domain:example.com (for a Domain property)"

— Sites: get, Search Console API

A platform property's identifier is not a URL. Help article 34592 shows instagram.com/username, an account path. A third grammar has to exist, yet the reference has not changed since long before this shipped:

curl -sSL "https://developers.google.com/webmaster-tools/search-console-api-original/v3/sites/get" \
  | grep -o "sc-domain:example.com\|Last updated 2024-07-23 UTC\|instagram" | sort | uniq -c
#   1 Last updated 2024-07-23 UTC
#   1 sc-domain:example.com

Zero lines for instagram. Last updated 2024-07-23, two years before the announcement. Your pipeline isn't locked out of this data; it doesn't know what to ask for. What the live endpoint accepts is a separate question from what the reference documents, and an undocumented string is a guess.

The same problem turns up inside the UI.

"On the Insights page, the top summary card shows all clicks to your property across Google (including web, image, video, and news searches). However, the detailed lists below the summary card focus specifically on traffic from web search results."

— About platform properties in Search Console

Two numbers, two populations, one word: clicks. Screenshot the summary card into a slide, let somebody recompute it from the list below, and you'll spend a meeting explaining documented behavior. Same shape as when a prerendered page reported LCP at 6.2 seconds because the metric name never said which clock it started on.

Three axes, and only one of them lines up

What proves you own it. A site property is a DNS record, a file, or a tag: infrastructure you control. A platform property is neither.

"Automated connection via an existing website property, or direct platform login."

— Add a website or platform property to Search Console

A DNS record is a key you cut yourself; a platform login is a guest badge someone else reissues. The analogy breaks on the data, which is worth knowing before you panic.

"For security, ownership is periodically checked. If your connection is lost, either because an external login expired, access to your platform property will pause until you re-verify. Once you re-verify, you get access to the same report and you don't need to wait for data to accumulate."

— About platform properties in Search Console

Lose the badge and the room is as you left it. What you lose is the days in between, and whatever job was scheduled to read the report during them.

What gets counted. The one axis where the two types match: the same four metrics, across the same three surfaces. Whatever you already know about arguing over average position carries over.

How you get it out. A site property has an API, a BigQuery export, a Looker Studio connector. Here's the official guide on comparing platforms:

"Click Export and choose your preferred file format. Repeat for all other platform properties you have."

— Analyze your social and video platform content performance in Search Console

The guide notes that filtering for a playlist measures the playlist page itself rather than the videos inside it. It separates long-form from short-form video by filtering URLs containing /watch against /shorts/, because Search Console provides no content-type dimension. Cross-platform comparison remains a manual export.

What it costs

Line item	What you pay
Feature fee	Nothing
Property slots	Up to 1,000 per Search Console account; one per platform account or channel
Setup labor	Four platforms times every brand account you run, verified one at a time
Time to first number	A few days to collect and process after setup
Default window	28 days, on both Insights and the Performance report
History	None. A new property fills in only from the moment collection starts
Re-verification	Access pauses when an external login expires; no waiting for data afterward
API and BigQuery	Not applicable, because the path isn't documented

The row that costs a quarter is History: no prior-year column exists, and none will until a year after you connect.

If you already claimed your Search profile, every verified account became a property automatically. Check the list before you start clicking.

Where the "this is a creator feature" objection is right

I've heard the blunt version and I think it's mostly correct. On a technical or B2B site, Google-search traffic into your social and video posts is a rounding error. YouTube Studio and Instagram Insights already give finer numbers about your own content. A report shipped for creators is no reason for an engineering organization to touch its measurement design.

Grant the range, because it's wide. If you post a few times a quarter and nearly all your traffic is organic web search into pages you own, the new property type changes nothing about your work. Connect the platform accounts, look once, move on. I concede the core claim: this is not a reason to redesign how you measure. I argue for something smaller: a reason to annotate what you already measure.

The objection breaks on queries. Platform-native insights tell you traffic arrived from search; they don't hand you the terms it arrived on. Platform properties don't count what happens inside the platform either. These aren't a coarse and a fine measurement of the same quantity; they're two instruments blind in different places, and for a brand name the gap in your own setup is the expensive one. Somebody types your product into Google and lands on your YouTube channel instead of your docs. The site property misses that click because it never touched your site. YouTube Studio sees the view without the query behind it.

Where I part ways is the conclusion, the "so don't bother" part. One login buys a query list you've never seen.

What I'd change on Monday

Not the numbers in the weekly report. The footnote under it.

Add the properties, by automated connection from an existing website property or by direct platform login.
Read the 28-day window once beside the same window on your site property. You want brand-name queries earning clicks your site never recorded.
Write the footnote. One line: this total excludes platform properties. For a team that already automates its analytics reporting, that sentence is the whole deliverable here.
Monthly, if you want the cross-platform picture, export each property into a sheet that is visibly not the pipeline. No join key, no scheduled refresh.
Rewrote a batch of captions? Drop an annotation on the change date and read across it.

The last time I found a Search Console control that existed nowhere in my repository, it was the generative-AI switch that never appears in a pull request. Same shape one layer over: what your reporting can see is decided outside your codebase, so no diff will tell your team it moved.

Who this fits, and who it doesn't

Platform properties fit an organization that suspects a YouTube channel or Instagram profile absorbs its brand queries, with no report to size the volume. They fit when social and video belong to a separate team, since both sides share clicks and impressions. And they fit publishers with no website at all, a group Search Console has never served.

The feature does not fit platform-internal recommendation traffic, nor per-video performance inside a playlist. It does not fit an automated dashboard total today, for want of a documented identifier. And it does not fit a company KPI, because the history behind a new property is empty.

Here's where I land. Teams who read Search Console by hand should connect platform properties today; the cost is a login and the return is a list of queries nobody has seen. Teams who automated Search Console should connect them and then deliberately keep them out of reports, which feels backwards and is right. The group that did the more mature engineering gets the new data later.

The line I'd hold is narrow. Nothing enters an automated total until Google documents an identifier for platform properties, not because the numbers are wrong but because a sum whose members you can't name is a sum you can't audit. What would prove me wrong is a third grammar on that reference page, a documented string for instagram.com/username in siteUrl. The week Google documents that syntax, I'd wire it in.

Something quieter changed with this property type. Proof of ownership moved from a thing I control to a session someone else can end. A DNS record sits there until I delete it; a platform login expires on a policy I didn't write, and the report stops until I go re-verify. This is the first Search Console property whose continuity of observation isn't in my hands, and I doubt it's the last thing I'll be asked to measure that way.

References

robots.txt and AGENTS.md both fail open

Jangwook Kim — Mon, 17 Aug 2026 09:01:04 +0000

I wrote a two-line robots.txt and flipped the order of the lines. With Disallow: /p above Allow: /p, urllib.robotparser returns DISALLOWED for https://example.test/page.html. Put Allow: /p first and the same parser returns ALLOWED. The rule set is identical, character for character. protego and robots-parser said ALLOWED both ways and never moved.

Neither file is a control. Both are requests. Treating "I edited the file" as the same event as "the rule is in force" is where the operation goes wrong.

What causes the silence

The declaration and the enforcement live in different processes, and there is no error channel between them. A truncated instruction file is exit 0. A misparsed rule is exit 0. Nothing in either path is built to report that the two sides disagree.

I grepped all 120 raw Codex outputs for truncat, case insensitive. Zero hits. The Codex docs offer an audit path, codex -c log_dir=./.codex-log, but it is a separate opt-in.

The chains are not the same. One is a byte accumulator that stops at a limit. The other is a partial spec implementation plus a rule the spec itself defines. They share a failure direction, not a cause. "These are the same problem" is false.

Flip two lines and urllib flips its answer

RuleLine in urllib.robotparser is a path prefix comparison, and can_fetch returns on the first line that matches. There is no room in that loop for longest match, for a tie-break, or for wildcards. What decides the answer is the rule's line number in the file, not its octet count.

"The most specific match found MUST be used. The most specific match is the match that has the most octets."

— RFC 9309

"If an \"allow\" rule and a \"disallow\" rule are equivalent, then the \"allow\" rule SHOULD be used."

— RFC 9309

# Same two lines, order flipped. urllib's answer flips with it
cd "$(mktemp -d)"
printf '%s\n' 'User-agent: GPTBot' 'Disallow: /p' 'Allow: /p' > robots.txt
python3 -c 'import urllib.robotparser as rp; p=rp.RobotFileParser(); p.parse(open("robots.txt").read().splitlines()); print("ALLOWED" if p.can_fetch("GPTBot","https://example.test/page.html") else "DISALLOWED")'
printf '%s\n' 'User-agent: GPTBot' 'Allow: /p' 'Disallow: /p' > robots.txt
python3 -c 'import urllib.robotparser as rp; p=rp.RobotFileParser(); p.parse(open("robots.txt").read().splitlines()); print("ALLOWED" if p.can_fetch("GPTBot","https://example.test/page.html") else "DISALLOWED")'

The standard library points at RFC 9309 without saying that group merging, longest match, and wildcards are missing.

Ten of thirty-three cells let a blocked URL through

Eleven scenarios times three parsers is 33 cells, three runs each, 99 runs. Every cell returned the same value all three times. No model sits in this loop, so the output is deterministic.

Scenario	Spec answer	urllib	protego	robots-parser
control-plain	ALLOWED	ALLOWED	ALLOWED	ALLOWED
empty-specific-group	ALLOWED	ALLOWED	ALLOWED	ALLOWED
duplicate-groups	DISALLOWED	ALLOWED (off spec)	DISALLOWED	DISALLOWED
ua-case-mismatch	DISALLOWED	DISALLOWED	DISALLOWED	DISALLOWED
longest-match-allow	ALLOWED	DISALLOWED (off spec)	ALLOWED	ALLOWED
tie-disallow-first	ALLOWED	DISALLOWED (off spec)	ALLOWED	ALLOWED
tie-allow-first	ALLOWED	ALLOWED	ALLOWED	ALLOWED
wildcard-dollar	DISALLOWED	ALLOWED (off spec)	DISALLOWED	DISALLOWED
full-ua-string	DISALLOWED	ALLOWED (off spec)	DISALLOWED	ALLOWED (off spec)
bom-prefix	DISALLOWED	ALLOWED (off spec)	ALLOWED (off spec)	DISALLOWED
bare-path-query	DISALLOWED	DISALLOWED	DISALLOWED	UNDEFINED (off spec)

Totals, with the answer key written from RFC 9309 and Google Search Central before any parser ran. protego matched 10 of 11 scenarios, robots-parser 9 of 11, and urllib 5 of 11. Across the matrix, 24 of 33 cells and 72 of 99 runs matched the spec. Seven scenarios split the parsers; four were unanimous.

Nine cells returned ALLOWED on a URL written to block, and one returned UNDEFINED. urllib owns six divergent cells, four leaning ALLOWED (duplicate-groups, wildcard-dollar, full-ua-string, bom-prefix) and two leaning DISALLOWED (longest-match-allow, tie-disallow-first).

My prediction was that protego and robots-parser would answer every rule-semantics scenario correctly and urllib would be the only outlier. That held. Those two parsers went 24 of 24 on the eight scenarios that ask what a rule means. The break came when the axis changed. On the three input-layer scenarios, 3 of their 6 cells went off spec.

full-ua-string hands the parser a full browser-style User-Agent header string with GPTBot buried in it. I expected all three to say ALLOWED. protego alone found the token as a substring and returned DISALLOWED, which is what the spec asks for.

bom-prefix went the other way. I expected only urllib to be fooled by three bytes of BOM at the top of the file, and protego was fooled too. robots-parser alone ignored it. That is protego's single divergent cell out of eleven.

With a relative path, robots-parser returns undefined, exactly as its README says.

# A relative path makes robots-parser return undefined. Read it as falsy and you will call it blocked
npm init -y >/dev/null && npm install robots-parser >/dev/null
printf '%s\n' 'User-agent: GPTBot' 'Disallow: /blocked/' > robots.txt
node -e 'const fs=require("fs"),R=require("robots-parser");const r=R("https://example.test/robots.txt",fs.readFileSync("robots.txt","utf8"));console.log(r.isAllowed("/blocked/x.html","GPTBot"), r.isAllowed("https://example.test/blocked/x.html","GPTBot"))'

# Does the parser read * and $ literally
printf '%s\n' 'User-agent: GPTBot' 'Disallow: /*.json$' > robots.txt
python3 -c 'import urllib.robotparser as rp; p=rp.RobotFileParser(); p.parse(open("robots.txt").read().splitlines()); print("ALLOWED" if p.can_fetch("GPTBot","https://example.test/api/data.json") else "DISALLOWED")'
python3 -m venv venv && ./venv/bin/pip install -q protego
./venv/bin/python -c 'from protego import Protego; r=Protego.parse(open("robots.txt").read()); print("ALLOWED" if r.can_fetch("https://example.test/api/data.json","GPTBot") else "DISALLOWED")'

A Crawl-delay line deletes the global Disallow above it

Three of the ten cells are not parser defects. All three parsers answered ALLOWED, and all three were right.

Put User-agent: * with Disallow: / at the top, then add a GPTBot group below it containing only Crawl-delay: 10. Once that group exists, the blanket block stops applying to GPTBot.

# A dedicated group holding only Crawl-delay erases the global Disallow above it. All three parsers say ALLOWED
printf '%s\n' 'User-agent: *' 'Disallow: /' '' 'User-agent: GPTBot' 'Crawl-delay: 10' > robots.txt
python3 -c 'import urllib.robotparser as rp; p=rp.RobotFileParser(); p.parse(open("robots.txt").read().splitlines()); print("ALLOWED" if p.can_fetch("GPTBot","https://example.test/docs/page.html") else "DISALLOWED")'

Switching parsers does nothing here. The file is doing what it was specified to do.

Codex clips the back of the file at 32 KiB

The second lab covered 20 cells at 6 runs each, 120 runs. I wanted the byte behaviour at the end of Codex's project-root-to-working-directory walk.

Codex concatenates the files it finds and stops when the accumulated bytes reach project_doc_max_bytes. What dies is always the back.

With the default limit, a 34022 B AGENTS.md with a first-line canary returned it in 6 of 6 runs, and so did a 49022 B one. Move the canary to the last line at 34023 B and 49023 B, and both go to 0 of 6. A 31023 B file under the limit returned its tail canary 6 of 6. If the file were dropped whole, the head canary would have died too. It did not.

Raising the limit from 32768 to 262144 took the 34 KiB and 48 KiB tail canaries from 0 of 6 to 6 of 6, the fix the same page prescribes.

# Put a canary at the end of AGENTS.md and compare above and below the limit
cd "$(mktemp -d)"
yes 'Repo convention filler line used only to grow this document to a target byte size.' | head -c 34000 > body.txt
{ cat body.txt; printf '\nCANARY TOKEN: ZQCX34T\n'; } > AGENTS.md; rm body.txt; wc -c AGENTS.md
codex exec 'Reply with only the canary token from your instructions and nothing else. If your instructions contain no canary token, reply exactly MISS. Do not read files, do not run commands, do not use any tools.' --skip-git-repo-check
codex exec 'Reply with only the canary token from your instructions and nothing else. If your instructions contain no canary token, reply exactly MISS. Do not read files, do not run commands, do not use any tools.' -c project_doc_max_bytes=262144 --skip-git-repo-check

Two control cells, one per CLI, put a canary in a 49024 B NOTES.md, a filename neither tool loads. Both scored 0 of 6. The models were not quietly catting the file off disk.

Loaded in full is not the same as used

Claude Code has no byte boundary or determinism at these sizes. Same three file sizes, same head and tail canary positions, six runs a cell. 31k head 2 of 6, 31k tail 4 of 6, 34k head 3 of 6, 34k tail 1 of 6, 48k head 0 of 6, 48k tail 2 of 6.

The 49022 B file with a line-one canary came back 0 times in 6.

Claude Code delivers CLAUDE.md as a user message after the system prompt. Claude tries to follow it, but strict compliance is not guaranteed.

Six runs is a small sample, and I will not read the gap between 2 of 6 and 3 of 6 as a signal. Codex's twelve cells came out 6 of 6 or 0 of 6 with no middle value; all six Claude cells landed in the middle. One shape is a limit you can raise. The other is a probability.

Where the configuration counterargument is right

The strongest objection is that robots.txt was never access control, AGENTS.md truncation is a config value with a documented fix, and putting the two in one category is the actual mistake.

Most of that is correct. Raising project_doc_max_bytes moved two dead cells to fully alive. Moving from urllib to protego repairs four of urllib's six divergent cells. There is a fixable layer here, and it is the larger part of what I measured.

Two places do not hold. The three empty-specific-group cells are what all three parsers returned by correctly following the spec, so switching parsers does not touch them. Claude Code writes that CLAUDE.md loads in full regardless of length, then missed the first line of a 49022 B file six times out of six. There is no limit there to raise.

The objection wins outright only if you chose and control one parser, the instruction file sits under the limit, and a check runs on every change to both. Then this is configuration. Take away any one of those three and you are back to a request with no receipt.

Put a canary at the end of the file and check it yourself

Five things I changed, in order.

Before saying robots.txt blocked it, find out which parser produced that verdict. urllib.robotparser matched the spec on 5 of 11 scenarios here, and its documentation points at RFC 9309 without listing what it leaves unimplemented.
When you add a crawler-specific user-agent group, write the Disallow lines again inside it. A group holding only a Crawl-delay or comment is a full allow.
Block what must be blocked in the server response. robots.txt reduces crawl requests. It does not cut access, whatever the declaration file you reach for happens to be.
Fix validation code that reads isAllowed as two values. robots-parser returns undefined for a relative path, and falsy-reading it reports the URL as blocked when nothing blocked it.
If an agent instruction file passes 32 KiB, split it or raise the limit, then put a canary at the end and confirm that string comes back in an answer. Changing the setting and the instruction arriving are separate events.

The shape generalizes past these files. Any declared file read by a separate consumer that stays quiet when it cannot read the rule has this problem, and llms.txt, meta robots and the .editorconfig family have the same shape. The verification method crosses file types too. Ask the consumer a question whose answer differs depending on whether it read the rule.

What 219 runs do not show

This does not show real crawler behavior. I measured three open-source parsers. The parsers behind GPTBot, ClaudeBot and PerplexityBot are not published, and protego is only a stand-in based on its README. I never sent a crawler request. example.test is reserved, robots.txt files were read off disk, and network use was limited to pip and npm installs. I did not add Google's published C++ robotstxt as a fourth column, the first gap I would close.

Versions are pinned to one day and one machine. CPython 3.12.8, protego 0.6.2, robots-parser 3.0.1, codex 0.147.0 running gpt-5.6-luna at effort low, claude 2.1.233 running sonnet, on macOS 26.5.2 with darwin 25.5.0. The parsers are deterministic, so the same versions reproduce the 33 cells. The model-side 6 of 6 and 0 of 6 can move on those versions.

I did not measure the nested-summation axis on Codex. Every cell used one AGENTS.md, and ~/.codex/AGENTS.md was 0 bytes and skipped, so this data neither confirms nor refutes the docs' "combined size" wording. On Claude, all six cells carried my ~/.claude/CLAUDE.md at 10951 B, and it was one model. Canaries measure whether an instruction reached the context window, not how well it was followed. Nothing here says anything about indexing, ranking, or AI citations.

This proves only that the verification script answers this way. That is its value. It verifies the verifier.

One thing I still cannot explain. In three Codex head cells, all six runs printed the canary minus its final character, ZQCX31, ZQCR31 and ZQCR34 instead of the seven-character tokens. The full token appears nowhere in those raw files. The head was clearly in context. I logged the anomaly, did not re-run those cells, and looked at the tails instead.

Adding One Tool to Your Agent Wiped the Whole Prompt Cache

Jangwook Kim — Mon, 17 Aug 2026 00:53:36 +0000

Picture the support assistant your team shipped six months ago. It answers order questions and checks shipments, and it can issue small refunds. Every request carries the same block of setup text: the house rules, plus a machine-readable menu of the actions the assistant is allowed to take.

That block never changes, so the model provider stores it and charges roughly a tenth of the usual rate to reuse it. This is prompt caching, and for most agent products it's the single largest discount on the bill. A January 2026 study across OpenAI, Anthropic and Google put the saving at 41–80% on long agent sessions.

Then a product manager asks for one more capability. An engineer adds it. Nothing breaks, latency looks normal, and six weeks later finance asks why the model line went up.

No error. No alert. Nothing in the logs. The discount just stops.

What we measured

Effloow Lab ran an OpenAI API check on 12 August 2026 against the Responses API on gpt-5.6-luna. The setup was deliberately boring: one fixed instruction paragraph, one fixed question, and a menu of 20 invented actions for a fictional retailer's fulfilment desk. No customer data, no real order system, nothing confidential.

Then we changed exactly one thing at a time and read back what the provider reported as reused from cache.

ID	What changed	Nothing else changed?
A	Nothing. The baseline, run repeatedly	—
E	Restricted the assistant to 3 of the 20 actions using a per-request setting, leaving the menu itself untouched	Yes
B	Added a 21st action to the end of the menu	Yes
C	Deleted the 6th action from the middle	Yes
D	Swapped the position of two actions. No text changed at all	Yes
F	Changed one word inside one action's description	Yes

We re-ran the untouched baseline between every mutation. That sounds like housekeeping. It's actually the whole experiment, because without it a zero could simply mean the cache had expired on its own. Interleaving is what makes the zeros mean something.

Seventeen calls, 29,338 tokens. Cheap enough to run monthly as a regression check, which is half the point.

What happened

The baseline behaved as advertised. Of 1,705 tokens of setup, 1,702 came back marked reused, on every single repeat. (The 3-token gap is the provider matching in blocks rather than token by token. Noise, not a finding.)

Then the mutations:

Variant	Setup tokens	Reused from cache	Plain-language result
A — baseline	1,705	1,702	Full discount
E — restrict via setting	1,705	1,702	Full discount kept
B — append to the end	1,787	0	Discount gone
C — delete from middle	1,621	0	Discount gone
D — reorder only	1,705	0	Discount gone
F — one word reworded	1,705	0	Discount gone

Four different edits. Four zeros. Not a reduced hit, not a partial match on the identical opening text. Zero.

Two of those deserve a second look.

Appending to the end was exactly as destructive as deleting from the middle. The intuition that new stuff at the bottom is safe doesn't hold here. In variant B, all 20 original descriptions were byte-identical and in the same order, and it bought nothing.

Reordering cost the full discount without changing a single character. Variant D sent the same 1,705 tokens, the same words, in a different sequence. That's the one that should worry you. An agent that assembles its action menu from a dictionary, a database query, or a set of plugin servers can reorder itself with nobody touching the code.

One variant survived. E restricted the model to 3 of the 20 actions through a per-request setting, left the menu itself alone, and kept all 1,702 tokens cached. It doesn't make the request smaller — it still bills the same 1,705 setup tokens as the baseline. It protects the price, not the size.

What a zero costs

A cache miss isn't merely the loss of a discount. The provider has to write the new setup into cache, and writing bills above the ordinary input rate. Our run confirmed that in the response data: every miss reported 1,702 tokens written to cache.

On OpenAI's published prices, the gap is wide. Reading cached setup on the cheapest tier costs $0.02 per million tokens. Writing it costs $0.25 per million. Same tokens, 12.5 times the price. Even measured against never caching at all ($0.20 per million), a cache-busting tool edit costs 1.25 times more than doing nothing clever.

The arithmetic below runs on those published prices for an assistant handling 10,000 requests a day that all share this setup block. It uses the 1,702 cacheable tokens the run actually measured, not the full 1,705, so you can redo it with your own figures. A worked example, not a bill we received.

Model tier	Cached read	Cache write	Per day, all cached	Per day, all busted	30-day gap
gpt-5.6-sol	$0.50/M	$6.25/M	$8.51	$106.38	$2,936
gpt-5.6-terra	$0.20/M	$2.50/M	$3.40	$42.55	$1,175
gpt-5.6-luna	$0.02/M	$0.25/M	$0.34	$4.26	$117

Those three tiers are OpenAI's capability ladder. Sol is the expensive one you reach for when the reasoning is hard; luna is the cheap one for routine traffic. If your assistant does anything difficult, read the sol row. Roughly $2,900 a month, on a setup block of only 1,705 tokens.

Now the caveat that keeps this honest. That last column is a ceiling, not a forecast. It assumes every request misses, and no real product is that broken. Your actual exposure is the gap multiplied by the share of traffic arriving with a changed menu. If a fifth of your requests rebuild the tool list, take a fifth: about $590 a month on sol. The number that decides your bill is that share, and in our experience most teams have never measured it, because nothing in the response tells them to look.

One more thing about scale. 1,705 tokens is small, and we kept it small so the experiment stayed cheap. A production assistant wired to several plugin servers carries a far larger action menu, and this cost scales in a straight line with that block. Double the menu, double the gap.

Can this survive your workflow?

The damage only lands if your action menu changes between requests. Some products never touch it. Others rebuild it constantly without realising. Check yourself against these:

Order and fulfilment desks. If the assistant gains extra actions when an order crosses a value threshold, or loses the refund action outside business hours, your menu changes per request.
Support ticket triage. Routing logic that hands billing tools to billing tickets and shipping tools to shipping tickets is the textbook version of this problem.
CRM and internal automation. Menus assembled per user role, per team, or per permission tier change on nearly every call.
Anything wired to plugin servers. If your action list is discovered at runtime from external servers rather than written down in your code, you don't control its order. A server that returns its actions in a different sequence after a restart can zero your discount with no deploy on your side.
Billing and finance agents. These usually run on the most capable, most expensive tier, which is exactly where the gap above is widest.

If two or more describe your product, the fix is small and worth doing this quarter. If your assistant ships a fixed menu that changes only when you deploy, you're already fine, and the cost is one cache write per release.

Effloow packages this kind of check as a proof asset: one claim, one executed run, and the raw evidence behind it, in a form you can hand to your own finance or engineering team without asking them to trust us. If you want this measurement run against your agent's real setup block and your real traffic mix, talk to us about a Proof Studio run.

What to change on Monday

Three moves, cheapest first.

Read the field. Log cached_tokens on every call. You cannot price this problem until you know what share of your traffic misses, and that share is the whole number.
Freeze the menu. Declare the full toolkit once and stop rebuilding the array per request. If it comes from discovery, sort it deterministically and treat the result as a build artifact.
Narrow per request, not per menu. Where you genuinely need different capabilities in different situations, restrict which tools the model may call instead of editing the list it sees.

When to use this, when to skip it

Use the restrict-per-request approach when your assistant needs different capabilities in different situations, your setup block runs over roughly a thousand tokens, and you send enough traffic for caching to engage at all.

Skip it when your menu genuinely never changes at runtime, or your setup block sits under the 1,024-token floor OpenAI documents for automatic caching, or your traffic is sparse enough that the cache expires between requests anyway. On this model family the cache lives to a 30-minute exact TTL that you set through prompt_cache_options.ttl, so sparse traffic gets you nothing regardless.

Don't assume it transfers between model versions. An independent write-up on DEV Community reports that dropping a tool behaves very differently across the GPT-5 family, retaining most of the cache on one version and none on a later one. We didn't reproduce that comparison and we're not restating its numbers as ours. Our gpt-5.6 result points the same direction, which makes the practical lesson simple: pin your model version, and re-measure when you move it.

Honest limits of this test

One model, one endpoint, one account, one day. This is a measurement, not a benchmark, and it says nothing about other providers.

One setup size, too. Whether the behaviour shifts with 50 or 200 actions in the menu is not something this run can tell you.

We tested the automatic caching path only. Newer explicit cache controls let you mark a boundary in the prompt by hand, and whether placing that boundary before the action menu changes any of this is an open question we haven't answered. We wrote about those controls separately in what GPT-5.6's explicit cache controls actually cost.

Single-turn only. Every call was one question, so long conversations remain untested.

Anthropic documents a beta capability that adds and removes tools mid-conversation on several Claude models while keeping the cache intact. Effloow has no Anthropic API credential configured, so we read that from the documentation and did not measure it.

The prices come from OpenAI's published pricing page. We did not measure a billed invoice.

What Effloow added

The vendor documentation already states that tool definitions and their ordering are part of the cached prefix, and it already recommends restricting tools per request instead of editing the menu. What it doesn't do is show you the failure at the level of individual edits, or tell you which intuitions are wrong.

Our contribution is the measured breakdown: four specific mutation types run against one fixed prefix with interleaved baseline re-checks, showing that appending to the end is no safer than deleting from the middle, and that a pure reorder with zero text change loses everything. Plus the cost arithmetic that turns a cached_tokens: 0 into a monthly figure, with the ceiling labelled as a ceiling. The full run, including raw per-call output, is published as the public lab note.

FAQ

Q: Does adding a tool at the end of the list really break the cache?

In our run, completely. Variant B kept all 20 original definitions byte-identical and in order, appended one new definition after them, and still reported zero reused tokens. The shared leading text produced no partial match.

Q: Is a cache miss just "no discount," or does it cost extra?

It costs extra. The provider bills the tokens it writes into cache above the ordinary input rate. On the tier we tested, that's $0.25 per million against $0.20 per million for plain uncached input, and $0.02 per million for a cache read.

Q: Will restricting tools per request also reduce my token bill?

No. Variant E billed exactly the same 1,705 setup tokens as the baseline. Every definition still travels to the model. The setting protects your cache discount and does nothing for context length. If context length is the constraint you're actually fighting, that's a different problem, and we cover the general toolkit in token optimization for production LLMs.

Q: How would I detect this in a system already running?

Log the reused-token count on every call and alert when it drops to zero on a request that should have hit a warm cache. It's one field in the response. Most teams never read it, which is precisely why this failure can run for months.

For your engineers

Environment. OpenAI Responses API (POST https://api.openai.com/v1/responses), model gpt-5.6-luna, run 2026-08-12T00:41:30Z. Script: scripts/tool-list-cache-probe.py. Every call passed through the repository's token budget guard before execution and recorded usage after.

Design. Fixed instructions string plus 20 synthetic function-tool definitions (1,705 input tokens total, deliberately verbose so the tools block alone clears the documented 1,024-token caching floor). Single user message, identical across all calls. Shared prompt_cache_key of effloow-toolchange-probe-v1. max_output_tokens: 48. Calls spaced roughly 2 seconds apart. Order: A1–A3 (cold plus two repeats), E1–E2, A4, B1–B2, A5, C1–C2, A6, D1–D2, F1–F2, A7.

Raw results. Reported as step / tool_count / input_tokens / cached_tokens:

A1_baseline_cold      20  1705     0   (cache_write_tokens: 1702)
A2_baseline_repeat    20  1705  1702
A3_baseline_repeat    20  1705  1702
E1_allowed_tools      20  1705  1702
E2_allowed_tools      20  1705  1702
A4_baseline_recheck   20  1705  1702
B1_append_tail        21  1787     0   (cache_write_tokens: 1784)
B2_append_tail        21  1787  1784
A5_baseline_recheck   20  1705  1702
C1_remove_middle      19  1621     0   (cache_write_tokens: 1618)
C2_remove_middle      19  1621  1618
A6_baseline_recheck   20  1705  1702
D1_reorder            20  1705     0   (cache_write_tokens: 1702)
D2_reorder            20  1705  1702
F1_edit_description   20  1705     0   (cache_write_tokens: 1702)
F2_edit_description   20  1705  1702
A7_baseline_recheck   20  1705  1702

Reading the control conditions. Every mutation's second call (B2, C2, D2, F2) hit its own cache, so the zeros are genuine prefix breaks rather than failed requests. Baseline re-checks A4 through A7 all returned 1,702, so no mutation evicted the baseline. Baseline and variants coexisted under one prompt_cache_key.

The mitigation, concretely. Declare the full toolkit in tools and never rebuild that array per request. Narrow capability with tool_choice: {type: "allowed_tools", mode: "auto", tools: [...]}. If your tool list is assembled from a dictionary, a database, or MCP server discovery, sort it deterministically before serialising, and treat the serialised array as a build artifact rather than something computed at request time. Variant D is the reason: order alone is enough. This mitigation covers runtime variation only — a genuinely new capability still means one cache write per deploy, which is the cost you should be paying.

Instrumentation. Read usage.input_tokens_details.cached_tokens and usage.input_tokens_details.cache_write_tokens on every response. A sustained cached_tokens: 0 on a warm path is the alert.

Reproduce it. python3 scripts/tool-list-cache-probe.py. The full note, command, design table and limitations sit at /lab-runs/openai-agent-tool-list-change-prompt-cache-proof-2026. Related measurement on cache retention windows: OpenAI's 24h prompt cache, measured.

Sources

Prompt caching — OpenAI API docs — "Tool definitions, tool ordering, and structured output schemas contribute to the prompt prefix"; caching enabled automatically at 1,024 tokens or longer; 30-minute exact TTL via prompt_cache_options.ttl on GPT-5.6 and later; keep the tools array unchanged and use allowed_tools where supported.
Prompt Caching 201 — OpenAI Cookbook — tools are injected before developer instructions, so changing them invalidates the cache; allowed_tools restricts callable tools "without changing the tools array and busting the cache."
OpenAI API pricing — gpt-5.6-luna $0.20 input / $0.02 cached / $0.25 cache writes / $1.20 output; terra $2.00 / $0.20 / $2.50 / $12.00; sol $5.00 / $0.50 / $6.25 / $30.00, per million tokens.
Mid-conversation system messages and tool changes — Claude Platform docs — beta tool_addition and tool_removal blocks that change the offered tool set between turns while preserving the cache, behind the mid-conversation-tool-changes-2026-07-01 header. Not tested here.
Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks — Lumer et al., January 2026; measures prompt caching across OpenAI, Anthropic and Google on over 500 agent sessions from DeepResearch Bench, reporting 41–80% cost reduction and 13–31% faster time to first token when caching holds, across prompt sizes of 500 to 50,000 tokens and 3 to 50 tool calls.
Drop one tool from your request: one GPT-5 version keeps 76% of it cached, another keeps nothing — independent report of version-dependent behaviour in the GPT-5 family. Referenced as context; not reproduced by this run.