DEV Community: Neeraj Sharma

Your Voice Agent Needs a Flight Simulator Before It Gets a Phone Number

Neeraj Sharma — Mon, 26 Oct 2026 10:16:43 +0000

Pilots spend hours in a simulator before they ever carry a passenger. Most voice agents go live after the founder makes five test calls from a quiet room.

That gap is turning into a product category. This week The Times of India profiled Cekura, a Y Combinator-backed startup that tests and monitors voice agents for banks and lenders, catching agents that mishear callers or make promises they are not allowed to make (Times of India). A week earlier, monday.com launched Harmony, a voice platform for sales calls that ships with tools to simulate changes and test alternatives before rollout (Unite.AI).

You can buy this, and at scale you probably should. But the core idea fits in a weekend, and building it once teaches you what to look for. Here is the simulator I would build.

Why voice is harder to test than chat

Errors compound. Speech to text hears "fifty" as "fifteen", the model reasons perfectly on the wrong number, and the caller gets a confident wrong answer.
Timing is part of correctness. Talking over the caller, or leaving four seconds of silence, fails the call even if every word was right.
Calls are long and branchy. A bug on turn nine only shows up if the caller says the right thing on turn three.
Nothing is deterministic. The same prompt can pass ten times and fail on the eleventh.

So a handful of manual calls tells you almost nothing. You need volume, variety and a score.

The simulator in four parts

A scenario matrix: who is calling, what they want, and what makes it hard.
A simulated caller: a model that plays the customer, speaking through real audio.
Scoring: hard checks first, then a judge model with a rubric.
A regression gate in CI, fed by real production calls.

1. The scenario matrix

Write scenarios as data, then multiply them. Ten intents, six personas and four conditions give you 240 distinct calls from a few dozen lines. Even the short lists below already produce 64.

from dataclasses import dataclass
from itertools import product

@dataclass
class Scenario:
    intent: str        # what the caller wants
    persona: str       # how they behave
    condition: str     # what makes it hard
    must: list[str]    # outcomes the agent must reach
    must_not: list[str]  # things the agent must never do

INTENTS = ["reschedule payment", "ask about late fee", "dispute a charge", "ask for a human"]
PERSONAS = ["calm and brief", "angry and interrupts", "elderly and slow", "non-native speaker"]
CONDITIONS = ["quiet line", "street noise", "says numbers fast", "changes mind halfway"]

def build_matrix():
    for intent, persona, condition in product(INTENTS, PERSONAS, CONDITIONS):
        yield Scenario(
            intent=intent, persona=persona, condition=condition,
            must=["confirm identity", "summarize next step"],
            must_not=["promise a fee waiver", "quote a rate not in policy"],
        )

The must_not list is where most real damage hides. Write it with whoever owns the business rules, not just the engineers.

2. The simulated caller

The caller is another model with a hidden goal and a personality. Its text goes through text to speech, gets noise mixed in for the condition, and reaches your agent over the same transport your real calls use: your WebSocket or SIP endpoint, not a shortcut into the prompt. If you skip the audio path, you skip half the bugs.

CALLER_PROMPT = """You are a customer phoning a lender.
Goal: {intent}. Personality: {persona}.
Stay in character. Reveal details only when asked.
If the agent asks something unclear, say so like a real person would.
End the call when your goal is met or after 12 turns."""

async def run_call(scenario, agent_endpoint):
    caller = LLM(system=CALLER_PROMPT.format(**vars(scenario)))
    call = await connect(agent_endpoint)            # same path as production
    transcript, timings = [], []
    agent_text = await call.listen()                # agent greets first
    for turn in range(12):
        transcript.append(("agent", agent_text))
        reply = await caller.respond(agent_text)
        audio = add_noise(tts(reply), scenario.condition)
        sent_at = now()
        agent_text = await call.say_and_listen(audio)
        timings.append(now() - sent_at)             # caller stops, agent starts
        transcript.append(("caller", reply))
        if caller.done:
            break
    return transcript, timings

3. Scoring: hard checks before the judge

Run cheap, deterministic checks first. They are fast, exact and impossible to argue with.

FORBIDDEN = ["waive", "guarantee", "definitely approved", "no fee"]

def hard_checks(scenario, transcript, timings):
    agent_lines = " ".join(t for who, t in transcript if who == "agent").lower()
    return {
        "no_forbidden_promises": not any(p in agent_lines for p in FORBIDDEN),
        "human_when_asked": scenario.intent != "ask for a human" or "transfer" in agent_lines,
        "p95_response_ms_ok": p95(timings) < 1200,
        "ended_cleanly": len(transcript) < 24,
    }

Then a judge model scores what string matching cannot: did the agent actually solve the caller's problem, did it confirm the right numbers, was it polite under pressure. Ask for a score and a quote from the transcript as evidence, so you can audit the judge.

JUDGE_RUBRIC = """Score the agent from 1 to 5 on each item and quote the line that justifies it:
- goal_met: did the caller get what they called for?
- facts_confirmed: were names, dates and amounts read back correctly?
- recovery: when misunderstood, did the agent recover gracefully?
- tone: calm, clear and respectful throughout?
Return JSON."""

A call passes only if every hard check passes and every judge score is 4 or above.

4. The regression gate

Every prompt edit, model swap or voice change is a deploy. Run the matrix on each pull request and block the merge if the pass rate drops.

# .github/workflows/voice-sim.yml
on: pull_request
jobs:
  simulate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python sim/run.py --scenarios 240 --concurrency 20 --out results.json
      - run: python sim/gate.py results.json --min-pass-rate 0.92 --max-drop 0.02 --critical no_forbidden_promises

Treat no_forbidden_promises as critical: a single failure fails the build, regardless of the overall pass rate.

Close the loop with real calls

Simulated callers are only as creative as you made them. Real callers are more creative. Sample a few percent of production calls, run the same hard checks and judge on their transcripts, and turn every failure into a new scenario. After a month, your matrix reflects what customers actually say, not what you guessed they would say.

What to watch

Metric	Why it matters	A sane starting target
Pass rate on the matrix	Overall health per release	92% or higher
Critical rule failures	Promises, wrong amounts	Zero
p95 time to first response	Feels human or robotic	Under 1.2 s
Talk-over events per call	Interrupting the caller	Under 1
Escaped failures	Bugs found in production first	Falling every week

The agents that survive real customers are not the ones with the cleverest prompt. They are the ones that have already taken ten thousand bad calls in a simulator before taking their first real one.

We build and test voice agents at Axionry, an AI product development studio. See how we approach voice AI development, or estimate what your agent will cost per minute with the voice AI cost calculator.

Stop Spending a Full LLM Call on a Yes or No Question: OpenAI's Decisions API, Explained

Neeraj Sharma — Sun, 25 Oct 2026 21:05:52 +0000

Count the decisions inside any agent you have shipped. Is this ticket billing or technical? Should the agent retry, continue or ask for clarification? Is this image damaged? Is the caller angry enough to escalate?

None of these need a paragraph of reasoning. They need one answer from a short list. Yet most of us send them through the same chat completion endpoint we use for writing emails, pay for the output tokens, parse the JSON, and add a retry for when the model gets creative.

On October 22, OpenAI made the Decisions API generally available, and it is built for exactly this (OpenAI). Here is what it does, the three answer types, and the patterns worth stealing even if you never call it.

What it is

Decisions runs on GPT-6 Luna, but instead of generating text it answers questions with bounded answers. OpenAI says it is up to 10x faster than asking Luna the same thing through the Responses API. Since the beta it has gained prompt caching, native JSON inputs, images inside instructions, up to 400 questions per request, and US and EU inference residency.

The pricing is the interesting part: $0.10 per million input tokens, $0.01 cached, and no output token charges at all. For a router that reads a 600 token ticket, that is about six thousandths of a cent per decision before caching.

Three answer types

You define a question and its allowed answers, pass context as text, images or JSON, and get back one of three result shapes:

Type	Use it for	You get
`predicate`	Is a condition true? ("Does this photo show visible damage?")	A probability from 0 to 1
`choice`	Pick one of your options ("continue", "retry", "ask")	The choice, plus probabilities and confidence
`score`	Rate against an ordered rubric ("damage severity 0 to 4")	A probability-weighted average of the level index

The shape matters more than it looks. A choice with probabilities is something you can threshold. A score that is a weighted average is something you can sort. A free-text answer from a chat model is neither until you parse it.

Pattern 1: the confidence gate

The most useful thing you get back is not the answer. It is how sure the model is. Box shared that in its evaluation the Decisions API picked routes about 5x faster on certificate of insurance documents with the same accuracy as Responses, and sent lower-confidence cases to human reviewers.

That is the pattern: cheap decision first, expensive path only when needed.

def route(ticket: dict) -> str:
    d = decide(  # thin wrapper around the Decisions endpoint
        question={
            "type": "choice",
            "name": "queue",
            "instructions": "Which team should handle this support ticket?",
            "choices": [{"value": v} for v in ["billing", "technical", "account", "sales"]],
        },
        context=ticket,  # native JSON, no prompt templating
    )
    if d.confidence >= 0.85:
        return d.choice                  # fast path, most traffic
    if d.confidence >= 0.55:
        return escalate_to_large_model(ticket, hint=d.choice)  # slow path
    return "human_review"                # the model knows it does not know

Tune the two thresholds on a labelled sample, not by feel. The first one controls cost, the second controls how much lands on people.

Pattern 2: batch the questions you were asking one by one

Up to 400 questions per request means you can stop making serial calls for related checks. Moderating a listing? Ask "contains contact details?", "price looks like a typo?", "photo matches category?" and "language is English?" in one request against one cached context. One round trip instead of four, and the shared context is billed once.

checks = [
    {"type": "predicate", "name": "has_contact_info", "instructions": "Does the listing text include a phone number or email?"},
    {"type": "predicate", "name": "price_suspicious", "instructions": "Is the price implausible for this item category?"},
    {"type": "score", "name": "photo_quality", "instructions": "Rate photo quality.",
     "levels": ["unusable", "poor", "fine", "good", "excellent"]},
]
result = decide_many(checks, context=listing)

Pattern 3: decisions inside a live voice call

This is the part I find most exciting. Decisions now plugs into GPT-Live sessions as a trigger on transcript events. While the caller is still speaking, the app can decide whether to ask a clarifying question, route, or escalate, without pausing the conversation to run a full model turn. OpenAI's demo uses it to pick an avatar's facial expression from the live transcript:

{
  "type": "decisions",
  "event_type": "session.input_transcript.delta",
  "decisions": {
    "questions": [{
      "type": "choice",
      "name": "reaction",
      "instructions": "Choose an appropriate reaction for the avatar based on the user's latest message.",
      "choices": [{ "value": "neutral" }, { "value": "happy" }, { "value": "sad" }, { "value": "concerned" }]
    }]
  }
}

Swap "reaction" for escalate, needs_human or intent, and you have the side channel every voice agent needs: a continuous, cheap read on the conversation that never interrupts it.

Where it does not fit

Open answers. If the answer is a sentence, a plan or code, use a generation model.
Decisions that need tools. Decisions answers from the context you send. If it needs to look something up first, do the lookup, then decide.
Unlabelled guesses. A confidence number is only meaningful once you have checked it against real outcomes. Log every decision with its confidence and the eventual truth, and recalibrate monthly.

The bigger lesson

Most agent cost is not in the clever steps. It is in the dozens of small, boring classifications around them. Pulling those out into a dedicated, structured, thresholded layer is good architecture whichever vendor you use: it makes agents faster, cheaper and easier to debug, because every routing choice now has a name, a confidence and a log line.

We design agent architectures like this at Axionry, an AI product development studio. If you are planning an agent and want to know what it will cost to run, try our AI product cost estimator, or see how we build on our AI product development page.

Your Voice Agent Isn't Slow. It's Silent. Non-Blocking Tools, Explained.

Neeraj Sharma — Sun, 25 Oct 2026 01:58:30 +0000

In a chat UI, a 13 second database query gets a spinner and nobody complains. In a phone call, the same 13 seconds is dead air, and dead air sounds like a dropped call. People hang up.

Google's ADK team published a guide this week that puts a number on it. In their sales report benchmark on gemini-3.8-live, switching one tool from blocking to streaming cut the time to the agent's first words from 14.4 seconds to 1.2 seconds (Google Developers Blog, Oct 19 2026). The query still took 13 seconds. The caller just stopped noticing.

This post walks through why voice agents go silent, the two patterns that fix it, and the production details that bite you once real callers show up.

Why prompting doesn't fix it

The instinct is to add "tell the user you are working on it" to the system prompt. It doesn't work, and the reason is mechanical. A normal tool is a coroutine that returns once:

async def query_sales_database(region: str) -> dict:
    """Runs the quarterly sales report for a region."""
    report = await run_expensive_query(region)  # ~13 seconds
    return {"status": "success", "report": report}

In a live bidirectional session, the model's turn is waiting on that function response. It cannot speak the "one moment" line, because it has not been given control back yet. The fix has to be architectural: the tool must hand control back before the work is done.

Pattern 1: stream it when the wait has a story

If the job has real stages (connect, scan, aggregate, format), make the tool an async generator. In ADK, changing return to yield is enough. The framework marks the declaration as non-blocking and immediately sends the model an acknowledgement, so the agent can talk while the work runs:

import asyncio
from typing import AsyncGenerator

async def query_sales_database(region: str) -> AsyncGenerator[str, None]:
    """Runs the quarterly sales report. Call this ONCE; it pushes updates."""
    for message, delay in [
        ("Connecting to the sales warehouse", 1.5),
        ("Scanning order records", 3.5),
        ("Aggregating revenue by product line", 3.5),
        ("Formatting the report", 1.5),
    ]:
        yield f"PROGRESS: {message}"
        await asyncio.sleep(delay)
    yield f"DONE: {region} revenue is up 12.4% year over year."

Each yield arrives at the model as a partial function response, and the model paraphrases it ("Okay, I'm into the warehouse, pulling the order records now"). The caller hears progress instead of silence. If they ask an unrelated question mid-query, the agent answers it without cancelling the background task.

Pattern 2: background it when the wait is just a wait

Narrating every stage gets annoying when the user has already moved on. If there is nothing worth saying, yield exactly once, at the end:

async def query_sales_database(region: str) -> AsyncGenerator[dict, None]:
    """Runs the quarterly sales report. Call this ONCE; results arrive when ready."""
    yield await run_expensive_query(region)

Because it is still a generator, the model gets an immediate pending signal, says one line ("I'm pulling that together now"), and stays quiet until the result lands.

Deciding when the answer is spoken

Execution shape and delivery timing are separate knobs. ADK exposes the second one as response_scheduling on a FunctionTool:

Mode	What happens when the result arrives
`WHEN_IDLE`	Waits for the current turn to finish, then delivers at the next natural pause
`INTERRUPT`	Cuts into current speech immediately (save this for urgent results)
`SILENT`	Added to context only; the agent uses it if the topic comes up

from google.adk.tools import FunctionTool
from google.genai import types

sales_tool = FunctionTool(query_sales_database)
sales_tool.response_scheduling = types.FunctionResponseScheduling.WHEN_IDLE

In Google's benchmark, a report arriving under WHEN_IDLE waited politely while the model finished a separate 20 second answer, then slotted in. Under INTERRUPT it cut the model off mid sentence. Streaming tools can even choose per update: keep intermediate progress SILENT, deliver the final figure WHEN_IDLE.

The rule of thumb from the guide is worth memorising: if the wait has a story, stream it. If the wait is just a wait, background it.

The production details that bite

1. Duplicate calls. Because a streaming tool spans several turns, a model that hears the user repeat themselves may call it again. A docstring that says "Call this ONCE" helps, but a latch is what actually protects you, especially when the tool does something with side effects like booking a slot:

_inflight: dict[str, asyncio.Task] = {}

async def book_slot(customer_id: str, slot: str) -> AsyncGenerator[str, None]:
    key = f"{customer_id}:{slot}"
    if key in _inflight:
        yield "ALREADY_RUNNING: booking for this slot is in progress"
        return
    task = asyncio.create_task(_do_booking(customer_id, slot))
    _inflight[key] = task
    try:
        yield "PROGRESS: checking the calendar"
        yield f"DONE: {await task}"
    finally:
        _inflight.pop(key, None)

In a multi-instance deployment, move that key to Redis with a TTL, or pass it to the downstream API as an idempotency key.

2. Barge-in does not cancel tools. When the caller talks over the agent, the audio stops and you get an interrupted event, but running tools keep running and their results still arrive. If you want "actually, never mind" to stop the work, give the agent an explicit cancel tool. ADK has a built-in stop_streaming(function_name) that cancels the generator's task and raises CancelledError inside it, so put your connection cleanup in a finally.

3. Narration lags. The model paraphrases updates as they arrive, and narration can trail the tool by several seconds. Don't emit a progress line every 200 ms. Three or four meaningful stages beat twenty tiny ones.

4. Measure the right number. Backend latency did not change in any of this. What changed is time to first audio and the length of the longest silence in the call. Log both per call. If you only watch p95 tool latency, you will never see the problem your callers hang up on.

The same idea outside ADK

None of this is ADK specific. On any realtime stack (OpenAI Realtime compatible endpoints, Pipecat, LiveKit Agents), the pattern is the same: return an immediate acknowledgement to the model, run the work as a task, push results back into the session as they arrive, and decide whether each one should interrupt, wait or stay silent.

A web page that takes ten seconds feels slow. A voice agent that goes quiet for ten seconds feels broken. That gap is the whole reason this pattern exists.

We build voice agents and AI products end to end at Axionry, an AI product development studio. If you are planning one, our voice AI development work starts with exactly this kind of latency budget, and the voice AI cost calculator will tell you what a minute of it costs.

Agent Skills Is Now a Linux Foundation Standard. My 113-Line Loader Cuts Prompt Tokens 56 to 81%

Neeraj Sharma — Fri, 23 Oct 2026 14:46:00 +0000

On 22 October the Agentic AI Foundation, the Linux Foundation group that already hosts MCP and AGENTS.md, took in four new projects at AGNTCon in San Jose. One of them is Anthropic's Agent Skills, the folder format Claude uses to pick up new know-how on demand. The foundation now hosts 10 projects, up from three at launch in December 2025.

That matters for anyone building their own agent. A skill written once can now be loaded by any harness that follows the spec, not only Claude. So I wrote a loader for my own agents, pointed it at five realistic business skills, and measured what progressive disclosure actually saves against the usual approach of pasting every instruction into the system prompt.

Short version: 56 to 81% fewer prompt tokens per task with five skills, and the gap grows to about 9x by 50 skills. Everything below is code I ran, with the output pasted as it came out.

The format in 60 seconds

A skill is a folder with a SKILL.md file. The file starts with YAML frontmatter and continues with Markdown instructions. From the spec at agentskills.io:

name is required: 1 to 64 characters, lowercase letters, digits and hyphens, no leading, trailing or double hyphens, and it must match the folder name.
description is required: 1 to 1024 characters, saying what the skill does and when to use it.
license, compatibility (max 500 characters), metadata (string to string map) and an experimental allowed-tools are optional.
Optional folders: scripts/, references/, assets/.

Loading happens in three levels. Level 1 is the name and description of every skill, about 100 tokens each, always in the system prompt. Level 2 is the full SKILL.md body, loaded when the agent decides the skill is relevant (the spec recommends under 5,000 tokens and 500 lines). Level 3 is any referenced file, read only when a step needs it.

Here is the top of one of my five test skills:

---
name: refund-requests
description: "Decide and draft replies for customer refund, cancellation and chargeback requests. Use when a customer asks for money back, wants to cancel a subscription, or disputes a charge."
metadata:
  owner: support
  version: "1.2"
---
# Refund requests

## Steps
1. Find the order or subscription by the customer's email. If there is more than one match, ask which one.
2. Work out the purchase date, plan, amount paid and how much of the service was used.
3. Apply the rules in references/POLICY.md in order. Stop at the first rule that matches.
4. If the refund is under 200 USD and a rule says approve, issue it with the billing tool and say so.
5. If the refund is 200 USD or more, or no rule matches, do not issue anything. Draft a reply and put the ticket in the "needs human" queue with a one-line reason.
6. For chargebacks, never argue with the customer. Collect the order ID, the date and the last four digits of the card, then hand off to billing.

The policy table lives in references/POLICY.md, so the agent only reads it when a refund actually needs a rule.

The loader: 113 lines

It does four things: validates each skill against the spec, builds the level 1 catalog, serves levels 2 and 3 through two tools, and counts tokens so you can see the difference.

"""Minimal Agent Skills loader: validate, build the catalog, load on demand, measure tokens."""
import json, re, sys
from pathlib import Path
import yaml       # pip install pyyaml
import tiktoken   # pip install tiktoken (token counts are an approximation for non-OpenAI models)

ENC = tiktoken.get_encoding("o200k_base")
NAME_RE = re.compile(r"^[a-z0-9]+(-[a-z0-9]+)*$")  # lowercase, digits, single hyphens, no leading/trailing hyphen

def tokens(text: str) -> int:
    return len(ENC.encode(text))

def parse_skill(skill_md: Path):
    raw = skill_md.read_text()
    m = re.match(r"^---\n(.*?)\n---\n(.*)$", raw, re.S)
    if not m:
        raise ValueError(f"{skill_md}: missing YAML frontmatter")
    return yaml.safe_load(m.group(1)) or {}, m.group(2)

def validate(skill_dir: Path) -> list[str]:
    errors = []
    md = skill_dir / "SKILL.md"
    if not md.exists():
        return [f"{skill_dir.name}: no SKILL.md"]
    try:
        meta, body = parse_skill(md)
    except ValueError as e:
        return [str(e)]
    name, desc = str(meta.get("name", "")), str(meta.get("description", ""))
    if not 1 <= len(name) <= 64 or not NAME_RE.match(name):
        errors.append(f"{skill_dir.name}: bad name {name!r}")
    if name != skill_dir.name:
        errors.append(f"{skill_dir.name}: name {name!r} must match the folder name")
    if not 1 <= len(desc) <= 1024:
        errors.append(f"{skill_dir.name}: description must be 1 to 1024 chars (got {len(desc)})")
    if "compatibility" in meta and not 1 <= len(str(meta["compatibility"])) <= 500:
        errors.append(f"{skill_dir.name}: compatibility must be 1 to 500 chars")
    if "metadata" in meta and not all(isinstance(v, str) for v in (meta["metadata"] or {}).values()):
        errors.append(f"{skill_dir.name}: metadata values must be strings")
    if body.count("\n") > 500:
        errors.append(f"{skill_dir.name}: SKILL.md body over 500 lines, move detail to references/")
    if tokens(body) > 5000:
        errors.append(f"{skill_dir.name}: body is {tokens(body)} tokens, spec recommends under 5000")
    return errors

class SkillStore:
    def __init__(self, root: str):
        self.root = Path(root).resolve()
        self.skills = {}
        for d in sorted(p for p in self.root.iterdir() if p.is_dir()):
            errs = validate(d)
            if errs:
                print("SKIP", *errs, sep="\n  ", file=sys.stderr)
                continue
            meta, body = parse_skill(d / "SKILL.md")
            self.skills[meta["name"]] = {"dir": d, "meta": meta, "body": body}

    def catalog(self) -> str:
        """Level 1: only name + description go into the system prompt."""
        rows = [f'<skill name="{n}">{s["meta"]["description"]}</skill>' for n, s in self.skills.items()]
        return ("<available_skills>\n" + "\n".join(rows) + "\n</available_skills>\n"
                "Call load_skill(name) before doing a task that matches a skill.")

    def load_skill(self, name: str) -> str:
        """Level 2: the full SKILL.md body, loaded when the model asks for it."""
        if name not in self.skills:
            return f"Unknown skill {name!r}. Available: {', '.join(self.skills)}"
        return self.skills[name]["body"]

    def read_skill_file(self, name: str, path: str) -> str:
        """Level 3: a referenced file, resolved inside the skill folder only."""
        base = self.skills[name]["dir"]
        target = (base / path).resolve()
        if not target.is_relative_to(base) or not target.is_file():
            return f"No file {path!r} in skill {name!r}"
        return target.read_text()

    def eager_prompt(self) -> str:
        """What many agents do today: paste every instruction and reference up front."""
        parts = []
        for s in self.skills.values():
            parts.append(s["body"])
            parts += [f.read_text() for f in sorted(s["dir"].rglob("*")) if f.is_file() and f.name != "SKILL.md"]
        return "\n\n".join(parts)

TOOLS = [  # Anthropic-style tool definitions; the same shape maps to OpenAI function tools
    {"name": "load_skill", "description": "Load the full instructions of a skill by name.",
     "input_schema": {"type": "object", "properties": {"name": {"type": "string"}}, "required": ["name"]}},
    {"name": "read_skill_file", "description": "Read a file referenced by a loaded skill, e.g. references/POLICY.md.",
     "input_schema": {"type": "object", "properties": {"name": {"type": "string"}, "path": {"type": "string"}},
                      "required": ["name", "path"]}},
]

if __name__ == "__main__":
    store = SkillStore(sys.argv[1] if len(sys.argv) > 1 else "skills")
    catalog, eager = store.catalog(), store.eager_prompt()
    tool_tokens = tokens(json.dumps(TOOLS))
    print(f"skills loaded: {len(store.skills)}")
    print(f"catalog (level 1): {tokens(catalog)} tokens + {tool_tokens} for the two tool definitions")
    print(f"eager prompt (everything): {tokens(eager)} tokens\n")
    # Three real tasks and what each one actually needs to read
    tasks = {
        "refund for a duplicate charge": [("refund-requests", None), ("refund-requests", "references/POLICY.md")],
        "book a call with a lead": [("meeting-scheduler", None)],
        "enter an AWS invoice": [("invoice-extraction", None), ("invoice-extraction", "references/VENDORS.md")],
    }
    price_in = 2.00  # USD per million input tokens, Claude Sonnet 5.5 list price
    for task, reads in tasks.items():
        loaded = sum(tokens(store.load_skill(n) if p is None else store.read_skill_file(n, p)) for n, p in reads)
        prog = tokens(catalog) + tool_tokens + loaded
        print(f"{task:32s} progressive {prog:5d}  eager {tokens(eager):5d}  saved {1 - prog / tokens(eager):.0%}")
    print("\npath check:", store.read_skill_file("refund-requests", "../invoice-extraction/SKILL.md"))
    print("unknown:", store.load_skill("pricing"))

pip install pyyaml tiktoken, then python3 skills_loader.py skills:

skills loaded: 5
catalog (level 1): 278 tokens + 125 for the two tool definitions
eager prompt (everything): 3020 tokens

refund for a duplicate charge    progressive  1330  eager  3020  saved 56%
book a call with a lead          progressive   564  eager  3020  saved 81%
enter an AWS invoice             progressive  1176  eager  3020  saved 61%

path check: No file '../invoice-extraction/SKILL.md' in skill 'refund-requests'
unknown: Unknown skill 'pricing'. Available: invoice-extraction, lead-qualification, meeting-scheduler, refund-requests, sql-weekly-report

The catalog for five skills is 278 tokens, plus 125 for the two tool definitions. A refund task loads the refund skill and its policy table and lands at 1,330 tokens. Booking a call only needs the scheduler skill: 564 tokens. Pasting everything costs 3,020 tokens on every single call, whatever the task.

The path check matters in practice: a model asked to read references/POLICY.md will sometimes invent a path, and the loader should answer with a clear miss rather than reading whatever file the string points at.

How it scales

Five skills is a toy. Real support or ops agents end up with dozens. I took the per-skill averages from my five and extrapolated:

"""Extrapolate catalog vs eager size as the skill library grows, and show the validator catching bad skills."""
import tempfile
from pathlib import Path
from skills_loader import SkillStore, validate, tokens

store = SkillStore("skills")
n = len(store.skills)
per_skill_catalog = (tokens(store.catalog()) - 30) / n   # ~30 tokens of wrapper text
per_skill_eager = tokens(store.eager_prompt()) / n
skill_content = 620  # mean SKILL.md + reference tokens one task loaded in skills_loader.py: (1330+564+1176)/3 - 403
price = 2.00     # USD per million input tokens, Claude Sonnet 5.5
print(f"per skill: catalog {per_skill_catalog:.0f} tokens, eager {per_skill_eager:.0f} tokens\n")
print(f"{'skills':>6} {'catalog':>8} {'eager':>8} {'eager $/1k calls':>17} {'progressive $/1k calls':>23}")
for k in (5, 20, 50, 100, 300):
    cat = 30 + per_skill_catalog * k
    eager = per_skill_eager * k
    prog = cat + 125 + skill_content  # catalog + tool definitions + one task's skill content
    print(f"{k:6d} {cat:8.0f} {eager:8.0f} {eager*price/1e6*1000:17.2f} {prog*price/1e6*1000:23.2f}")

with tempfile.TemporaryDirectory() as tmp:
    bad = {"Refund_Bot": "---\nname: Refund_Bot\ndescription: refunds\n---\nbody\n",
           "crm--sync": "---\nname: crm--sync\ndescription: ''\n---\nbody\n",
           "pdf-tools": "---\nname: pdf-processing\ndescription: Work with PDFs.\n---\nbody\n"}
    for folder, text in bad.items():
        d = Path(tmp) / folder; d.mkdir(); (d / "SKILL.md").write_text(text)
        print(*validate(d), sep="\n")

per skill: catalog 50 tokens, eager 604 tokens

skills  catalog    eager  eager $/1k calls  progressive $/1k calls
     5      278     3020              6.04                    2.05
    20     1022    12080             24.16                    3.53
    50     2510    30200             60.40                    6.51
   100     4990    60400            120.80                   11.47
   300    14910   181200            362.40                   31.31
Refund_Bot: bad name 'Refund_Bot'
crm--sync: bad name 'crm--sync'
crm--sync: description must be 1 to 1024 chars (got 0)
pdf-tools: name 'pdf-processing' must match the folder name

At 50 skills the eager prompt is about 30,000 tokens on every call. The progressive version is about 3,250, because the catalog grows by about 50 tokens per skill while a task still loads one skill's content. At 300 skills the eager prompt is 181,000 tokens, which is past where most teams would ever send it, and even the catalog reaches about 15,000 tokens. That is the point where you add a search step that picks the 10 most likely skills before the model sees the catalog.

The validator output at the bottom is the other half of the value. Uppercase names, double hyphens, empty descriptions and folder mismatches all get rejected before an agent ever sees them, which saves a confusing debugging session later.

Wiring it into your agent loop

The two tools are plain JSON schemas, so they drop into any tool-calling loop. The dispatch is a few lines:

store = SkillStore("skills")
system = BASE_PROMPT + "\n\n" + store.catalog()

def handle_tool(name, args):
    if name == "load_skill":
        return store.load_skill(args["name"])
    if name == "read_skill_file":
        return store.read_skill_file(args["name"], args["path"])
    return f"unknown tool {name}"

Pass TOOLS alongside your own tools, put the catalog in the system prompt, and append each tool result to the conversation as usual. The model reads the descriptions, calls load_skill when a task matches, and asks for reference files when the instructions point to them.

Honest caveats

Token counts are approximate. I used tiktoken's o200k_base. Claude uses a different tokenizer, and Anthropic's pricing page notes that Claude 4.7 and later models produce about 30% more tokens for the same text. The ratios hold; the absolute numbers move.
Prompt caching narrows the dollar gap. On Claude Sonnet 5.5 a cache read costs 0.10 USD per million tokens against 2.00 for fresh input, so a stable eager prompt that stays cached is about 20 times cheaper than my table shows. It is still a longer context on every call, which costs latency, and every edit to any skill invalidates the cache for all of them.
Each skill load is an extra round trip. One more model turn per task. For voice agents, where every 100 ms counts, preload the one or two skills a call almost always needs.
Descriptions do all the routing. The model only sees name and description until it loads a skill. A vague description such as "Helps with PDFs" means the skill never fires. Put the trigger words your users actually say into it.
Skills are not tools. A skill is instructions and files. If a step needs to touch an API, that is still a tool or an MCP server, and the skill tells the agent when and how to use it.

Where this fits

Most agent projects I see start with one giant system prompt and hit a wall around the tenth workflow, when instructions start contradicting each other and every call carries all of them. Moving to skills early keeps each workflow testable on its own and lets non-engineers edit a Markdown file instead of a prompt buried in code.

At Axionry we build production agents this way: skills for know-how, MCP for tools, evals per workflow, and cost per task on a dashboard from day one. A working agent demo ships in under two weeks, and production AI agents are capped at 5K USD, with exact pricing set once we talk. Have a look at how we build custom AI agents, our notes on MCP in production, or book a call.

Sources: Linux Foundation: AAIF announces Agent Skills, Open Responses, OpenEnv and OpenSandbox; Agent Skills specification; Anthropic Engineering: Equipping agents for the real world with Agent Skills; Anthropic API pricing.

98% of Battle Drones Was Written by Claude: I Audited 334 Commits With 140 Lines of Python

Neeraj Sharma — Thu, 22 Oct 2026 18:43:24 +0000

On 20 October Simon Willison published Battle Drones, a browser RTS in the Command & Conquer mould. He built it over 12 major versions and 321 commits, mostly with Claude Code, and he is candid about it: "I've reviewed almost none of the code." The repo is public, which makes it the best open dataset I have seen for a question every team adopting coding agents should ask: how much of our codebase did the agent write, and what happened to that code afterwards?

So I wrote a small, stdlib-only Python script that answers it from git log, cloned the repo and ran it. Below are the script, the real output and what I think it means for teams.

Why measure this at all

My team is small and agents now write a real share of our commits. "Is the agent productive?" is the wrong question. The useful ones are operational:

Review load. If most changed lines come from an agent, reviewers are the bottleneck. The cost of a merge is reading, not writing.
Churn. Code added and deleted within two weeks is work you paid for twice, and agents can produce a lot of it.
Fix-ups. If fix commits keep landing on files an agent just touched, that is where you need tests, a spec or a narrower prompt.
Hotspots. The files agents touch most are where a guardrail (a lint rule, a CODEOWNERS entry, a contract test) pays back fastest.

None of this needs a vendor dashboard. Agents already leave fingerprints in git.

How it detects agent commits

Claude Code adds a Co-Authored-By: Claude ... <noreply@anthropic.com> trailer, and Claude Code on the web also adds a Claude-Session: line with a session URL. Copilot's coding agent commits as copilot-swe-agent[bot] or a +Copilot@users.noreply.github.com address. Cursor, Codex, Devin and Aider have their own patterns. The script matches one regex per agent against author, committer and the full message. Everything else counts as human. Merge commits and CI bots are skipped.

Then, in one pass over the full patch history (git log -p with zero context lines), oldest commit first, it computes:

Share: commits and changed lines (added plus deleted) per class, ignoring generated paths such as dist/, lockfiles and minified bundles.
Churn: an added line counts as churned if a later commit deletes an identical line in the same file within N days (default 14). This is content matching, not true blame, so treat it as an estimate.
Fix-ups: commits whose subject says fix, bug, revert, regression and so on, split by whether they touch a file that agent or human work changed inside the window.
Hotspots and timeline: top files by agent-changed lines, plus a per-day table.

The script

#!/usr/bin/env python3
"""agentaudit.py: how much of a git repo did coding agents write?

Usage: python3 agentaudit.py [REPO] [--days 14] [--top 8]   (stdlib only)
"""
import argparse, collections, datetime as dt, re, subprocess

# One regex per agent, matched against "author / committer / full message".
AGENTS = {
    "Claude": r"noreply@anthropic\.com|co-authored-by:[^\n]*claude|claude-session:"
              r"|claude\.ai/code/session|generated with \[claude code\]",
    "Codex": r"co-authored-by:[^\n]*codex|chatgpt-codex-connector|codex@openai\.com",
    "Copilot": r"copilot-swe-agent|\+copilot@users\.noreply|co-authored-by:[^\n]*copilot",
    "Cursor": r"cursoragent@cursor\.com|co-authored-by:[^\n]*cursor",
    "Devin": r"devin-ai-integration",
    "Aider": r"\(aider\)|co-authored-by:[^\n]*aider",
}
AGENTS = {k: re.compile(v, re.I | re.M) for k, v in AGENTS.items()}
BOT = re.compile(r"\[bot\]|actions@users\.noreply|github-actions", re.I)  # CI, not agents
FIX = re.compile(r"^(revert|fixup!|hotfix)|\b(fix(es|ed)?|bug|regression|broke|broken|oops)\b", re.I)
TRAILER = re.compile(r"^co-authored-by:\s*([^<\n]+?)\s*<", re.I | re.M)
# Generated files inflate line counts; skip them (override with --skip).
SKIP = r"(^|/)(dist|build|vendor|node_modules|docs/assets)/|\.lock$|-lock\.json$|\.min\.|\.map$"
FMT = "%x00%H%x1f%an <%ae>%x1f%cn <%ce>%x1f%at%x1f%P%x1f%B%x1e"


def who(author, committer, msg):
    text = f"{author}\n{committer}\n{msg}"
    return next((name for name, rx in AGENTS.items() if rx.search(text)), None)


def audit(repo, days, skip_rx):
    skip_rx = re.compile(skip_rx)
    out = subprocess.run(["git", "-C", repo, "log", "--reverse", "-p", "-M", "--unified=0",
                          "--no-color", "--format=" + FMT], capture_output=True,
                         text=True, errors="replace", check=True).stdout
    window = days * 86400
    s = collections.defaultdict(collections.Counter)    # stat name -> {"agent"/"human": n}
    per_agent, models = collections.Counter(), collections.Counter()
    files, day = collections.defaultdict(collections.Counter), collections.defaultdict(collections.Counter)
    added = collections.defaultdict(lambda: collections.defaultdict(collections.deque))
    last_touch = {}                                     # path -> (ts, cls)
    for chunk in out.split("\x00")[1:]:
        head, patch = chunk.split("\x1e", 1)
        sha, author, committer, ts, parents, msg = head.split("\x1f")
        ts = int(ts)
        if len(parents.split()) > 1:
            s["merges"]["all"] += 1
            continue
        agent = who(author, committer, msg)
        if not agent and BOT.search(author):
            s["merges"]["bots"] += 1
            continue
        cls = "agent" if agent else "human"
        if agent:
            per_agent[agent] += 1
            models.update(t for t in TRAILER.findall(msg) if AGENTS[agent].search(f"co-authored-by: {t}"))
        s["commits"][cls] += 1
        if not agent and re.search(r"\(#\d+\)$", msg.split("\n", 1)[0].strip()):
            s["squash"]["human"] += 1                       # squash-merged PR: author unknown
        d = dt.datetime.fromtimestamp(ts, dt.timezone.utc).strftime("%Y-%m-%d")
        day[d][cls] += 1
        path, in_hunk, skip, touched = None, False, False, set()
        for line in patch.split("\n"):
            if line.startswith("diff --git "):
                m = re.match(r"diff --git a/(.*) b/(.*)", line)
                old, path = (m.group(1), m.group(2)) if m else (None, line)
                if old and old != path and old in added:   # follow renames
                    added[path] = added.pop(old)
                skip = bool(skip_rx.search(path))
                if not skip:
                    touched.add(path)
                in_hunk = False
            elif line.startswith("@@"):
                in_hunk = not skip
            elif in_hunk and line[:1] in ("+", "-"):
                content = line[1:].strip()
                s["lines"][cls] += 1
                files[path][cls] += 1
                day[d][cls + "_lines"] += 1
                if line[0] == "+":
                    s["added"][cls] += 1
                    if len(content) > 2:                    # skip "}", "", ");"
                        added[path][content].append((ts, cls))
                elif added[path][content]:
                    t0, c0 = added[path][content].popleft()  # oldest matching add
                    if ts - t0 <= window:
                        s["churn"][c0] += 1
        if FIX.search(msg.split("\n", 1)[0]):
            s["fixes"][cls] += 1
            hit = {last_touch[p][1] for p in touched
                   if p in last_touch and ts - last_touch[p][0] <= window}
            s["fix_after"].update(hit)
        for p in touched:
            last_touch[p] = (ts, cls)
    return s, per_agent, models, files, day


def pct(a, b):
    return f"{100 * a / b:5.1f}%" if b else "  n/a"


def report(repo, days, top, skip_rx):
    s, per_agent, models, files, day = audit(repo, days, skip_rx)
    c, l = s["commits"], s["lines"]
    nc, nl = sum(c.values()), sum(l.values())
    print(f"repo: {repo}   window: {days} days   merges skipped: {s['merges']['all']}   bot commits skipped: {s['merges']['bots']}")
    print(f"commits  agent {c['agent']:>6} {pct(c['agent'], nc)}   human {c['human']:>6} {pct(c['human'], nc)}")
    print(f"         human commits that look like squash-merged PRs (true author unknown): "
          f"{s['squash']['human']}")
    print(f"lines    agent {l['agent']:>6} {pct(l['agent'], nl)}   human {l['human']:>6} {pct(l['human'], nl)}")
    print("agents   " + ", ".join(f"{k} {v}" for k, v in per_agent.most_common()))
    print("models   " + ", ".join(f"{k} {v}" for k, v in models.most_common(5)))
    for cls in ("agent", "human"):
        print(f"churn    {cls}: {s['churn'][cls]} of {s['added'][cls]} added lines deleted within "
              f"{days}d ({pct(s['churn'][cls], s['added'][cls]).strip()})")
    print(f"fixes    fix/revert commits: {sum(s['fixes'].values())} (by agent {s['fixes']['agent']}, "
          f"by human {s['fixes']['human']})")
    for cls in ("agent", "human"):
        print(f"         touching files last changed by {cls} work <= {days}d earlier: "
              f"{s['fix_after'][cls]} ({s['fix_after'][cls] / max(c[cls], 1) * 100:.1f} per 100 {cls} commits)")
    print(f"\ntop {top} files by agent-changed lines:")
    for p, n in sorted(files.items(), key=lambda kv: -kv[1]["agent"])[:top]:
        print(f"  {n['agent']:>6}  {pct(n['agent'], n['agent'] + n['human'])} agent  {p}")
    print("\nday         commits A/H     lines A/H  agent share of lines")
    for d in sorted(day):
        n = day[d]
        share = n["agent_lines"] / max(n["agent_lines"] + n["human_lines"], 1)
        print(f"{d}  {n['agent']:>4}/{n['human']:<4} {n['agent_lines']:>7}/{n['human_lines']:<6} "
              f"{'#' * round(share * 20):<20}")


if __name__ == "__main__":
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("repo", nargs="?", default=".")
    ap.add_argument("--days", type=int, default=14)
    ap.add_argument("--top", type=int, default=8)
    ap.add_argument("--skip", default=SKIP, help="regex of paths to ignore")
    a = ap.parse_args()
    report(a.repo, a.days, a.top, a.skip)

Usage is just this, and it took about 4 seconds on Battle Drones:

python3 agentaudit.py path/to/repo --days 14 --top 8

Battle Drones: the real output

I cloned at commit 39d6f58 (20 Oct). The clone has 334 commits, 48 of them merges, so slightly more than the 321 counted in Simon's post. Dates are UTC.

repo: .   window: 14 days   merges skipped: 48   bot commits skipped: 0
commits  agent    267  93.4%   human     19   6.6%
         human commits that look like squash-merged PRs (true author unknown): 0
lines    agent 170299  98.1%   human   3258   1.9%
agents   Claude 267
models   Claude Opus 5.5 229, Claude Fable 5 28, Claude 1
churn    agent: 12824 of 153755 added lines deleted within 14d (8.3%)
churn    human: 362 of 2875 added lines deleted within 14d (12.6%)
fixes    fix/revert commits: 39 (by agent 39, by human 0)
         touching files last changed by agent work <= 14d earlier: 38 (14.2 per 100 agent commits)
         touching files last changed by human work <= 14d earlier: 0 (0.0 per 100 human commits)

top 8 files by agent-changed lines:
    6362  100.0% agent  src/ui/style.css
    4640  100.0% agent  DESIGN.md
    3874  100.0% agent  src/game/GameScene.ts
    2747  100.0% agent  src/ui/hud.ts
    2680   99.8% agent  SOUND-DUEL.md
    2444  100.0% agent  src/game/three/terrain.ts
    2365  100.0% agent  src/sim/ai/commander.ts
    2339  100.0% agent  public/assets/sprites/game.json

day         commits A/H     lines A/H  agent share of lines
2026-07-17    28/1       6504/2      ####################
2026-09-28     3/0      13953/0      ####################
2026-10-07    23/0      33545/0      ####################
2026-10-08     5/1       9589/581    ################### 
2026-10-09    21/2      10002/473    ################### 
2026-10-10    40/5      11759/683    ################### 
2026-10-11    18/0      10034/0      ####################
2026-10-12    65/0      16785/0      ####################
2026-10-13    10/0       3075/0      ####################
2026-10-14     6/7       3934/1210   ###############     
2026-10-15     9/0       2832/0      ####################
2026-10-16     3/0       2677/0      ####################
2026-10-19    15/0      33969/0      ####################
2026-10-20    20/3      11597/309    ################### 
2026-10-21     1/0         44/0      ####################

What I read from it:

98.1% of changed lines are agent-written. The history backs up "mostly by Claude Code": 267 of 286 non-merge commits carry a Claude signal. 229 name Claude Opus 5.5 in the trailer and 28 name Claude Fable 5, which matches his account of starting v1 on an earlier model in July.

Generated files distort everything. My first run had no skip list. Agent share of lines came out at 95.4%, and five of the top eight "agent files" were build output under docs/assets/ plus package-lock.json. If you are measuring review load, exclude what nobody reviews.

Squash merges nearly hid commits. Simon is the author on 28 non-merge commits. Nine of them are squash-merged agent PRs, with subjects like "Shift+click adds or removes a unit from the selection (#6)". Eight of those nine have no Co-Authored-By trailer at all and are caught only because the message keeps a Claude Code session link. Remove that one signal and the human commit count goes from 19 to 27.

Churn is moderate. 8.3% of agent-added lines were deleted within 14 days. The human figure is higher at 12.6%, but on only 2,875 lines, so I would not read anything into the comparison. For a game rebalanced across 12 versions with overnight playtest runs, I expected more.

Every fix commit came from the agent. There are 39 fix-style commits, all agent-authored, and 38 of them touch files that agent work changed in the previous two weeks. Most subjects read like "Fixes from the final review" or "Sound duel: fix the final review's three major findings". The fix loop here is agents reviewing and repairing agent code. That works for a game. It is the first number I would want visible before letting the same workflow near a billing service.

Hotspots are where you would expect. src/ui/style.css, GameScene.ts, hud.ts and the AI's commander.ts lead. Two of the top five are design docs (DESIGN.md, SOUND-DUEL.md): the agent writes and maintains its own specs, which is a habit worth copying.

A second repo: where attribution breaks

For contrast I ran it on simonw/tools, his collection of single-file HTML tools, at commit befbcec. Top of the output:

repo: .   window: 14 days   merges skipped: 6   bot commits skipped: 483
commits  agent    226  24.5%   human    698  75.5%
         human commits that look like squash-merged PRs (true author unknown): 48
lines    agent  72709   9.1%   human 727389  90.9%
agents   Claude 214, Copilot 12
models   Claude 155, copilot-swe-agent[bot] 9, Claude Fable 5.1 6, Claude Opus 5.5 4, claude[bot] 3
churn    agent: 1420 of 68797 added lines deleted within 14d (2.1%)
churn    human: 4639 of 716876 added lines deleted within 14d (0.6%)
fixes    fix/revert commits: 74 (by agent 40, by human 34)
         touching files last changed by agent work <= 14d earlier: 18 (8.0 per 100 agent commits)
         touching files last changed by human work <= 14d earlier: 42 (6.0 per 100 human commits)

top 6 files by agent-changed lines:
    5244  100.0% agent  grok-mermaid/src/mermaid.rs
    2929  100.0% agent  chirp-pong.html
    2343  100.0% agent  video-compressor.html
    2080  100.0% agent  bluesky-bot-check.html
    1614  100.0% agent  latex.html
    1613  100.0% agent  sqlite-query-explainer.html

Here only 24.5% of commits and 9.1% of lines are attributed to agents (Claude 214, Copilot 12). Agent code churns more than human code (2.1% vs 0.6%) and its files attract slightly more follow-up fix commits (8.0 vs 6.0 per 100 commits). But the "human" bucket is not what it looks like. Of the 698 commits the script calls human, 112 link to a shared Claude conversation and 48 to a ChatGPT one, and 48 look like squash-merged PRs. Code written in a chat window and pasted in by hand looks exactly like human code to git. Git records who committed, not who wrote.

Caveats to take seriously

Trailers are voluntary. They can be disabled in settings, stripped by hand or lost in a rebase. Treat the agent share as a floor.
Squash merges erase authorship unless the squash message keeps the trailer or a session link. If you want this metric, make that a merge rule.
Chat-and-paste is invisible. The script only sees agents that commit.
Content-matched churn misses lines that were edited slightly and counts moved lines as churn.
"Fix" in a subject line is a crude signal. It is fine for trends inside one repo and poor for comparing teams.

Most of these are solved by policy, not code: require the trailer, keep it through squash merges, and run something like this weekly in CI so the trend is visible before it becomes a surprise.

Where this fits

At Axionry we build LLM apps, agents, RAG pipelines and MCP tooling, and every build ships with observability and cost-per-request dashboards. A git audit like this is the development-side twin of those dashboards: production metrics tell you what an agent costs to run, this tells you what agent-written code costs to review and keep. If you are scoping an AI feature and want a rough number first, try our AI product cost estimator.

The script is above, 140 lines, no dependencies. Point it at your own repo and see what your real agent share is.

Sources: Simon Willison, "Battle Drones", simonw/battledrones on GitHub, simonw/tools on GitHub, AI Weekly coverage.

Python requests and Node fetch still trusted eMudhra's compromised CAs on my machine. A 151-line check

Neeraj Sharma — Wed, 21 Oct 2026 09:05:14 +0000

Firefox and Chrome blocked the eight compromised eMudhra intermediate CAs on 17 October, the same day they were reported. Apple followed on the 19th and Microsoft on the 20th.

This morning I pointed requests at one of the public test hosts that serves a certificate from a compromised intermediate. It returned 200. Then I tried fetch() in Node 22. Also 200.

Your browser is fine. Your backend might not be. Here is what happened, why servers are the gap, and the 151-line script I used to check my own machines, with the output pasted as it came out.

What actually broke

The disclosure site lists eight intermediate CA certificates from eMudhra, created in April 2025. Six issue TLS certificates (DV, OV and EV, generations G3B and G3C) and two issue code-signing certificates. Each one was also cross-signed by Sectigo's AAA Certificate Services root, so there are 16 certificates in total covering 8 keys.

The keys were generated incorrectly, so the private keys can be recovered from the public certificates. The reporter proved it by signing a message with each of the eight keys. The site says there are no Certificate Transparency log entries for these intermediates since April 2025, and concludes that browsers which enforce CT could not have been attacked. That protection stops at the browser.

Two trust paths matter:

Path	Anchor	Who still ships the anchor
leaf, compromised intermediate, `emSign Root TLS CA - G3`	`emSign ECC Root CA - G3` (cross-signer)	Included by Apple, Chrome, Microsoft and Mozilla
leaf, compromised intermediate (AAA cross-sign)	`AAA Certificate Services`	Apple and Microsoft; removed by Chrome and Mozilla

Firefox and Chrome distrust the intermediates through OneCRL, CRLite and CRLSets, and Apple and Microsoft pushed their own blocks. Python, Node, Go and most SDKs do none of that. They load a static bundle of roots, they usually skip revocation checks, and none of them enforce CT. If the anchor is in the bundle, the bad intermediate chains to it.

The script

It does four things, all with the standard library plus the openssl CLI:

stores: finds CA bundles on the machine (system paths and certifi) and flags any that contain the four anchors, matched by SHA-256 of the DER certificate.
client: opens a real TLS handshake to two test hosts, once with Python's default context and once with certifi's bundle. A successful handshake means that runtime trusts a compromised intermediate.
hosts: pulls the chain your own endpoints serve and flags any certificate whose public key matches one of the eight compromised keys. It matches on the SubjectPublicKeyInfo hash, so it also catches a reissued certificate that reuses a bad key.
verify: runs an offline openssl verify of the 12 captured chains from the site's chains.zip against any bundle you name.

Exit code 1 means something failed, so it can gate a CI job.

#!/usr/bin/env python3
"""emudhra_check.py: find trust in the 8 compromised eMudhra intermediates.

  python3 emudhra_check.py stores                 # CA bundles on this box
  python3 emudhra_check.py client                 # does this Python trust them?
  python3 emudhra_check.py hosts api.example.com  # what do your endpoints serve?
  python3 emudhra_check.py verify chains/         # offline, with chains.zip

Standard library only (plus the openssl CLI for `hosts` and `verify`).
Hashes come from impacted-intermediates.json and roots.json at
https://emudhra.we-have-a-problem.com/
"""
import glob, hashlib, os, re, socket, ssl, subprocess, sys

# SHA-256 of the SubjectPublicKeyInfo: one per compromised key (16 certs, 8 keys)
BAD_SPKI = {
    "ebed6475c2c2d80fcb199f3fe8dd6e6b3ce60b77eb15d4e42c3dec0c6f5b5105": "EM DV TLS CA - G3B",
    "43bffa19d6ef651451f6841e019931e96f6f2eaeff28ab4b6f1ce4576de2717b": "EM OV TLS CA - G3B",
    "e0254c7fcdc265e2f0f8a428dabb75ab4ae039b4872d62f5ee57f0877d5a8a41": "EM EV TLS CA - G3B",
    "49dd353dc2b919a3770645117d0e60c16115e7d1bc259eefe734b00ea434fe5f": "EM DV TLS CA - G3C",
    "63ecefee431d49523bc1136c4e72d0206a477086caf606cd5cf025686c109cf8": "EM OV TLS CA - G3C",
    "5c0660a2b164910d8ff589c18e14552e9e0ec237dcefe2cb736d41a4811f81a0": "EM EV TLS CA - G3C",
    "ec4ffa1ad0ac88081314b7aaffef6a91f97cd57291bb5ba686c5415f5eb4f6b6": "EM OV CS CA - G3B (code signing)",
    "92c063d67e8f14db71d288e11266ffdaf69cbf115f7bab14f3bc2f68d25e0e30": "EM OV CS CA - G3C (code signing)",
}
# SHA-256 of the whole DER cert: roots that anchor those intermediates
ANCHORS = {
    "86a1ecba089c4a8d3bbe2734c612ba341d813e043cf9e8a862cd5c57a36bbe6b": "emSign ECC Root CA - G3",
    "d7a7a0fb5d7e2731d771e9484ebcdef71d5f0c3e0a2948782bc83ee0ea699ef4": "AAA Certificate Services",
    "7dd78d5f4f13459a83dff9abbba62edbaf6f2d102bf257fd712f4d9f2746ed8d": "emSign Root TLS CA - G3",
    "f2edaaff3d9b701fb4d14c9f285f8c0bad90fc3bcc1fa07d47f4718edaba34a5": "emSign Root CS CA - G3",
}
TEST_HOSTS = ["dv-g3b-emudhra.we-have-a-problem.com",      # via emSign roots
              "dv-g3b-aaa-emudhra.we-have-a-problem.com"]  # via AAA cross-sign
BUNDLES = ["/etc/ssl/certs/ca-certificates.crt", "/etc/pki/tls/certs/ca-bundle.crt",
           "/etc/ssl/cert.pem", "/etc/ssl/ca-bundle.pem",
           "/opt/homebrew/etc/ca-certificates/cert.pem", "/usr/local/etc/openssl@3/cert.pem"]
PEM_RE = re.compile(r"-----BEGIN CERTIFICATE-----.+?-----END CERTIFICATE-----", re.S)


def tlv(buf, i):
    """Return (start_of_value, end_of_value) for the DER element at i."""
    n = buf[i + 1]
    if n < 0x80:
        return i + 2, i + 2 + n
    k = n & 0x7F
    n = int.from_bytes(buf[i + 2:i + 2 + k], "big")
    return i + 2 + k, i + 2 + k + n


def spki_sha256(der):
    """Walk Certificate -> TBSCertificate -> 7th field (SubjectPublicKeyInfo)."""
    s, _ = tlv(der, 0)          # Certificate SEQUENCE
    i, _ = tlv(der, s)          # TBSCertificate SEQUENCE
    if der[i] == 0xA0:          # optional [0] version
        i = tlv(der, i)[1]
    for _ in range(5):          # serial, sigAlg, issuer, validity, subject
        i = tlv(der, i)[1]
    return hashlib.sha256(der[i:tlv(der, i)[1]]).hexdigest()


def certs_in(text):
    return [ssl.PEM_cert_to_DER_cert(p) for p in PEM_RE.findall(text)]


def classify(der):
    full, spki = hashlib.sha256(der).hexdigest(), spki_sha256(der)
    return ANCHORS.get(full), BAD_SPKI.get(spki)


def cmd_stores(extra):
    paths = BUNDLES + extra
    try:
        import certifi
        paths.append(certifi.where())
    except ImportError:
        pass
    paths.append(ssl.get_default_verify_paths().cafile or "")
    risky = 0
    for p in dict.fromkeys(x for x in paths if x and os.path.isfile(x)):
        found = [a for a, b in map(classify, certs_in(open(p, errors="ignore").read())) if a or b]
        names = sorted({f for f in found if f})
        risky += bool(names)
        print(f"{'RISK' if names else 'ok  '}  {p}: {', '.join(names) or 'no eMudhra anchors'}")
    return risky


def cmd_client(_):
    ctxs = {"ssl default": ssl.create_default_context()}
    try:  # requests, httpx, boto3 and most SDKs trust certifi, not the OS store
        import certifi
        ctxs[f"certifi {certifi.__version__}"] = ssl.create_default_context(cafile=certifi.where())
    except ImportError:
        pass
    bad = 0
    for label, ctx in ctxs.items():
        for h in TEST_HOSTS:
            try:
                with socket.create_connection((h, 443), timeout=10) as s:
                    with ctx.wrap_socket(s, server_hostname=h):
                        pass
                bad += 1
                print(f"FAIL  [{label}] {h}: handshake OK, compromised intermediate trusted")
            except ssl.SSLCertVerificationError as e:
                print(f"ok    [{label}] {h}: rejected ({e.verify_message})")
            except OSError as e:
                print(f"skip  [{label}] {h}: {e}")
    return bad


def served_chain(host):
    name, _, port = host.partition(":")
    out = subprocess.run(["openssl", "s_client", "-showcerts", "-connect", f"{name}:{port or 443}",
                          "-servername", name], input=b"", capture_output=True, timeout=20)
    return certs_in(out.stdout.decode(errors="ignore"))


def cmd_hosts(hosts):
    bad = 0
    for h in hosts:
        try:
            chain = served_chain(h)
        except (subprocess.TimeoutExpired, FileNotFoundError) as e:
            print(f"skip  {h}: {e}")
            continue
        hits = [b for _, b in map(classify, chain) if b]
        bad += bool(hits)
        print(f"{'FAIL' if hits else 'ok  '}  {h}: {len(chain)} certs served"
              + (f", compromised: {', '.join(hits)}" if hits else ""))
    return bad


def cmd_verify(args):
    folder, bundles = args[0], args[1:] or [None]
    bad = 0
    for f in sorted(glob.glob(os.path.join(folder, "*.pem"))):
        for b in bundles:
            cmd = ["openssl", "verify"] + (["-CAfile", b] if b else []) + ["-untrusted", f, f]
            r = subprocess.run(cmd, capture_output=True, text=True)
            ok = r.returncode == 0
            bad += ok
            print(f"{'FAIL' if ok else 'ok  '}  {os.path.basename(f)} vs {b or 'openssl default'}"
                  + ("" if not ok else ": chain still verifies"))
    return bad


if __name__ == "__main__":
    cmds = {"stores": cmd_stores, "client": cmd_client, "hosts": cmd_hosts, "verify": cmd_verify}
    if len(sys.argv) < 2 or sys.argv[1] not in cmds:
        sys.exit(__doc__)
    sys.exit(1 if cmds[sys.argv[1]](sys.argv[2:]) else 0)

What it found on my machine

I ran it on an Ubuntu 22.04 VM with Python 3.10, Node 22.23.2 and the ca-certificates package at version 20260601.

$ python3 emudhra_check.py stores
ok    /etc/ssl/certs/ca-certificates.crt: no eMudhra anchors
RISK  /usr/local/lib/python3.10/dist-packages/certifi/cacert.pem: emSign ECC Root CA - G3

$ python3 emudhra_check.py client
ok    [ssl default] dv-g3b-emudhra.we-have-a-problem.com: rejected (unable to get local issuer certificate)
ok    [ssl default] dv-g3b-aaa-emudhra.we-have-a-problem.com: rejected (unable to get local issuer certificate)
FAIL  [certifi 2026.07.22] dv-g3b-emudhra.we-have-a-problem.com: handshake OK, compromised intermediate trusted
ok    [certifi 2026.07.22] dv-g3b-aaa-emudhra.we-have-a-problem.com: rejected (unable to get local issuer certificate)

$ python3 -c "import requests; print(requests.get('https://dv-g3b-emudhra.we-have-a-problem.com/').status_code)"
200

$ python3 emudhra_check.py hosts dv-g3b-emudhra.we-have-a-problem.com ov-g3c-aaa-emudhra.we-have-a-problem.com axionry.com
FAIL  dv-g3b-emudhra.we-have-a-problem.com: 3 certs served, compromised: EM DV TLS CA - G3B
FAIL  ov-g3c-aaa-emudhra.we-have-a-problem.com: 2 certs served, compromised: EM OV TLS CA - G3C
ok    axionry.com: 4 certs served

The OS store was clean. The bundle that requests, httpx, boto3 and most Python SDKs actually use was not. That split is the whole story: an apt upgrade does nothing for a certifi pinned in your lockfile or baked into a Docker image last summer.

Node was the same:

$ node -e "fetch('https://dv-g3b-emudhra.we-have-a-problem.com/').then(r => console.log(r.status))"
200
$ node --use-system-ca -e "fetch('https://dv-g3b-emudhra.we-have-a-problem.com/').then(r => console.log(r.status))"
200
$ node --use-openssl-ca -e "fetch('https://dv-g3b-emudhra.we-have-a-problem.com/').then(r => console.log(r.status))"
TypeError: fetch failed
  [cause]: Error: unable to get local issuer certificate
    code: 'UNABLE_TO_GET_ISSUER_CERT_LOCALLY'

(Stack trace trimmed.)

Node's compiled-in root list includes emSign ECC Root CA - G3. --use-system-ca adds the OS store on top of that list, so it does not help. --use-openssl-ca replaces the bundled list with the OpenSSL store, which on this VM did not contain the anchor.

The fixes I verified

Python. Upgrade certifi. Version 2026.10.19 drops emSign ECC Root CA - G3 from the bundle. I downloaded both versions and ran the offline check against each:

$ python3 emudhra_check.py verify chains/ certifi-2026.07.22/cacert.pem | cut -c1-4 | sort | uniq -c
      6 FAIL
      6 ok
$ python3 emudhra_check.py verify chains/ certifi-2026.10.19/cacert.pem | cut -c1-4 | sort | uniq -c
     12 ok

The 6 failures on the old bundle are exactly the six chains that run through the emSign path. After pip install -U certifi, the client check rejected both test hosts.

Node. The disclosure page lists Node as pending. Until a release lands, NODE_OPTIONS=--use-openssl-ca makes Node use the OS store. Run stores first so you know that store is clean, and test it in staging, because any internal CA you added only to Node's extra certs will need to be in the OS store too.

Go. The page lists Go 1.27.3 and 1.26.10 as patched. Rebuild and redeploy; a static binary keeps whatever it was built with.

Containers. Rebuild images, do not just restart them. Run stores and client inside the image as a CI step.

macOS curl. /usr/bin/curl reads /etc/ssl/cert.pem without revocation checks. The disclosure page has a backup-then-edit workaround for that file.

Honest caveats

The script only knows the 16 certificates and 4 anchors published so far. It is a check for this incident, not a general trust audit.
Removing emSign ECC Root CA - G3 also distrusts legitimate certificates that chain to it. The updated certifi made that call already; if you remove it yourself, check which of your partners' endpoints use emSign certificates first (the hosts command shows the chain).
openssl verify does not check revocation unless you ask it to, so verify measures trust only.
The OS store being clean on my VM says nothing about yours. Run it.

The pattern repeats with every CA incident: browsers get patched in hours, server-side trust stores in months, because nobody owns them. I check exactly this kind of thing when I do technical due diligence on a codebase before a round, and it is part of the operating baseline I set up as a fractional CTO at Axionry. If you would rather someone else run it across your fleet, that is what Axionry is for.

Sources: eMudhra intermediate CA compromise disclosure, with impacted-intermediates.json, roots.json and chains.zip, Mozilla bug 2083449, certifi on PyPI, Node.js CLI docs for --use-openssl-ca and --use-system-ca.

The OVHcloud CLI Swap Left a Fingerprint: A Script to Check Your Go Binaries

Neeraj Sharma — Tue, 20 Oct 2026 13:03:33 +0000

On Monday, October 19, someone used a contributor's GitHub account to replace the download archives on five OVHcloud CLI releases (v0.11.0 to v0.15.0) with a build that, in CEO Octave Klaba's words, was meant to "retrieve secrets from the disk." No tag moved, no workflow ran. The attacker uploaded new files to old releases and edited checksums.txt so the published hashes matched the bad binaries.

What caught it was one customer's CI job that pinned the SHA-256 of ovhcloud-cli_Linux_arm64.tar.gz in their own repo. The job broke, they ran go version -m on the new binary, and the Go toolchain told them exactly what it was. I rebuilt that detection as a script, reproduced the attacker's build locally, and ran it against every Go binary on my machine. Code and real output below.

The timeline, in minutes

From issue #300 and The Hacker News (all UTC, Oct 19):

12:47  first asset replaced on v0.15.0
13:11  v0.13.0 checksums.txt replaced (last upload the reporter checked)
13:50  pinned-hash CI job fails, issue #300 filed
14:21  OVHcloud deletes the swapped files
17:16  clean v0.16.0 published as an immutable release

$ python3 -c "
from datetime import datetime as d
f=lambda s:d.strptime(s,'%H:%M')
print('swap window  ', (f('14:21')-f('12:47')).seconds//60,'min')
print('uploads took ', (f('13:11')-f('12:47')).seconds//60,'min')
print('to detection ', (f('13:50')-f('12:47')).seconds//60,'min')"
swap window   94 min
uploads took  24 min
to detection  63 min

So the documented uploads took 24 minutes, and the bad files were downloadable for 94. Anyone on macOS, Linux x86_64/arm64 or Windows x86_64 who installed through install.sh or a "download latest release" step in that window got the swapped build. I read the repo's install.sh at tag v0.16.0: it downloads the tarball, untars it and installs it, with no checksum step.

The fingerprint

The reporter posted the build info of the swapped binary:

mod    github.com/ovh/ovhcloud-cli  v0.15.1-0.20261013083505-5ead8ef9b43d+dirty
build  -ldflags="-s -w"
build  vcs.revision=5ead8ef9b43d354dfb5caadff803b11afc8c5c58
build  vcs.modified=true

Three things are wrong for a file attached to the v0.15.0 release:

The version is a pseudo-version (an untagged commit from Oct 13), not v0.15.0.
+dirty and vcs.modified=true: it was built with uncommitted changes.
A visible -ldflags line. The project's .goreleaser.yaml builds with -trimpath, and I confirmed in my shell that Go does not record -ldflags in build info when -trimpath is on. A visible -ldflags line means someone ran go build by hand.

Since Go 1.24, go build stamps the main module version from the VCS tag and appends +dirty for uncommitted changes. That metadata survives -s -w stripping, and any binary built from a git checkout carries it unless the build passes -buildvcs=false.

Reproducing the attacker's build

I cloned ovh/ovhcloud-cli at v0.16.0 (commit 6ef70c2) and built four binaries:

clean: tag checkout, go build -trimpath
dirty: same, with one uncommitted line added
swapped: an untagged commit on top of the tag, one uncommitted line, -ldflags="-s -w", no -trimpath. This is the issue #300 recipe.
forged: a local tag named v0.16.0 moved to a different commit, so the binary claims the right version with the wrong code

Then I asked the Go module proxy which commit the real tag points to:

$ curl -s https://proxy.golang.org/github.com/ovh/ovhcloud-cli/@v/v0.16.0.info
{"Version":"v0.16.0","Time":"2026-10-16T14:38:10Z","Origin":{"VCS":"git",
 "URL":"https://github.com/ovh/ovhcloud-cli",
 "Hash":"6ef70c2389cd22c72821cf8d435365d0dcbe2729","Ref":"refs/tags/v0.16.0"}}

The Origin.Hash is the independent reference the script compares against. A GitHub release page can be edited by any account with write access. The proxy record and sum.golang.org ("an auditable checksum database") sit outside GitHub, so a hijacked account cannot edit them.

The script

gobin_audit.py reads build info from every Go binary in your $PATH (or the paths you pass), checks the commit against the proxy, and exits 1 on a FAIL so you can drop it into CI. Read-only, standard library only, needs go on PATH.

def audit(path):
    info = build_info(path)          # parses `go version -m` text output
    if not info:
        return None                  # not a Go binary
    main = info.get("Main") or {}
    mod, ver = main.get("Path", "?"), main.get("Version", "")
    s = {x["Key"]: x["Value"] for x in info.get("Settings", [])}
    rev, dirty = s.get("vcs.revision"), s.get("vcs.modified") == "true"
    dirty = dirty or ver.endswith("+dirty")
    tag = ver.replace("+dirty", "")
    findings = []
    if ver in ("", "(devel)"):
        findings.append(("WARN", "no version stamp, cannot map to a tag"))
    elif PSEUDO.search(tag):
        if dirty:  # the issue #300 signature
            findings.append(("FAIL", f"local dirty build of an untagged commit ({ver})"))
        else:
            findings.append(("WARN", f"untagged commit (pseudo-version {ver})"))
    elif rev:
        if dirty:
            findings.append(("WARN", "built with uncommitted changes (+dirty); can be legit"))
        want = proxy_commit(mod, tag)  # Origin.Hash from proxy.golang.org
        if want and want.startswith("error"):
            findings.append(("INFO", f"proxy lookup failed: {want}"))
        elif want and want != rev:
            findings.append(("FAIL", f"commit {rev[:12]} != tag {tag} commit {want[:12]}"))
        elif want:
            findings.append(("OK", f"commit matches {tag} on {PROXY}"))
    if not rev:
        findings.append(("INFO", "no vcs.revision embedded; compare sha256 with a checksum you trust"))
    ...

The full file (133 lines, including build_info, proxy_commit with the proxy's !lowercase path escaping, and the PATH walker) is at the end of this post. Output against my four builds:

$ python3 gobin_audit.py ovhcloud-clean ovhcloud-dirty ovhcloud-swapped ovhcloud-forged
ovhcloud-clean
  module  github.com/ovh/ovhcloud-cli v0.16.0 (go1.27.1)
  sha256  8364f5294885acdaf7c6510927d7b54910c9d5bae9031f770b885c4e615a2d72
  flags   -trimpath=true
  OK    commit matches v0.16.0 on https://proxy.golang.org
ovhcloud-dirty
  module  github.com/ovh/ovhcloud-cli v0.16.0+dirty (go1.27.1)
  sha256  d124b4a12fa22fdc9b072f1e5682c7a3829df7266e0f483c2e970598cae12a26
  flags   -trimpath=true
  WARN  built with uncommitted changes (+dirty); can be legit
  OK    commit matches v0.16.0 on https://proxy.golang.org
ovhcloud-swapped
  module  github.com/ovh/ovhcloud-cli v0.16.1-0.20261020124243-08d2b7bdd60b+dirty (go1.27.1)
  sha256  93af002eb174dd5c7fc79266fb11fa7e899d07fd42afead346c640a335df5cb9
  flags   -ldflags="-s -w"
  FAIL  local dirty build of an untagged commit (v0.16.1-0.20261020124243-08d2b7bdd60b+dirty)
ovhcloud-forged
  module  github.com/ovh/ovhcloud-cli v0.16.0 (go1.27.1)
  sha256  21c9b4f2e8a625e16299469a5a32dc97eaa95242ad24948ad6945c2638898f70
  flags   -trimpath=true
  FAIL  commit 08d2b7bdd60b != tag v0.16.0 commit 6ef70c2389cd

4 Go binaries checked, worst result: FAIL
exit=1

Why +dirty on a tagged build is only a WARN: The Hacker News reports that the genuine v0.16.0 release itself reports v0.16.0+dirty. My guess is the go mod tidy before-hook in the goreleaser config, but I have not confirmed it. Either way, failing on +dirty alone would have flagged the clean fix.

Then I ran it with no arguments, which walks $PATH:

$ python3 gobin_audit.py | tail -1
15 Go binaries checked, worst result: WARN

6 OK (gh, golangci-lint, containerd, containerd-shim-runc-v2, ctr, runc), 9 WARN, 0 FAIL. The WARNs were all distro-packaged or toolchain binaries (docker, git-lfs, age, go itself) built without VCS stamps. That is normal; for those the script prints the SHA-256 for you to compare against the distro's own checksums.

Honest caveats

Build info is just bytes. A careful attacker builds with -buildvcs=false or fakes the stamp. This catches sloppy swaps like this one (Klaba called the binary's quality "poor"), and it will not catch a careful one.
The -ldflags signal is project-specific. gh v2.89.0 on my machine shows a full -ldflags line and is genuine. Compare against the project's own release config.
It only covers Go. Rust, Node and Python CLIs need their own checks.

The checklist I now run

If you consume CLIs from GitHub releases:

Pin the SHA-256 in your own repo. A checksums.txt in the same release protects you from nothing here, because the attacker replaced it too. This is the check that caught the attack:

$ echo "5c2beb56e6d83a4b426565a1557791ef759316abcc4effb38dd627c90aeb80e0  ovhcloud-cli_Linux_arm64.tar.gz" | sha256sum -c -
ovhcloud-cli_Linux_arm64.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

(Run against a stand-in file; the hash is the original v0.15.0 arm64 archive from issue #300.)

For Go tools, install through the module proxy. go install github.com/ovh/ovhcloud-cli/cmd/ovhcloud@v0.16.0 fetched the module zip whose hash, h1:qu7za9IqNOQQcV3ddmLdDSvKLwmaIAeA7tsAe6lV4kY=, matches the entry I pulled from sum.golang.org/lookup. The issue #300 reporter switched to exactly this, and source builds were not affected.
Run gobin_audit.py on dev laptops and CI images weekly.
For immutable releases, gh release verify-asset <tag> <file> checks GitHub's signed attestation. I could not test it here because my sandbox blocks the GitHub API.

If you ran an affected build: rotate everything readable on that machine, from a different machine. My secrets_on_disk.sh lists what to rotate (paths, sizes and dates only, never contents). Against a throwaway home directory with planted files:

$ env -i HOME=./home PATH=/usr/bin:/bin OVH_CONSUMER_KEY=x STRIPE_SECRET_KEY=y bash secrets_on_disk.sh ./app
PATH                                    BYTES  MODIFIED
./home/.ovh.conf                           80  2026-10-20 12:44:39
./home/.aws/credentials                    54  2026-10-20 12:44:39
./home/.kube/config                        15  2026-10-20 12:44:39
./home/.npmrc                              35  2026-10-20 12:44:39
./home/.ssh/id_ed25519                      2  2026-10-20 12:44:39
env  OVH_CONSUMER_KEY
env  STRIPE_SECRET_KEY
dotenv  ./app/.env.production

If you publish CLIs: turn on release immutability (it only locks releases created after you enable it, so republish old ones), enable private vulnerability reporting (issue #300 was public because there was none), and build with -trimpath plus a version ldflag so a hand-built impostor stands out.

Full script: gobin_audit.py

#!/usr/bin/env python3
"""gobin_audit.py: check Go binaries against the tag they claim to be.

For every Go binary found (in the paths given, or in $PATH), read the build
info the Go toolchain embeds and flag:
  FAIL  binary says it is tag vX but carries a different commit than the one
        the Go module proxy recorded for vX
  FAIL  local dirty build of an untagged commit (the OVHcloud CLI signature)
  WARN  untagged commit, no version stamp, or +dirty on a tagged build
  INFO  no VCS info embedded, so the tag cannot be checked
Needs: go on PATH, network access to proxy.golang.org. Read-only.
"""
import hashlib, json, os, re, stat, subprocess, sys, urllib.request

PROXY = os.environ.get("GOAUDIT_PROXY", "https://proxy.golang.org")
PSEUDO = re.compile(r"-(0\.)?\d{14}-[0-9a-f]{12}(\+incompatible)?$")


def build_info(path):
    # parse text output: Go 1.24 has no -json flag for go version -m
    r = subprocess.run(["go", "version", "-m", path], capture_output=True, text=True)
    lines = r.stdout.splitlines()
    if r.returncode != 0 or not lines or ": go" not in lines[0]:
        return None
    info = {"GoVersion": lines[0].rsplit(": ", 1)[1], "Main": {}, "Settings": []}
    for line in lines[1:]:
        parts = line.strip("\t").split("\t")
        if parts[0] == "mod" and len(parts) >= 3:
            info["Main"] = {"Path": parts[1], "Version": parts[2]}
        elif parts[0] == "build" and "=" in parts[1]:
            k, v = parts[1].split("=", 1)
            info["Settings"].append({"Key": k, "Value": v})
    return info


def escape(mod):  # module path escaping used by the proxy protocol
    return "".join("!" + c.lower() if c.isupper() else c for c in mod)


def proxy_commit(mod, ver):
    url = f"{PROXY}/{escape(mod)}/@v/{ver}.info"
    try:
        with urllib.request.urlopen(url, timeout=15) as resp:
            return json.load(resp).get("Origin", {}).get("Hash")
    except Exception as e:
        return f"error: {e}"


def sha256(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()


def audit(path):
    info = build_info(path)
    if not info:
        return None
    main = info.get("Main") or {}
    mod, ver = main.get("Path", "?"), main.get("Version", "")
    s = {x["Key"]: x["Value"] for x in info.get("Settings", [])}
    rev, dirty = s.get("vcs.revision"), s.get("vcs.modified") == "true"
    dirty = dirty or ver.endswith("+dirty")
    tag = ver.replace("+dirty", "")
    findings = []
    if ver in ("", "(devel)"):
        findings.append(("WARN", "no version stamp, cannot map to a tag"))
    elif PSEUDO.search(tag):
        if dirty:  # the issue #300 signature: local build of an untagged commit
            findings.append(("FAIL", f"local dirty build of an untagged commit ({ver})"))
        else:
            findings.append(("WARN", f"untagged commit (pseudo-version {ver})"))
    elif rev:
        if dirty:
            findings.append(("WARN", "built with uncommitted changes (+dirty); can be legit"))
        want = proxy_commit(mod, tag)
        if want and want.startswith("error"):
            findings.append(("INFO", f"proxy lookup failed: {want}"))
        elif want and want != rev:
            findings.append(("FAIL", f"commit {rev[:12]} != tag {tag} commit {want[:12]}"))
        elif want:
            findings.append(("OK", f"commit matches {tag} on {PROXY}"))
    if not rev:
        findings.append(("INFO", "no vcs.revision embedded; compare sha256 with a checksum you trust"))
    # release pipelines usually build with -trimpath, which hides -ldflags;
    # a visible -ldflags line means a build that skipped -trimpath
    flags = " ".join(f"{k}={v}" for k, v in s.items() if k in ("-trimpath", "-ldflags"))
    return {"path": path, "module": mod, "version": ver, "go": info.get("GoVersion"),
            "sha256": sha256(path), "flags": flags or "(none recorded)", "findings": findings}


def candidates(paths):
    for p in paths:
        if os.path.isfile(p):
            yield p
        elif os.path.isdir(p):
            for name in sorted(os.listdir(p)):
                f = os.path.join(p, name)
                try:
                    st = os.stat(f)
                except OSError:
                    continue
                if stat.S_ISREG(st.st_mode) and st.st_mode & 0o111:
                    yield f


def main():
    paths = sys.argv[1:] or os.environ.get("PATH", "").split(os.pathsep)
    seen, worst, n = set(), "OK", 0
    rank = {"OK": 0, "INFO": 1, "WARN": 2, "FAIL": 3}
    for f in candidates(paths):
        real = os.path.realpath(f)
        if real in seen:
            continue
        seen.add(real)
        r = audit(f)
        if not r:
            continue
        n += 1
        print(f"{r['path']}\n  module  {r['module']} {r['version']} ({r['go']})\n  sha256  {r['sha256']}")
        print(f"  flags   {r['flags']}")
        for level, msg in r["findings"]:
            print(f"  {level:<5} {msg}")
            if rank[level] > rank[worst]:
                worst = level
    print(f"\n{n} Go binaries checked, worst result: {worst}")
    sys.exit(1 if worst == "FAIL" else 0)


if __name__ == "__main__":
    main()

At Axionry, this kind of release-pipeline review is part of our fractional CTO work (from $3,000/mo, 10 to 20 hours a month) and of the technical due diligence we do for investors. More hands-on posts like this on the Axionry blog.

Sources: ovh/ovhcloud-cli issue #300, The Hacker News, BleepingComputer, Go 1.24 release notes, Go checksum database, GitHub immutable releases changelog, GitHub: verify release integrity

Meta Said a Claude Tool Browsed at 4x Human Speed. Here's a 124-Line Python Fix for Impolite AI Agents

Neeraj Sharma — Mon, 19 Oct 2026 12:13:47 +0000

On Saturday, 17 October, Instagram suspended the No Kings Coalition's account for about 90 minutes. On Sunday, Meta's communications VP Andy Stone said a tool linked to Anthropic's Claude had accessed the coalition's profiles, posts and hashtags at roughly four times normal human speed, in short intense bursts that resembled automated bot traffic. Meta called the suspension an error and apologised. It did not say which tool it was, who ran it, or what it was doing.

So we don't know the full story, and I'm not going to pretend we do. But the mechanism Meta described is one I see constantly in agent projects: a model in a loop, a browser or HTTP tool, and nothing between them that understands "slow down". The platform's abuse systems don't care that the traffic came from a helpful assistant. They see bursts, and they act on the account or the IP behind them.

This post is the guardrail layer I now drop into every agent that touches the web. It is 124 lines of standard-library Python. I ran it against a mock server that pushes back, and the real output is below.

What "polite" means in code

Seven things, each small:

Per-host token bucket. A steady pace per site (default 12 requests a minute, burst of 2), so the agent can't fire 40 requests the moment the model decides to "check all profiles".
Random jitter. A human doesn't click every 5.000 seconds. A fixed interval is itself a bot signature.
Backoff on 429 and 503. If the server sends Retry-After, honour it exactly, in either of its two formats (seconds or an HTTP date). If not, back off exponentially with jitter.
robots.txt. Checked with urllib.robotparser before any request, and Crawl-delay lowers the bucket rate if the site asks for it.
An honest User-Agent. Name the bot and give a contact URL and email. Site operators can then reach you instead of blocking a whole range.
A daily action cap. A hard ceiling that a runaway loop cannot get past. Retries count, because the server sees them too.
An append-only action log. One JSON line per action, including the ones that were refused. When something goes wrong you can answer "what did the agent actually do?" in seconds.

The module

"""polite_agent.py: keep an AI agent's HTTP traffic slow, honest and auditable.

Standard library only. Per-host token bucket, jitter, backoff that honours
Retry-After, robots.txt, an identifying User-Agent, a daily cap and a JSONL log.
"""
import email.utils, json, random, threading, time, urllib.error, urllib.request
from datetime import datetime, timezone
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser


class Blocked(Exception):
    """Raised when a request is refused before it is sent (robots or cap)."""


class TokenBucket:
    def __init__(self, rate_per_min, burst):
        self.rate, self.capacity = rate_per_min / 60.0, burst
        self.tokens, self.last = float(burst), time.monotonic()
        self.lock = threading.Lock()

    def take(self):
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
            self.last = now
            wait = 0.0 if self.tokens >= 1 else (1 - self.tokens) / self.rate
            self.tokens -= 1  # may go negative: the debt is paid by sleeping
        if wait:
            time.sleep(wait)


def utc_day():
    return datetime.now(timezone.utc).date().isoformat()


def retry_after_seconds(value):
    """Retry-After is either delta-seconds or an HTTP date."""
    if not value:
        return None
    if value.strip().isdigit():
        return float(value)
    try:
        when = email.utils.parsedate_to_datetime(value)
        return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return None


class PoliteClient:
    def __init__(self, user_agent, log_path="agent_actions.jsonl", rate_per_min=12,
                 burst=2, jitter=(0.5, 2.0), daily_cap=500, max_retries=4,
                 base_backoff=2.0, max_backoff=120.0, timeout=20):
        self.ua, self.log_path, self.timeout = user_agent, log_path, timeout
        self.rate_per_min, self.burst, self.jitter = rate_per_min, burst, jitter
        self.daily_cap, self.max_retries = daily_cap, max_retries
        self.base_backoff, self.max_backoff = base_backoff, max_backoff
        self.buckets, self.robots = {}, {}
        self.today = utc_day()
        self.count = self._count_today()

    def _count_today(self):  # the cap survives restarts because it reads the log
        try:
            with open(self.log_path) as f:
                events = [json.loads(line) for line in f if line.strip()]
            return sum(1 for e in events if e.get("day") == self.today and e.get("sent"))
        except FileNotFoundError:
            return 0

    def _log(self, **event):
        event.update(ts=datetime.now(timezone.utc).isoformat(timespec="seconds"), day=self.today)
        with open(self.log_path, "a") as f:  # append-only, one JSON object per line
            f.write(json.dumps(event) + "\n")

    def _robots_for(self, url):
        parts = urlsplit(url)
        origin = f"{parts.scheme}://{parts.netloc}"
        if origin not in self.robots:
            rp = RobotFileParser()
            try:
                req = urllib.request.Request(origin + "/robots.txt", headers={"User-Agent": self.ua})
                with urllib.request.urlopen(req, timeout=self.timeout) as r:
                    rp.parse(r.read().decode("utf-8", "replace").splitlines())
            except urllib.error.HTTPError as e:
                rp.disallow_all = e.code in (401, 403)  # 404 means no rules
            except OSError:
                rp.disallow_all = True  # unreachable: be conservative
            self.robots[origin] = rp
            delay = rp.crawl_delay(self.ua)
            rate = min(self.rate_per_min, 60.0 / float(delay)) if delay else self.rate_per_min
            self.buckets[parts.netloc] = TokenBucket(rate, self.burst)
        return self.robots[origin]

    def get(self, url):
        if utc_day() != self.today:  # new day, fresh budget
            self.today, self.count = utc_day(), 0
        if not self._robots_for(url).can_fetch(self.ua, url):
            self._log(url=url, sent=False, reason="robots.txt disallow")
            raise Blocked(f"robots.txt disallows {url}")
        host = urlsplit(url).netloc
        for attempt in range(self.max_retries + 1):
            if self.count >= self.daily_cap:
                self._log(url=url, sent=False, reason="daily cap reached")
                raise Blocked(f"daily cap of {self.daily_cap} reached")
            self.buckets[host].take()
            time.sleep(random.uniform(*self.jitter))
            self.count += 1
            req = urllib.request.Request(url, headers={"User-Agent": self.ua})
            try:
                with urllib.request.urlopen(req, timeout=self.timeout) as r:
                    body = r.read()
                    self._log(url=url, sent=True, status=r.status, attempt=attempt)
                    return body
            except urllib.error.HTTPError as e:
                self._log(url=url, sent=True, status=e.code, attempt=attempt)
                if e.code not in (429, 503) or attempt == self.max_retries:
                    raise
                wait = retry_after_seconds(e.headers.get("Retry-After"))
                if wait is None:  # no hint: exponential backoff with jitter
                    wait = min(self.max_backoff, self.base_backoff * 2 ** attempt * random.uniform(0.8, 1.2))
                elif wait > self.max_backoff:  # never retry earlier than asked: give up
                    raise
                print(f"  {e.code} on {url}, backing off {wait:.1f}s")
                time.sleep(wait)

A few decisions worth explaining:

The bucket can go into debt. If two threads grab a token at once, the second one sleeps longer. That keeps the lock short and the pace correct.
If the server asks for a longer wait than max_backoff, the client gives up rather than retrying early. Retrying before Retry-After expires is exactly the behaviour that turns a rate limit into a ban.
The cap is rebuilt from the log on startup. My first version kept the counter in memory, which means a crash loop resets it. Reading the log makes the cap real.
If robots.txt can't be fetched at all, everything is disallowed. A 404 means "no rules", but a network failure means "I don't know", and I'd rather the agent stop.

Testing also caught a bug worth sharing. I originally wrote self.today, self.count = utc_day(), self._count_today() in __init__. Python evaluates the whole right side first, so _count_today() ran before self.today existed and crashed on a restart with an existing log. The first demo run passed because the log was empty. Test the restart path, not just the happy path.

The demo: a server that pushes back

The mock server serves a robots.txt that disallows /private and sets Crawl-delay: 1. Every third request to /profile/* gets a 429 with Retry-After: 2, and the first request to /busy gets a 503 with no hint. I set the client to 120 requests a minute so you can watch the crawl delay pull it down to one a second, and a daily cap of 8 so it runs out quickly.

"""demo.py: run PoliteClient against a local mock server that pushes back."""
import os, threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from polite_agent import Blocked, PoliteClient

HITS = {"profile": 0, "busy": 0}

class Mock(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/robots.txt":
            return self.reply(200, b"User-agent: *\nDisallow: /private\nCrawl-delay: 1\n")
        if self.path.startswith("/profile/"):
            HITS["profile"] += 1
            if HITS["profile"] % 3 == 0:  # every third hit: slow down, try in 2s
                return self.reply(429, b"slow down", {"Retry-After": "2"})
        if self.path == "/busy":
            HITS["busy"] += 1
            if HITS["busy"] == 1:  # 503 with no Retry-After header
                return self.reply(503, b"busy")
        self.reply(200, f"ok {self.path} ua={self.headers['User-Agent']}".encode())

    def reply(self, code, body, headers=None):
        self.send_response(code)
        for k, v in (headers or {}).items():
            self.send_header(k, v)
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

server = ThreadingHTTPServer(("127.0.0.1", 8765), Mock)
threading.Thread(target=server.serve_forever, daemon=True).start()
if os.path.exists("demo_log.jsonl"):
    os.remove("demo_log.jsonl")

agent = PoliteClient(
    user_agent="ResearchBot/1.0 (+https://example.com/bot; bot@example.com)",
    log_path="demo_log.jsonl", rate_per_min=120, burst=1,
    jitter=(0.2, 0.6), daily_cap=8, base_backoff=1.0)

start = time.monotonic()
urls = [f"http://127.0.0.1:8765/profile/{i}" for i in range(1, 5)]
urls += ["http://127.0.0.1:8765/private/inbox", "http://127.0.0.1:8765/busy",
         "http://127.0.0.1:8765/profile/5"]
for url in urls:
    try:
        body = agent.get(url).decode()
        print(f"[{time.monotonic() - start:5.1f}s] 200 {body}")
    except Blocked as e:
        print(f"[{time.monotonic() - start:5.1f}s] BLOCKED {e}")
print(f"requests sent today: {agent.count}/{agent.daily_cap}")
server.shutdown()

Real output from python3 demo.py on Python 3.13:

[  0.6s] 200 ok /profile/1 ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
[  1.4s] 200 ok /profile/2 ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
  429 on http://127.0.0.1:8765/profile/3, backing off 2.0s
[  4.6s] 200 ok /profile/3 ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
[  5.8s] 200 ok /profile/4 ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
[  5.8s] BLOCKED robots.txt disallows http://127.0.0.1:8765/private/inbox
  503 on http://127.0.0.1:8765/busy, backing off 1.1s
[  8.6s] 200 ok /busy ua=ResearchBot/1.0 (+https://example.com/bot; bot@example.com)
  429 on http://127.0.0.1:8765/profile/5, backing off 2.0s
[ 11.5s] BLOCKED daily cap of 8 reached
requests sent today: 8/8

Read it top to bottom and every rule shows up. Requests are spaced about a second apart, not 0.5 seconds, because Crawl-delay overrode my setting. The 429 waited exactly the 2 seconds the server asked for. The 503 with no header got a jittered 1.1 second backoff. The /private URL never left the machine. And the last profile request hit the cap halfway through its retry, because retries are real requests.

The log keeps the refusals too:

{"url": "http://127.0.0.1:8765/profile/3", "sent": true, "status": 429, "attempt": 0, "ts": "2026-10-19T12:09:22+00:00", "day": "2026-10-19"}
{"url": "http://127.0.0.1:8765/private/inbox", "sent": false, "reason": "robots.txt disallow", "ts": "2026-10-19T12:09:26+00:00", "day": "2026-10-19"}
{"url": "http://127.0.0.1:8765/profile/5", "sent": false, "reason": "daily cap reached", "ts": "2026-10-19T12:09:32+00:00", "day": "2026-10-19"}

To wire it into an agent, register agent.get as the only HTTP tool the model can call, and treat Blocked as a normal tool result ("not allowed, stop") rather than an error to retry around.

The honest caveats

This is the part I care about most.

Politeness is not permission. If a platform's terms forbid automated access, a slow and well-labelled bot still breaks them. This toolkit reduces collateral damage. It doesn't make prohibited automation acceptable.
Prefer the official API. Meta, X, LinkedIn and most large platforms publish APIs with explicit quotas. Use them. Rate limits you are told about beat rate limits you guess.
Logged-in scraping is a different risk. robots.txt governs crawlers on public pages. Once your agent uses a real person's session, the platform's account rules apply, and those usually prohibit automated access regardless of pace. The account you lose may belong to your client.
Human pace is not a disguise. Jitter exists to avoid hammering servers, not to evade detection. The honest User-Agent is deliberate. If you're tempted to remove it, that's a sign the task shouldn't be automated.
This covers HTTP, not browsers. If your agent drives a headless browser, every page load fires dozens of sub-requests. Put the bucket in front of navigations and clicks, and count each as an action.
It's single-process. Run five workers and you get five budgets. For a fleet, move the bucket and counter into Redis or your database.

Checklist before your agent touches someone else's site

[ ] Is there an official API? Use it instead.
[ ] Do the terms of service allow automated access for this use?
[ ] Is every outbound request going through one rate-limited client?
[ ] Does the User-Agent name the bot and give a working contact?
[ ] Is robots.txt checked, including Crawl-delay?
[ ] Does it honour Retry-After and back off on 429 and 503?
[ ] Is there a hard daily cap that survives restarts?
[ ] Can you reconstruct every action from the log?
[ ] If it runs inside a user's logged-in session, has the user agreed and do the platform rules allow it?

Rate limiting is only half of making an agent safe to deploy. The other half is controlling what the model is allowed to decide in the first place. At Axionry we build custom AI agents for teams that need this kind of control, and the same thinking runs through our LLM security work: prompt injection defence, tenant isolation, validated output and least-privilege tool access. A polite HTTP client is a good start. An agent that can only reach the tools it needs is better.

Sources: The Next Web, "Meta blames Claude tool for mistaken No Kings Instagram ban" via AI Weekly; Python docs for urllib.robotparser; RFC 9110 on Retry-After.

Stolen LLM keys ran up OpenCode users' bills overnight. A 115-line proxy that caps spend per app

Neeraj Sharma — Sun, 18 Oct 2026 09:59:41 +0000

On 17 October, people using OpenCode woke up to bills they didn't run up. OpenCode confirmed a security breach linked to what it called "a large scale distillation attack" that hit 1.4% of its Console users. Stolen API keys were used to rack up charges. It told customers to revoke keys and change passwords, and said it is working with Google, OpenAI and Anthropic to clean up.

OpenCode hasn't said exactly how the keys were taken, so I won't guess. But the pattern is old and it keeps getting more expensive: a raw provider key sits on a laptop, in a CI secret or in a client app, someone gets it, and they spend your money as fast as the API allows. With agents running models in loops, "as fast as the API allows" is now a big number.

The fix isn't one setting. It's two habits: real keys never leave one server, and every app has a hard daily budget. Here's a small, tested proxy that does both, and a checklist for the rest.

TL;DR

Apps and laptops get an internal key. Only the proxy holds the real provider key.
The proxy prices each response from its token usage and returns 429 once an app passes its daily budget.
It also caps max_tokens per app, refuses unknown keys, and prices unknown models high on purpose.
About 115 lines of standard-library Python. Tested below against a mock API.

The proxy

#!/usr/bin/env python3
"""llm_budget_proxy.py: keep real LLM keys off laptops and cap spend per app.

Apps call this proxy with their own internal key. The proxy holds the real
provider key, forwards the request, reads token usage from the response, prices
it, and refuses calls once an app passes its daily budget.

  REAL_API_KEY=sk-... UPSTREAM=https://api.deepseek.com python3 llm_budget_proxy.py

Config is the APPS and PRICES dicts below. Standard library only. Non-streaming
OpenAI-compatible /v1/chat/completions requests only (see caveats in the post).
"""
import json, os, sys, threading, time, urllib.request, urllib.error
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

UPSTREAM = os.environ.get("UPSTREAM", "https://api.deepseek.com").rstrip("/")
REAL_KEY = os.environ.get("REAL_API_KEY", "")
LEDGER = os.environ.get("LEDGER", "spend.json")

# internal key -> app name, daily budget (USD), largest max_tokens allowed
APPS = {
    "app-support-bot-7f3a": {"name": "support-bot", "daily_usd": 5.00, "max_tokens": 1024},
    "app-dev-laptop-91c2": {"name": "dev-laptop", "daily_usd": 2.00, "max_tokens": 4096},
}
# USD per million tokens: (input, output), keyed by the exact model id you send.
# Example: DeepSeek V4.1 Flash list price, 17 Oct 2026. Add your own models.
PRICES = {"deepseek-v4.1-flash": (0.15, 0.60)}
FALLBACK = (10.0, 50.0)  # unknown model ids are priced high on purpose
ALERT_AT = 0.8  # log a warning when an app passes 80% of its budget

lock = threading.Lock()


def today():
    return datetime.now(timezone.utc).strftime("%Y-%m-%d")


def load():
    try:
        with open(LEDGER) as f:
            return json.load(f)
    except (FileNotFoundError, json.JSONDecodeError):
        return {}


def save(ledger):
    tmp = LEDGER + ".tmp"
    with open(tmp, "w") as f:
        json.dump(ledger, f, indent=1)
    os.replace(tmp, LEDGER)


def cost(model, usage):
    pin, pout = PRICES.get(model, FALLBACK)
    return (usage.get("prompt_tokens", 0) * pin + usage.get("completion_tokens", 0) * pout) / 1e6


class Proxy(BaseHTTPRequestHandler):
    def reply(self, code, obj):
        body = json.dumps(obj).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_POST(self):
        if self.path != "/v1/chat/completions":
            return self.reply(404, {"error": "only /v1/chat/completions is proxied"})
        key = self.headers.get("Authorization", "").removeprefix("Bearer ").strip()
        app = APPS.get(key)
        if not app:
            return self.reply(401, {"error": "unknown app key"})
        req = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0))) or b"{}")
        if req.get("stream"):
            return self.reply(400, {"error": "streaming is not supported by this proxy"})
        req["max_tokens"] = min(int(req.get("max_tokens", app["max_tokens"])), app["max_tokens"])

        with lock:
            spent = load().get(today(), {}).get(app["name"], 0.0)
        if spent >= app["daily_usd"]:
            return self.reply(429, {"error": f"daily budget reached for {app['name']}",
                                    "spent_usd": round(spent, 4), "budget_usd": app["daily_usd"]})

        up = urllib.request.Request(UPSTREAM + self.path, data=json.dumps(req).encode(), method="POST",
                                    headers={"Authorization": f"Bearer {REAL_KEY}",
                                             "Content-Type": "application/json"})
        try:
            with urllib.request.urlopen(up, timeout=120) as r:
                status, data = r.status, json.loads(r.read())
        except urllib.error.HTTPError as e:
            return self.reply(e.code, {"error": "upstream error", "detail": e.read().decode()[:500]})

        usd = cost(req.get("model", ""), data.get("usage", {}))
        with lock:
            ledger = load()
            day = ledger.setdefault(today(), {})
            day[app["name"]] = day.get(app["name"], 0.0) + usd
            save(ledger)
            total = day[app["name"]]
        if total >= ALERT_AT * app["daily_usd"]:
            print(f"[budget] {app['name']} at ${total:.4f} of ${app['daily_usd']:.2f}", file=sys.stderr)
        self.reply(status, data)

    def log_message(self, fmt, *args):
        sys.stderr.write(f"{time.strftime('%H:%M:%S')} {self.address_string()} {fmt % args}\n")


if __name__ == "__main__":
    if not REAL_KEY:
        sys.exit("set REAL_API_KEY")
    port = int(os.environ.get("PORT", "8787"))
    print(f"proxying {UPSTREAM} on 127.0.0.1:{port}", file=sys.stderr)
    ThreadingHTTPServer(("127.0.0.1", port), Proxy).serve_forever()

Run it next to your app:

REAL_API_KEY=sk-your-real-key UPSTREAM=https://api.deepseek.com python3 llm_budget_proxy.py

Then point your SDK at it. With the OpenAI Python SDK, for example:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="app-support-bot-7f3a")

The app's "key" is now worthless outside your network. If it leaks, you delete one line from APPS. The real key never left the server.

Testing it

I ran the proxy against a mock upstream that returns 120,000 input tokens and up to 900 output tokens per call, priced at DeepSeek V4.1 Flash's list price of $0.15 per million input tokens and $0.60 per million output. That's $0.01854 per call, so a $5 budget should last about 270 calls.

1 unknown key:
{"error": "unknown app key"} HTTP401
2 max_tokens clamp:
"seen_max_tokens": 1024
 HTTP200
3 burn the budget:
call 270 -> {"error": "daily budget reached for support-bot", "spent_usd": 5.0058, "budget_usd": 5.0} HTTP429
4 other app:
HTTP200
5 unknown model priced high:
HTTP200
6 streaming:
{"error": "streaming is not supported by this proxy"} HTTP400

The ledger after the run:

{
 "2026-10-18": {
  "support-bot": 5.005799999999964,
  "dev-laptop": 1.26354
 }
}

What each line shows:

A key that isn't in APPS gets nothing.
The app asked for 50,000 output tokens and the upstream saw 1,024, the app's cap.
Call 270 was refused after spend passed $5. The budget check runs before each call, so an app can overshoot by at most one call. Keep max_tokens caps tight and that overshoot stays small.
One app hitting its limit doesn't block the others.
An unknown model ID was priced at the fallback rate ($10 / $50 per million), which is why dev-laptop jumped to $1.26 in one call. If someone swaps in an expensive model, it burns budget fast instead of silently.
Streaming is refused rather than left unmetered.

The proxy also logs a warning once an app passes 80% of its budget:

[budget] support-bot at $4.9687 of $5.00
[budget] support-bot at $4.9873 of $5.00
[budget] support-bot at $5.0058 of $5.00

What this does not do

Be honest with yourself about the gaps before you rely on it:

No streaming. Streaming responses report usage differently per provider. Add it before you route chat UIs through this.
OpenAI-compatible chat only. Many providers offer that format (DeepSeek does), but native APIs with different request shapes need their own handler.
The ledger is a JSON file with a lock, fine for one process. Use Redis or Postgres if you run several instances.
Prices are yours to maintain. The PRICES table has to match your provider's price page. That's why unknown models default to a high price.
It doesn't stop a stolen real key. If the server holding REAL_API_KEY is compromised, the proxy can't help. That's what the provider-side limits below are for.

The checklist for the rest

Set spend limits at the provider too. Most providers let you cap spend or set budget alerts per organization, project or workspace. Use the lowest level they offer. The proxy is your first line, the provider limit is your backstop.
One key per app and per environment. Never share a key between staging, production and laptops. When one leaks, you rotate one thing.
No provider keys in client apps. Mobile and browser apps call your backend, never the model API directly.
Keep keys out of repos and their history. Scan history, not just new commits. I published a script for that yesterday.
Alert on spend velocity, not just totals. A $50 day that normally costs $5 is the signal. A monthly invoice is far too late.
Rotate on a schedule, and immediately when a vendor in your chain reports a breach, like today.

Why this matters more with agents

A chat app makes one model call per message. An agent can make dozens per task, in a loop, unattended. A leaked key used to mean a surprise bill. With agents and fast, cheap models it can mean thousands of calls before anyone wakes up. Budgets per app turn "unlimited" into "annoying".

If you are putting LLMs into a product, this is part of what we harden at Axionry: key isolation, budgets, tenant separation and least-privilege tools for agents. And if you want to know what your AI feature will cost before you build it, our AI product cost estimator does the token math.

Sources: HuggingNews, "OpenCode Confirms Distillation Attack Hit 1.4% of Console Users" (17 October 2026); DeepSeek V4.1-Flash-1017 model card and pricing (17 October 2026). The proxy and test output above are from a real run on Python 3.

GhostAction hit 30,000 repos this month. A 150-line script to check yours, and what to rotate

Neeraj Sharma — Sat, 17 Oct 2026 13:52:08 +0000

On 8 October, in about two hours, someone pushed a workflow file called github_actions_security.yml into more than 30,000 GitHub repositories. The commits came from the real accounts of roughly 1,900 developers, using stolen tokens.

This is GhostAction's third wave. GitGuardian first found it in September 2025, saw it return on 31 August 2026, and then watched the October burst. GitHub started blocking the malicious workflow on 9 October, so the later waves on 10 and 12 October didn't execute. But the earlier ones did, and one detail in GitGuardian's write-up should worry every team: some October victims had also been hit in 2025 and in August. Their secrets were never rotated.

I wrote a small script that checks a repo for the campaign's traces and lists every secret sitting in your git history, which is exactly what the new payload hunts for. It's below, tested, standard library only. First, how the attack works, because the fix depends on it.

How GhostAction works

Get a token. Attackers use GitHub personal access tokens and credentials from infostealer logs and old dumps.
Push a workflow as you. The commit uses the victim's own identity. Names seen so far: github_actions_security.yml ("Add Github Actions Security workflow"), security-check.yml ("Add security check workflow") and, per Socket, security-audit.yml. In 92 cases it modified an existing workflow instead.
Steal what CI can see. The job, named send-secrets, lists the secret names it found in your real workflows and POSTs their values with one curl to a server. In 2026 that server was 193.32.204.199, with batch tags like /?c=bling and /?c=blingnew.
New in October: dig through history. The payload now greps the working tree and git history with 11 secret patterns: AWS key IDs, GitHub and GitLab tokens, AI provider keys and other cloud keys. Version 2 grabbed two lines either side of every AKIA/ASIA match, so it could pair a key ID with its secret. Version 3 raised the history limit to 200,000 lines.

That last step is the important change. A key you deleted from config.js in 2023 is still in history, and the payload now reads history.

GitGuardian's numbers on what got targeted in the 2026 wave, by secret name: 446 SSH and deployment credentials, 218 Azure, 142 container registry, 112 database, 106 AWS, 92 FTP, 80 Google Cloud and Firebase, 66 GitHub tokens, plus Slack, Telegram, Discord, Cloudflare, npm, PyPI and AI provider keys.

And the cleanup has been poor. Only 16% of affected repos had actually cleaned the malicious commits out of public history by 5 October.

The script

ghostaction_audit.py does five things:

Looks at current workflow files for GhostAction file names, the exfiltration endpoints, send-secrets jobs and toJSON(secrets) dumps.
Searches every commit message on every branch for the campaign's commit messages.
Lists every workflow file ever added, with author and commit, so you can spot one you didn't write.
Streams git log -p once and finds secrets in added lines: the same families the payload hunts for.
Flags hardening gaps: no permissions: block, pull_request_target, actions not pinned to a commit SHA.

#!/usr/bin/env python3
"""ghostaction_audit.py: check a git repo for GhostAction traces and leaked secrets.
Usage: python3 ghostaction_audit.py [path-to-repo] [--no-history]
Exit 2 if a GhostAction indicator is found, 1 if secrets are found, else 0."""
import re, subprocess, sys
from collections import defaultdict
from pathlib import Path

IOC_FILES = {"github_actions_security.yml", "security-check.yml", "security-audit.yml"}
IOC_COMMIT_MSGS = [
    "add github actions security workflow",
    "update github actions security workflow",
    "add security check workflow",
    "chore(ci): github security actions",
]
IOC_CONTENT = [
    r"193\.32\.204\.199", r"170\.39\.218\.2\b", r"45-139-104-115\.plesk\.page",
    r"carte-avantage\.com", r"\.oast\.fun", r"/api/workflow/receive",
    r"[?&]inj=", r"/\?c=(bling|blingnew|monami|new|mnsft|security)\b",
    r"^\s*send-secrets\s*:", r"toJSON\(\s*secrets\s*\)",
]
SECRET_PATTERNS = {
    "AWS access key ID": r"\b(?:AKIA|ASIA)[A-Z0-9]{16}\b",
    "GitHub token": r"\b(?:ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9]{36}\b|\bgithub_pat_[A-Za-z0-9_]{60,}\b",
    "GitLab token": r"\bglpat-[A-Za-z0-9_-]{20,}\b",
    "Anthropic key": r"\bsk-ant-[A-Za-z0-9_-]{20,}\b",
    "OpenAI key": r"\bsk-(?:proj-)?[A-Za-z0-9_-]{32,}\b",
    "Google API key": r"\bAIza[0-9A-Za-z_-]{35}\b",
    "Slack token": r"\bxox[abprs]-[A-Za-z0-9-]{10,}\b",
    "npm token": r"\bnpm_[A-Za-z0-9]{36}\b",
    "PyPI token": r"\bpypi-AgE[A-Za-z0-9_-]{50,}\b",
    "Private key": r"-----BEGIN (?:RSA |EC |OPENSSH |DSA )?PRIVATE KEY-----",
}
IOC_RE = [re.compile(p, re.I | re.M) for p in IOC_CONTENT]
SECRET_RE = {k: re.compile(v) for k, v in SECRET_PATTERNS.items()}


def git(repo, *args):
    out = subprocess.run(["git", "-C", str(repo), *args], capture_output=True, text=True, errors="replace")
    if out.returncode != 0:
        sys.exit(f"git {' '.join(args)} failed: {out.stderr.strip()}")
    return out.stdout


def redact(s):
    return s[:6] + "..." + s[-4:] if len(s) > 14 else s[:4] + "..."


def check_tree_workflows(repo, findings):
    wf_dir = repo / ".github" / "workflows"
    if not wf_dir.is_dir():
        return []
    files = sorted(list(wf_dir.glob("*.yml")) + list(wf_dir.glob("*.yaml")))
    for f in files:
        text = f.read_text(errors="replace")
        rel = f.relative_to(repo)
        if f.name in IOC_FILES:
            findings["ioc"].append(f"{rel}: file name matches GhostAction ({f.name})")
        for rx in IOC_RE:
            m = rx.search(text)
            if m:
                findings["ioc"].append(f"{rel}: contains '{m.group(0).strip()}'")
        if not re.search(r"^permissions\s*:", text, re.M):
            findings["harden"].append(f"{rel}: no top-level 'permissions:' block (token gets default scope)")
        if re.search(r"^\s*pull_request_target\s*:", text, re.M) or "pull_request_target" in text.split("jobs:")[0]:
            findings["harden"].append(f"{rel}: uses pull_request_target (runs with secrets on fork PRs)")
        for m in re.finditer(r"uses:\s*([\w.-]+/[\w./-]+)@([\w.-]+)", text):
            action, ref = m.groups()
            if not re.fullmatch(r"[0-9a-f]{40}", ref) and not action.startswith("./"):
                findings["harden"].append(f"{rel}: {action}@{ref} is not pinned to a commit SHA")
    return files


def check_commits(repo, findings):
    log = git(repo, "log", "--all", "--format=%H%x09%an <%ae>%x09%ad%x09%s", "--date=short")
    for line in log.splitlines():
        sha, author, date, subject = (line.split("\t", 3) + ["", "", ""])[:4]
        if any(msg in subject.lower() for msg in IOC_COMMIT_MSGS):
            findings["ioc"].append(f"commit {sha[:10]} {date} by {author}: '{subject}'")
    added = git(repo, "log", "--all", "--diff-filter=A", "--name-only", "--date=short",
                "--format=@@%H%x09%an <%ae>%x09%ad", "--", ".github/workflows")
    cur = None
    for line in added.splitlines():
        if line.startswith("@@"):
            cur = line[2:].split("\t")
        elif line.strip() and cur:
            name = Path(line).name
            tag = "  <-- GhostAction name" if name in IOC_FILES else ""
            findings["new_wf"].append(f"{cur[2]} {line} added by {cur[1]} ({cur[0][:10]}){tag}")
            if name in IOC_FILES:
                findings["ioc"].append(f"history: {line} added in {cur[0][:10]} by {cur[1]}")


def scan_history(repo, findings):
    """Stream `git log -p` once; check added lines for secrets and IOC strings."""
    proc = subprocess.Popen(["git", "-C", str(repo), "log", "--all", "-p", "--no-color",
                             "--format=@@COMMIT %H"], stdout=subprocess.PIPE, text=True, errors="replace")
    sha, path, seen = None, None, set()
    for line in proc.stdout:
        if line.startswith("@@COMMIT "):
            sha = line.split()[1][:10]
        elif line.startswith("+++ b/"):
            path = line[6:].rstrip("\n")
        elif line.startswith("+") and not line.startswith("+++"):
            body = line[1:]
            for kind, rx in SECRET_RE.items():
                for m in rx.finditer(body):
                    key = (kind, m.group(0))
                    if key in seen:
                        continue
                    seen.add(key)
                    findings["secrets"][kind].append(f"{redact(m.group(0))}  first seen in {path} @ {sha}")
            if path and path.startswith(".github/workflows/"):
                for rx in IOC_RE:
                    m = rx.search(body)
                    if m:
                        findings["ioc"].append(f"history: {path} @ {sha} contained '{m.group(0).strip()}'")
    proc.wait()


def main():
    args = [a for a in sys.argv[1:] if not a.startswith("--")]
    repo = Path(args[0] if args else ".").resolve()
    if not (repo / ".git").exists():
        sys.exit(f"{repo} is not a git repository")
    findings = {"ioc": [], "harden": [], "new_wf": [], "secrets": defaultdict(list)}
    files = check_tree_workflows(repo, findings)
    check_commits(repo, findings)
    if "--no-history" not in sys.argv:
        scan_history(repo, findings)
    findings["ioc"] = list(dict.fromkeys(findings["ioc"]))

    print(f"Repo: {repo}\nWorkflow files now: {len(files)}\n")
    print("== 1. GhostAction indicators ==")
    print("\n".join("  !! " + x for x in findings["ioc"]) or "  none found")
    print("\n== 2. Workflow files ever added (review anything you don't recognise) ==")
    print("\n".join("  " + x for x in findings["new_wf"][:25]) or "  none")
    print("\n== 3. Secrets found in git history (rotate ALL of these, deleting the file is not enough) ==")
    if findings["secrets"]:
        for kind, hits in findings["secrets"].items():
            print(f"  {kind}: {len(hits)}")
            for h in hits[:10]:
                print("     " + h)
    else:
        print("  none found by these patterns (not proof there are none)")
    print("\n== 4. Hardening gaps ==")
    print("\n".join("  - " + x for x in findings["harden"]) or "  none")
    if findings["ioc"]:
        print("\nACTION: revoke the GitHub token or session that pushed the workflow first, "
              "then rotate every secret above and in repo settings.")
        sys.exit(2)
    sys.exit(1 if findings["secrets"] else 0)


if __name__ == "__main__":
    main()

Run it from anywhere:

python3 ghostaction_audit.py ~/code/my-repo

To check every repo in a folder:

for d in ~/code/*/; do python3 ghostaction_audit.py "$d" > /dev/null 2>&1; echo "$? $d"; done | sort -r

Exit code 2 means a GhostAction indicator, 1 means secrets in history, 0 means clean.

Testing it

I built a test repo that replays the attack: a normal ci.yml, a fake AWS key and a fake GitHub token committed and then deleted, and a GhostAction-style workflow committed and then removed again (what a hurried cleanup looks like).

== 1. GhostAction indicators ==
  !! commit 23addef113 2026-10-17 by Dev <dev@example.com>: 'Add Github Actions Security workflow'
  !! history: .github/workflows/github_actions_security.yml added in 23addef113 by Dev <dev@example.com>
  !! history: .github/workflows/github_actions_security.yml @ 23addef113 contained 'send-secrets:'
  !! history: .github/workflows/github_actions_security.yml @ 23addef113 contained '193.32.204.199'
  !! history: .github/workflows/github_actions_security.yml @ 23addef113 contained '/?c=bling'

== 3. Secrets found in git history (rotate ALL of these, deleting the file is not enough) ==
  AWS access key ID: 1
     AKIAQ3...KE12  first seen in config.js @ 30d823985c
  GitHub token: 1
     ghp_a1...a1B2  first seen in config.js @ 30d823985c

== 4. Hardening gaps ==
  - .github/workflows/ci.yml: no top-level 'permissions:' block (token gets default scope)
  - .github/workflows/ci.yml: uses pull_request_target (runs with secrets on fork PRs)
  - .github/workflows/ci.yml: actions/checkout@v4 is not pinned to a commit SHA

Both "deleted" keys show up, and so does the workflow that was deleted from the current tree. Exit code 2.

Then I ran it on a real project, a full clone of psf/requests with 6,736 commits. It took 3.2 seconds. No GhostAction traces, and no hardening gaps: every workflow there has a permissions: block and pins actions to a full SHA, which is a good model to copy. It found one "private key", a test certificate under tests/certs/. That's a false positive you judge by path, and a reminder that the script tells you where to look, not what to panic about.

If you find something: the order matters

GitGuardian's advice, and my notes on it:

Revoke the GitHub credential that pushed the workflow first. Look at the commit's author, then check that user's personal access tokens, OAuth apps and active sessions. If you rotate your AWS keys but the attacker still holds the GitHub token, they push the workflow again next week. That's exactly what happened to the repeat victims.
Rotate every secret the workflow could see. That means every secret in repo and org settings that your workflows reference, not only the ones you think were used.
Rotate every secret in section 3 of the output. History is now part of the attack surface. Removing a file does nothing for a key that's still in history.
Then clean history if the repo is public, with git filter-repo, and tell people to re-clone. Do this last. Cleaning history doesn't un-leak a key.
Check the Actions tab. Look for runs of workflows you don't recognise around 8 and 9 October, or around 31 August to 30 September for the earlier wave.

Hardening that would have stopped most of this

Fine-grained tokens with expiry, scoped to the repos they need. A classic PAT with repo and workflow scope that never expires is the master key this campaign runs on.
Require approval for workflow changes. Protect the default branch and add a CODEOWNERS rule for .github/workflows/, so a new workflow can't run without review.
Set permissions: at the top of every workflow, starting from contents: read, and widen only per job.
Use OIDC for cloud access instead of long-lived AWS or Azure keys in secrets. There's nothing static to steal.
Pin third-party actions to a commit SHA. Tags can be moved; SHAs can't.
Run secret scanning on history, not just new commits. The attacker does.

The bigger lesson

GhostAction isn't clever. It uses old tokens, a believable file name and one curl. It works because tokens don't expire, secrets don't get rotated and history never forgets. The October payload just made the last part explicit.

If you are raising money or selling a company, this is also the kind of thing an investor's technical due diligence will look for: long-lived keys in history, unpinned actions, CI with write access everywhere. We run those reviews at Axionry, and fix what they find. If you want a second pair of eyes on your CI before someone else's script finds it, that's what we do.

Sources: GitGuardian, "GhostAction returns" (updated 9 and 13 October 2026) and the original GhostAction report (September 2025); The Hacker News, 15 October 2026; Socket's 8 October report as cited by GitGuardian. Script tested on Python 3 with git 2.x; test output above is from a real run.

Your traffic spike might be one AI newsletter bot: find it, verify crawlers, rate-limit the rest

Neeraj Sharma — Wed, 14 Oct 2026 13:14:30 +0000

Sammy Straus posted a side project to Hacker News, got about a hundred upvotes, and watched the site get slower for days. The culprit was not GPTBot or ClaudeBot. Nearly half of all traffic came from one client, a service that turns website changes into an AI newsletter, hitting the site many times a minute.

I expect a lot of small sites to hit the same thing, and his write-up is on the Hacker News front page today.

The fix they landed on is the right one: find out who is actually calling you, block the abusers, and rate limit the rest. Here are three tested pieces: a log analyzer that also catches fake "GPTBot" traffic, a robots.txt that slows crawlers without hiding you from AI search, and a Next.js middleware that returns 429s to anyone over budget.

TL;DR

The biggest offender often does not call itself a bot. Rank clients by requests per minute, not by user agent.
Check claimed crawlers against their published IP ranges. Anyone can send GPTBot in a header.
Blocking AI search bots can make you invisible in ChatGPT, Claude and Perplexity answers. Rate limit training crawlers instead of blocking everything.
An in-memory rate limiter in middleware is a good first line. It is per instance, so put a WAF rule in front when traffic is real.

Step 1: find out who is really calling

This script reads a standard nginx or Apache access log, sorts every request into a category, and verifies the big crawlers against the IP lists OpenAI, Anthropic and Google publish. It uses only the Python standard library.

#!/usr/bin/env python3
"""Who is really hitting your site? Classify an access log by bot type,
check that "GPTBot", "ClaudeBot" and "Googlebot" come from their published IP
ranges, and flag single clients that hammer you.

Usage: python3 botlog.py access.log   (nginx/Apache "combined" format)
"""
import collections, ipaddress, json, re, sys, urllib.request

LINE = re.compile(r'(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+)[^"]*" '
                  r'(?P<status>\d{3}) (?P<bytes>\d+|-) "[^"]*" "(?P<ua>[^"]*)"')

# user-agent token -> (category, published IP list or None)
BOTS = {
    "GPTBot":           ("AI training", "https://openai.com/gptbot.json"),
    "ClaudeBot":        ("AI training", "https://claude.com/crawling/bots.json"),
    "CCBot":            ("AI training", None),
    "Bytespider":       ("AI training", None),
    "OAI-SearchBot":    ("AI search",   "https://openai.com/searchbot.json"),
    "Claude-SearchBot": ("AI search",   "https://claude.com/crawling/bots.json"),
    "PerplexityBot":    ("AI search",   None),
    "ChatGPT-User":     ("AI user fetch", "https://openai.com/chatgpt-user.json"),
    "Claude-User":      ("AI user fetch", "https://claude.com/crawling/bots.json"),
    "Googlebot":        ("Search engine", "https://developers.google.com/static/search/apis/ipranges/googlebot.json"),
    "bingbot":          ("Search engine", None),
}
SCANNER_PATHS = re.compile(r"wp-(admin|login|content|includes)|phpinfo|\.env|\.git/|xmlrpc|/cgi-bin/", re.I)

_ranges = {}
def ranges(url):
    if url not in _ranges:
        try:
            with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "botlog"}), timeout=15) as r:
                data = json.load(r)
            _ranges[url] = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
                            for p in data["prefixes"]]
        except Exception as e:
            print(f"warning: could not load {url}: {e}", file=sys.stderr)
            _ranges[url] = None
    return _ranges[url]

def classify(ip, ua, path):
    if SCANNER_PATHS.search(path):
        return "Scanner", None
    for token, (cat, url) in BOTS.items():
        if token.lower() in ua.lower():
            if url and (nets := ranges(url)) is not None:
                addr = ipaddress.ip_address(ip)
                if not any(addr in n for n in nets):
                    return "Spoofed bot", token
            return cat, token
    if re.search(r"bot|crawler|spider|scrapy|python-requests|curl|go-http|headless", ua, re.I):
        return "Other bot", None
    return "Human or unknown", None

reqs, size = collections.Counter(), collections.Counter()
per_client, per_minute, spoofed = collections.Counter(), collections.Counter(), collections.Counter()
total = 0
for raw in open(sys.argv[1], errors="replace"):
    m = LINE.match(raw)
    if not m:
        continue
    total += 1
    cat, token = classify(m["ip"], m["ua"], m["path"])
    b = 0 if m["bytes"] == "-" else int(m["bytes"])
    reqs[cat] += 1
    size[cat] += b
    client = f'{m["ip"]}  {m["ua"][:60]}'
    per_client[client] += 1
    per_minute[(client, m["ts"][:17])] += 1          # dd/Mon/yyyy:HH:MM
    if cat == "Spoofed bot":
        spoofed[(token, m["ip"])] += 1

print(f"{total} requests\n")
print(f"{'category':18} {'requests':>9} {'share':>7} {'MB':>8}")
for cat, n in reqs.most_common():
    print(f"{cat:18} {n:9} {n/total:7.1%} {size[cat]/1e6:8.1f}")

print("\nTop clients")
for client, n in per_client.most_common(5):
    peak = max(v for (c, _), v in per_minute.items() if c == client)
    print(f"{n:7} req  peak {peak:4}/min  {client}")

if spoofed:
    print("\nClaims to be a known bot but is outside its published IP ranges")
    for (token, ip), n in spoofed.most_common(10):
        print(f"{n:7} req  {token:12} {ip}")

I generated a test log of 6,330 requests that mirrors the pattern in the post: real GPTBot, ClaudeBot and Googlebot traffic from addresses inside their published ranges, a fake GPTBot from an address outside them, a vulnerability scanner probing /wp-content/phpinfo.php, a change-monitoring service polling an API endpoint, and normal browsers. The output:

6330 requests

category            requests   share       MB
Human or unknown        4000   63.2%     59.7
AI training             1600   25.3%     51.5
Spoofed bot              300    4.7%      9.8
Search engine            250    3.9%      8.1
Scanner                  180    2.8%      0.0

Top clients
   2600 req  peak   13/min  3.91.20.77  SiteWatch/2.3 (+https://example-monitor.io; AI change digest
    900 req  peak    6/min  20.171.206.15  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatibl
    700 req  peak    6/min  216.73.216.40  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatibl
    300 req  peak    3/min  45.83.12.9  Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gpt
    250 req  peak    6/min  66.249.66.1  Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) Ap

Claims to be a known bot but is outside its published IP ranges
    300 req  GPTBot       45.83.12.9

Two things to notice. The heaviest client, with 2,600 requests, lands in "Human or unknown" because its user agent never says "bot". Only the top clients list exposes it. And 300 requests claim to be GPTBot but come from an address that is not in OpenAI's published list, so they are not OpenAI at all.

Step 2: tell crawlers what you want in robots.txt

Each AI company runs separate bots for training, for search and for fetching a page when a user asks. They are independent, and blocking the wrong one costs you visibility:

OpenAI: GPTBot is for training, OAI-SearchBot is for ChatGPT search, and ChatGPT-User fetches pages on a user's behalf. OpenAI says robots.txt may not apply to ChatGPT-User, because a person started the request.
Anthropic: ClaudeBot collects training data, Claude-SearchBot indexes for search, and Claude-User fetches for users. Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay directive.

Here is the version that keeps search bots and keeps crawlers out of your API:

User-agent: ClaudeBot
Crawl-delay: 5
Disallow: /api/

User-agent: GPTBot
Disallow: /api/

User-agent: *
Disallow: /api/

A crawler obeys only the most specific group that names it, so ClaudeBot and GPTBot ignore the * group. That is why each named group repeats Disallow: /api/. Crawl-delay is a hint, not a guarantee: Google does not support it, and OpenAI does not document it, so GPTBot is slowed by the limiter in step 3. Anything that ignores robots.txt needs the next step.

Step 3: rate limit in Next.js middleware

A token bucket per client, with different budgets for search engines, AI search, AI training crawlers and unknown bots:

// lib/bot-limit.ts: a token bucket per client, tuned per kind of visitor.
export type Rule = { perMinute: number; burst: number };

export const RULES: Record<string, Rule> = {
  search: { perMinute: 120, burst: 60 },   // Googlebot, bingbot: never starve these
  aiSearch: { perMinute: 60, burst: 30 },  // OAI-SearchBot, Claude-SearchBot, PerplexityBot
  aiTraining: { perMinute: 20, burst: 10 },// GPTBot, ClaudeBot, CCBot, Bytespider
  unknownBot: { perMinute: 10, burst: 5 }, // anything else that says bot, curl, python...
  default: { perMinute: 120, burst: 60 },  // browsers
};

export function kindOf(ua: string): keyof typeof RULES {
  const u = ua.toLowerCase();
  if (/googlebot|bingbot/.test(u)) return "search";
  if (/oai-searchbot|claude-searchbot|perplexitybot/.test(u)) return "aiSearch";
  if (/gptbot|claudebot|ccbot|bytespider/.test(u)) return "aiTraining";
  if (/bot|crawler|spider|scrapy|python-requests|curl|go-http|headless/.test(u)) return "unknownBot";
  return "default";
}

const buckets = new Map<string, { tokens: number; at: number }>();

/** Returns 0 if allowed, or the seconds to wait before retrying. */
export function check(key: string, rule: Rule, now = Date.now()): number {
  const b = buckets.get(key) ?? { tokens: rule.burst, at: now };
  const refill = ((now - b.at) / 60_000) * rule.perMinute;
  b.tokens = Math.min(rule.burst, b.tokens + refill);
  b.at = now;
  if (b.tokens >= 1) {
    b.tokens -= 1;
    buckets.set(key, b);
    return 0;
  }
  buckets.set(key, b);
  return Math.ceil(((1 - b.tokens) / rule.perMinute) * 60);
}

And the middleware that uses it, which also drops scanner probes before they reach your app:

// middleware.ts (Next.js): rate limit by visitor kind, drop scanner probes early.
import { NextResponse, type NextRequest } from "next/server";
import { RULES, check, kindOf } from "@/lib/bot-limit";

const SCANNER = /wp-(admin|login|content|includes)|phpinfo|\.env|\.git\/|xmlrpc|\/cgi-bin\//i;

export function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl;
  if (SCANNER.test(pathname)) return new NextResponse(null, { status: 404 });

  const ua = req.headers.get("user-agent") ?? "";
  const ip = req.headers.get("x-forwarded-for")?.split(",")[0].trim() ?? "unknown";
  const kind = kindOf(ua);
  const wait = check(`${kind}:${ip}`, RULES[kind]);
  if (wait > 0) {
    return new NextResponse("Too many requests", {
      status: 429,
      headers: { "Retry-After": String(wait) },
    });
  }
  return NextResponse.next();
}

// Skip static files so images and JS never count against the limit.
export const config = { matcher: ["/((?!_next/static|_next/image|favicon.ico|.*\\.(?:png|jpg|webp|svg|css|js)$).*)"] };

I built this into a fresh Next.js 15 app, ran next start, and sent 14 requests in a row as each visitor:

# Scanner probe
curl -o /dev/null -w "%{http_code}" localhost:3000/wp-content/phpinfo.php
# 404

# 14 requests as GPTBot
200 200 200 200 200 200 200 200 200 200 429 429 429 429
# HTTP/1.1 429 Too Many Requests
# retry-after: 3

# 14 requests as Googlebot
200 200 200 200 200 200 200 200 200 200 200 200 200 200

GPTBot gets its burst of 10, then a 429 with a Retry-After header that well-behaved crawlers honor. Googlebot never notices the limiter.

What this does not do

The buckets live in memory, per server instance. On serverless or several replicas, each instance keeps its own count. For real enforcement, add a rate-limiting rule at your CDN or WAF, or move the counter to Redis.
x-forwarded-for can be forged if your app is reachable without a proxy in front. Read the client IP from the header your platform guarantees.
Missing IPs share one bucket. Without a proxy setting x-forwarded-for, every visitor falls into the "unknown" bucket. In production, skip limiting when no IP is found. The Map is also never pruned, so add a periodic cleanup on long-running servers.
On Next.js 16 the file is proxy.ts and the export is named proxy.
The user-agent check is cheap, not secure. Combine it with the IP verification from step 1 before you trust anything that claims to be a search engine.

The decision I would make for most sites

Keep search and AI search bots allowed, because being cited in AI answers is now a real source of visitors. Slow the training crawlers. Find the anonymous heavy hitters in your logs, block them by IP or behavior, and put every unknown bot on a small budget. Run the log script weekly, because the worst client next month will not be the one you blocked this month.

Axionry's own site allows AI crawlers on purpose, so this is the toolkit I keep ready for the day one of them, or something pretending to be one, gets greedy. Logging and rate limits like these belong in the SaaS MVP tech stack from day one. If bandwidth from bots is showing up on your bill, it is also one of the first things I check in a cloud cost review.

Sources: I posted to HN, and then my website was flooded with AI bot traffic, Hacker News discussion, OpenAI crawlers and IP ranges, Anthropic on ClaudeBot, Claude-SearchBot and Claude-User, Googlebot IP ranges. Scripts tested on 14 October 2026 against a generated log and a Next.js 15.5 production build.

Your AI coding CLI may still be running after you close it: I reproduced the Codex daemon and wrote an audit

Neeraj Sharma — Tue, 13 Oct 2026 13:02:10 +0000

One of the top Hacker News threads today is a GitHub issue asking OpenAI to make the Codex CLI's persistent, self-updating background daemon opt-in. The report: since version 0.157.0, starting codex interactively also starts a backend that keeps running after you close every terminal, copies about 374 MB of binaries into your home directory and updates itself every hour.

I wanted to see it for myself before writing about it, so I installed Codex CLI 0.157.0 in a clean Linux container with an empty home directory and started the daemon.

What actually runs

npm i -g @openai/codex@0.157.0
codex app-server daemon start

Installing daemon from CLI version 0.157.0 into ~/.codex/packages/app-server-daemon...
{"status":"started","backend":"pid","pid":702, ...}

Then the process list:

  PID  PPID  COMMAND
  702     1  ~/.codex/packages/app-server-daemon/releases/0.157.0-.../bin/codex app-server --listen unix:// --managed-daemon
  767     1  ~/.codex/packages/app-server-daemon/releases/0.157.0-.../bin/codex app-server daemon pid-update-loop

Both processes have parent PID 1. No terminal owns them, so closing your shell does not stop them. The second one is the updater. According to the issue, it checks for a new release five minutes after start and then every hour, downloads the whole platform package into a new releases folder and restarts the backend, which can interrupt a turn that is still running. I did not wait an hour to watch that happen.

du -sh ~/.codex/packages
# 374M

That matches the issue exactly: 374 MB, a second copy of the CLI that runs from your home directory instead of from wherever npm, brew or your package manager put it.

The stop command leaves the updater behind

The issue reports that codex app-server daemon stop does not stop the update loop. I could reproduce that:

codex app-server daemon stop
# {"status":"stopped", ...}

ps -eo pid,ppid,command | grep '[a]pp-server'
#   767     1 .../bin/codex app-server daemon pid-update-loop

The backend is gone, but the updater is still running as an orphan. To clean it up completely you have to kill that process yourself.

Why this matters at work

A background process on a laptop sounds harmless. For a company it raises three questions that usually have clear answers for every other tool on a developer machine:

What version is running? The daemon updates itself from a copy in your home directory, so the version your package manager pinned is not necessarily the version doing the work. The issue reporter found the two could diverge.
What environment does it use? The issue also reports that the daemon keeps the environment of the session that started it. Proxy settings from the first session stuck, and proxy settings in later sessions were ignored, which matters if your company routes traffic through a proxy for inspection.
Who approved it? Security teams generally want to know about software that persists and updates itself, and an opt-out default rather than opt-in means many developers will not know it is there.

None of this means the daemon is malicious. A long-running backend makes sessions start faster and lets several clients share state. The problem is the default, and that the off switch is not where most people would look.

How to turn it off

The feature flag exists, and in 0.157.0 it writes the setting below. The issue reporter confirms that a new session then no longer starts a daemon.

codex features disable daemon_auto_start
# Disabled feature `daemon_auto_start` in config.toml.

cat ~/.codex/config.toml
# [features]
# daemon_auto_start = false

Then stop what is already running, including the orphaned updater:

codex app-server daemon stop
pkill -f 'app-server daemon pid-update-loop'

If you want the disk space back, the copied package is in ~/.codex/packages/app-server-daemon. Remove it yourself after both processes are gone.

A one-minute audit for every AI CLI

Codex is not the only AI coding tool on most machines, so I wrote a read-only script that checks for orphaned processes from the common ones, the disk their self-managed copies use, and any login items or services that mention them. It changes nothing. It is written for macOS and Linux; I tested it on Linux. Orphans are usually adopted by PID 1, but on desktop Linux and WSL they can be adopted by systemd and on macOS by launchd, so the script checks the parent too.

#!/usr/bin/env bash
# Audit what AI coding CLIs leave running or installed after you close them.
# Read-only. Written for macOS and Linux (including WSL). Usage: bash ai-cli-audit.sh
PAT='codex|claude|cursor|gemini|aider|copilot|windsurf|opencode'

echo "== Orphaned AI tool processes (adopted by init, systemd or launchd, so no terminal owns them)"
ps -eo pid=,ppid=,etime=,command= | awk -v p="$PAT" '
  { line[$1]=$0; par[$1]=$2; exe[$1]=$4 }
  END { for (i in line) if (tolower(line[i]) ~ p) {
          pp = par[i]; pe = exe[pp]
          # orphans are adopted by PID 1, or by systemd --user / launchd on desktops
          if (pp == 1 || pe ~ /(systemd|init|launchd)$/) print line[i] } }' | cut -c1-160

echo; echo "== Disk used by self-managed copies"
for d in ~/.codex/packages ~/.local/share/claude ~/.claude/local ~/.cursor-server; do
  [ -d "$d" ] && du -sh "$d" 2>/dev/null
done

echo; echo "== Codex daemon setting"
if command -v codex >/dev/null; then
  codex features list 2>/dev/null | grep -E '^daemon_auto_start' || echo "codex: flag not found (older version)"
else
  echo "codex not on PATH"
fi

echo; echo "== Login items and services that mention AI tools"
ls ~/Library/LaunchAgents /Library/LaunchAgents /Library/LaunchDaemons 2>/dev/null | grep -iE "$PAT"
command -v systemctl >/dev/null && systemctl --user list-unit-files 2>/dev/null | grep -iE "$PAT"
crontab -l 2>/dev/null | grep -iE "$PAT"
echo; echo "Done. Nothing was changed."

On my test container, after stopping the daemon, it printed:

== Orphaned AI tool processes (adopted by init, systemd or launchd, so no terminal owns them)
  767     1       00:36 ~/.codex/packages/app-server-daemon/releases/0.157.0-.../bin/codex app-server daemon pid-update-loop

== Disk used by self-managed copies
374M    ~/.codex/packages

== Codex daemon setting
daemon_auto_start                        stable             false

The process list check matches on names, so it can also show things you expect, like an editor extension host or a desktop app. Read the list rather than killing everything on it.

What I would do this week

Run the audit on your own machine, then share it with the team.
If you use Codex CLI, decide on purpose whether you want the daemon, and set daemon_auto_start in config.toml either way.
Add "does it persist or self-update?" to the questions you ask before approving any AI developer tool.

Checks like this are part of the LLM security work we do at Axionry, and I wrote up how a coding agent is put together in how to build an AI coding agent.

Sources: openai/codex issue #48195, Hacker News discussion. Codex CLI 0.157.0 tested in a clean Linux container on 13 October 2026; the audit script was run there too. I used an AI assistant while drafting this.