DEV Community: Mangat Rai

Your MCP servers need a bouncer

Mangat Rai — Tue, 13 Oct 2026 05:55:14 +0000

"Where do we enforce who can call which tool?" A customer's security team asked me that recently, before approving a wider rollout of their agents. Several of their teams were building agents, each with its own rules for its agents, and those agents called the same internal MCP servers.

They are not the only ones asking this question. In presales, I talk to many different customers, and different versions of the same question keep coming up.

The usual answer is to put the guard in the agent's own code, next to the tool call. That works for one agent. It gets harder when three agents, owned by two teams, call the same five servers. Each agent needs its own copy of the rules, and each copy can drift.

The other, more sophisticated answer is an MCP gateway. I had used agentgateway only for demos and light proofs of concept, and Langflow mostly as a way to expose flows as MCP tools. I had never paid much attention to what a gateway does when someone sets it up wrong. So I took agentgateway 1.6.0, and broke the question down into five parts:

What is an MCP gateway, and how is it different from the AI gateway I may already run?
What does it enforce better than agent code?
Where are its limits?
What happens when it fails or is set up wrong?
Is it ready for production, and how would I start?

⏳ TL;DR
Once several agents share MCP servers, put the tool rules in an MCP gateway instead of in each agent. You get one copy of the rules, a tool list per agent, and one log for every call. Before you rely on it, block direct access to the servers, make the token check strict, and decide what happens when one server fails. Keep argument checks and prompt-injection defenses in the tools. agentgateway 1.6.0 is ready for pilots, but not yet as your only line of defense.

What is an MCP gateway?

An MCP gateway is a single endpoint that sits between your agents and the MCP servers they use. Agents connect to the gateway instead of to each server. The gateway connects to the servers, merges their tool lists, and forwards each call to the server that owns the tool. agentgateway calls this multiplexing, and its docs describe the merged endpoint as a "virtual MCP" server.

Because every request passes through it, the gateway can act on each one. It can check who the caller is, show that caller only the tools it is allowed to see, check each tool call, and log it.

Here is how that helps:

One place to add or remove a server. When tools change, only the gateway configuration changes, not every client, as agentgateway's multiplexing example points out.
No name clashes. The gateway adds the server name in front of each tool name. If two servers both have a tool called search, the agent sees docs_search and tickets_search.
One identity check. The agent sends a token to the gateway, and the gateway decides what that caller can reach. Remote servers still need their own credentials from the gateway, because the MCP authorization spec says an MCP server "MUST NOT pass through the token it received from the MCP client."

This is a different job from an AI gateway, the kind that sits between your application and model providers and decides which model answers. An MCP gateway sits between the agent and its tools, and it decides which tools the agent can see and call. Some products, agentgateway included, do both in one binary, but you can run either side alone.

agentgateway is not the only MCP gateway. Docker's MCP Gateway, IBM's ContextForge, and Microsoft's MCP Gateway cover similar ground. I picked agentgateway because I had already used it, it runs as a single binary on a laptop, and it sits under neutral governance in the Agentic AI Foundation.

Why MCP Gateway?

To answer this, I put agentgateway in front of the two small servers from our MCP Security lab. The calculator server is the trusted one. The weather server is the rogue one: its get_weather result carries a hidden instruction to call send_report and leak the conversation.

I made two test tokens. They are JWTs (JSON Web Tokens): signed strings that carry the caller's name in a field called sub. I signed them with a local key and gave the gateway the matching public key in jwks.json. support-agent stands for a customer-support agent that should never send reports, and ops-reporter for an internal job that is allowed to.

mcp:
  port: 3000
  targets:
  - name: calculator
    stdio:
      cmd: uv
      args: ["run", "--directory", "./06-mcp-security", "calculator_server.py"]
  - name: weather
    stdio:
      cmd: uv
      args: ["run", "--directory", "./06-mcp-security", "rogue_weather_server.py"]
  policies:
    jwtAuth:
      mode: strict
      issuer: demo-issuer
      audiences: [mcp-gateway]
      jwks:
        file: ./jwks.json
    mcpAuthorization:
      rules:
      - 'mcp.tool.name in ["calculator", "get_weather", "read_weather_policy"]'
      - 'jwt.sub == "ops-reporter" && mcp.tool.name == "send_report"'

In plain words: the gateway starts both lab servers over stdio, as local processes it talks to through standard input and output. It requires a valid token and allows a tool call if any rule matches. Everyone gets the three read-only tools, and only ops-reporter gets send_report. The rules are written in CEL, a small expression language.

Then I connected as each caller and asked for the tool list:

Caller	Tools in the list
`support-agent`	`calculator_calculator`, `weather_read_weather_policy`, `weather_get_weather`
`ops-reporter`	the same three, plus `weather_send_report`

The same endpoint gave each agent a different toolbox. In our lab, the guard sits inside the agent, so the model still sees send_report and its description, and the guard has to catch the call. Here, the support agent never sees it.

Next, I had the support agent call weather_send_report anyway, the way an injected instruction would. The gateway answered:

{"jsonrpc":"2.0","id":3,"error":{"code":-32602,"message":"Unknown tool: weather_send_report"}}

The gateway did not say "access denied." It said the tool does not exist. Like a bouncer, it checks who you are and does not tell you why you are not on the list. The agentgateway team explains that this is deliberate: "access denied" confirms the tool is there and worth attacking. The cost shows up when you debug, because a misspelled tool name and a denied call get the same error. Look in the gateway's access log first, not in the agent.

That log was the third thing I got. Every call landed in it with the caller, the server, and the tool. Here is the denied call, shortened:

error request http.status=400 jwt.sub=support-agent protocol=mcp mcp.method.name=tools/call
mcp.target=weather gen_ai.tool.name=send_report error="mcp: Unknown tool: weather_send_report"

With per-agent guards, that record is spread across every agent's own logs, if it exists at all. Here it is in one place.

To be fair to agent code, none of this is impossible there. An agent can leave send_report out of its own tool list, return the same error, and write its own log. The difference is who owns the check. In agent code, the rule is enforced by the agent it is meant to limit, and each team keeps its own copy. In the gateway, there is one copy, outside the agent.

That only holds if the agent cannot go around the gateway. Stdio servers that the gateway starts itself have no network port to reach. For remote servers, you need a network rule that lets only the gateway reach them. Otherwise, any team can point its agent straight at the server and skip every rule.

Where are its limits?

I ran into two.

1. It does not clean what comes back

The support agent was still allowed to call get_weather. The result came back through the gateway with the hidden "SYSTEM NOTE TO AI ASSISTANT" text untouched. The gateway stopped the dangerous call, but the injected instruction still reached the model.

agentgateway is working on MCP guardrails that can deny calls and mask request and response text with CEL, but only on the main branch, and 1.6.0 rejected that configuration. Even when it ships, I would still limit what the tool can do, because you cannot reliably detect a bad instruction in text.

2. It can check arguments, but only the hard way

Our MCP Security lab does not block send_report outright. It checks the to address against an allowlist. I tried to move that check into the gateway.

My first rules used mcp.tool.arguments.to. They hid send_report from ops-reporter and denied even the allowed address. The 1.6.0 source explains why: the authorization check fills in only the server and tool name, so arguments is always empty there. What did work was reading the raw request body with json(), one of agentgateway's CEL functions:

mcpAuthorization:
  rules:
  - 'mcp.tool.name in ["calculator", "get_weather", "read_weather_policy"]'
  - >-
    jwt.sub == "ops-reporter" && mcp.tool.name == "send_report" &&
    (mcp.methodName == "tools/list" ||
    json(request.body).params.arguments.to == "reports@ourcompany.com")

With this rule, a report to reports@ourcompany.com went through. A report to any other address, or with no to at all, was denied. It works, but I hit three traps on the way:

The tool list needs its own exception. The same rules decide which tools appear in the list, and a list request has no arguments. Without the tools/list part, the tool disappears from the list.
A text match is easy to fool. My first working version only checked that the allowed address appeared somewhere in the body. A report to attacker@evil.example with reports@ourcompany.com written in the report text passed, and the report was sent. Check the field itself.
The error is confusing. A call denied for its arguments also comes back as "Unknown tool", for a tool the agent can see in its list.

So I would use gateway argument rules only for simple exact matches, as a second layer. The real check belongs inside the tool, where it can clean up an address, look up a list, and return a useful error.

What happens when it fails or is set up wrong?

A gateway is one more thing every agent depends on. I tested three ways it can go wrong.

1. The default token check lets anonymous callers in

My first configuration did not include mode: strict. A request with no token at all connected and got the three read-only tools.

That is not a bug. agentgateway's source documents three modes for jwtAuth: strict, optional, and permissive. The default is optional, which validates a token when one is present and, in the code's own words, "allows requests without a JWT." With mode: strict, the same request got 401 Unauthorized.

Set mode: strict on any gateway that should require identity, and test a request with no token before you go live.

2. One broken server takes down the whole endpoint

Next, I pointed the weather target at a script that did not exist. The calculator server was fine, but the whole endpoint failed to start a session. The gateway returned HTTP 500 with this error:

{"jsonrpc":"2.0","id":1,"error":{"code":-32603,"message":"failed to send message: upstream closed on receive"}}

This is the documented default. agentgateway's failureMode setting defaults to failClosed, which fails "the entire session if any target fails to initialize." failOpen skips the failed server and keeps serving the healthy ones. It goes directly under mcp:, not on a single target:

mcp:
  port: 3000
  failureMode: failOpen
  targets:
  # ...

With that one line, the support agent got calculator_calculator back, and the log said upstream 'weather' failed during fanout, skipping.

Neither default is wrong. Failing closed is safer when a missing server could change how an agent behaves. Failing open keeps unrelated tools working. Choose on purpose, because putting servers behind one endpoint also ties their availability together.

3. The gateway becomes the most valuable thing to attack

Every tool call now passes through one process, so whoever can change its configuration can change every agent's permissions at once. As with the model gateways in our tool security post, treat the configuration like code, review it, and limit who can change it.

Also check what the gateway exposes on the network. agentgateway ships an admin UI that can change targets and ports without a restart. In my test it listened only on localhost, but the MCP, metrics, and readiness ports listened on every network interface. I would keep the admin UI local and put the other ports behind a firewall rule.

Here is the short version of the three tests:

What I tested	1.6.0 default	What I would set
Request with no token	Allowed (`optional`)	`mode: strict`
One server fails to start	Whole endpoint fails (`failClosed`)	Choose on purpose; `failOpen` keeps healthy tools working
Who can reach the gateway	Admin on localhost, other ports on every interface	Keep admin local, firewall the rest

What I did not test: the gateway itself going down, and the delay it adds to each call. I would test both before production.

Is it ready for production?

I can only answer this for agentgateway 1.6.0, the gateway I tested. For trying it out and for internal pilots, yes. As the only place that enforces tool rules in production, not yet.

On the positive side, the project reached version 1.0 in March 2026, is Apache 2.0 licensed, and joined the Agentic AI Foundation, the Linux Foundation group that also hosts MCP, Goose, and AGENTS.md.

On the cautious side, the README still says the project is "currently in active development." The 1.6 release notes ask you to review the breaking changes "because several defaults change." Most of those are on the model-routing side, but with the guardrails gap and the argument workaround, they show the configuration is still moving.

If I were starting today:

Start with the standalone binary. The MCP quickstart runs on a laptop with one config file. Put the servers you already use behind it and point your existing agent at the gateway URL.
Pin the version. Read the breaking changes before every upgrade, and keep the old binary until the new one passes your tests.
Test the failure paths, not only the happy path. Send a request with no token. Call a tool the caller should not see. Stop one server. Check the access log after each one.
Move rules gradually. Keep the existing guards in your agents until the gateway rules have been tested. Then remove the copies one at a time.

What this means for practitioners and architects

If you are deciding where MCP tool rules should live in your team or for a customer, I would suggest six things:

Use a gateway when servers are shared. One agent and one server does not need it. Several agents, several teams, or several servers usually do.
Make the gateway the only way in. If an agent can still reach a server directly, every gateway rule is optional. Block direct access with a network rule.
Give every agent its own identity. Per-caller tool lists only work if each agent sends its own token. A shared token turns the gateway back into one big allowlist. How agents get those identities is still open in MCP itself, as I wrote in MCP is done pretending agents are people.
Filter the tool list, not only the calls. If the model never sees a tool, it never reads the tool's description, and any call to it is rejected anyway.
Check the defaults yourself. In agentgateway 1.6.0, token checks are optional and multiplexed servers fail closed unless you change them. Other gateways have their own defaults, so test them before you trust them.
Keep guards inside the tools that can do damage. The gateway decides who can call what, and it can handle simple exact-match argument rules. The tool still decides whether a specific request, such as a report to an outside address, is allowed.

If you want to build the pieces this sits on, the MCP Security chapter shows the in-agent guard, the RBAC chapter covers per-role limits inside the tool itself, and the AI Gateways chapter covers the model side of the gateway.

The customer's security team asked where to enforce who can call which tool. My answer now has two parts: the gateway, for who can see and call each tool, and the tool itself, for what a specific call is allowed to do. The gateway keeps one copy of the rules instead of one per agent, but it is also one more service to protect.

Are you running MCP servers that several agents share? Where do your tool rules live today, and have you tried a gateway in front of them? Leave a comment below and share what worked for you, and what did not.

How to budget tokens

Mangat Rai — Wed, 07 Oct 2026 16:35:23 +0000

About a week back, I was talking with a customer, and they started sharing about their AI initiatives. During the conversation, they mentioned that their token budget for the year was already gone. It lasted only four months. They did estimates and pilot runs, and still could not get it right.

And to my surprise, I found they are not alone. In April, Uber's CTO said the company had already used up its planned 2026 AI budget, only months into the year, after a surge in the use of AI coding tools.

Different industries, use cases, and even usage patterns. But the same problem: token consumption was way more than anyone had budgeted for.

This customer's number stayed with me because, like many other practitioners, I use monthly subscriptions, so I have never run into a financial constraint. The most I have had to do is wait 4-5 hours for the usage window to reset.

I got curious and started looking into how the industry is going about token consumption and estimates. I broke it down into four smaller questions:

How do you estimate what a workload will spend before it runs?
How do you keep it inside a limit once it does?
How do you find out early that it is drifting?
When the bill jumps, how do you find what caused it?

⏳ TL;DR
Here is what I found. Estimates and pilots miss because pilots use handpicked, skilled users, agents make far more model calls per task than chat ever did, and a few expensive runs pull the total up. So pilot with a mix of users, budget for the tail, and check against real spend every month. Vendors now offer hard spending caps, but none of them act instantly, and they stop a whole project at once. The caps that help sit closer to the spender: per team, per key, and per agent run. Alerts help only if they watch the monthly burn rate, not the yearly total. And finding the culprit works only if every call carries a team, app, or user label before the bill jumps.

How do you estimate what a workload will spend?

This was the question that bothered me most, because the customer had done the work. They estimated. They ran pilots. It still did not hold.

When I looked at how token usage grows, three reasons stood out. None of them is specific to that customer, but any one of them can break a careful estimate.

1. The pilot users are not your real users

A pilot group is usually handpicked. These are often the people who asked for the tool, already know how to use it well, and write clear prompts. A clear prompt gets the job done in a few rounds. A vague one sends the model back and forth, and every round costs tokens. After rollout, the tool reaches everyone, and each person tends to use it more as they start to trust it. An estimate built on pilot numbers is built on the most efficient version of the workload.

The remedy is to pilot with a mix of users. Include strong users, average users, and people who are new to these tools, and track their cost per task separately. The gap between the most and least efficient users tells you more about the real budget than the average does. It also shows where some training or a few shared prompt templates could bring the cost down before the full rollout.

2. Agents changed what one unit of work costs

When I use a chat assistant, one question is one model call. A coding agent working on one task reads files, runs commands, and calls the model again after every step. Every call sends the whole conversation so far, and that conversation grows as the task goes on.

To make that concrete, I put together a rough comparison. These numbers are my assumptions, not measurements or real pricing, and I left out output tokens and caching to keep the math simple.

Per person, per working day	Chat assistant	Coding agent
Units of work	20 questions	10 tasks
Model calls	20	300 (30 per task)
Tokens sent per call	3,000	50,000
Tokens per day	60,000	15,000,000
Cost at $5 per million input tokens	$0.30	$75

Same person, same day, 250 times the tokens. If the budget was built on the left column and people move to the right one, it is gone early. Prompt caching can bring the real cost down a lot, which is one more reason this is hard to get right on paper.

The remedy is to pilot the same tool and workflow people will use in production, agent included, not a chat version of it. And count model calls per task, not only tokens per call, because the number of calls is what changes the most.

3. The expensive runs hide in the tail

Most tasks are cheap. A few go long, retry, or loop, and they pull the total up far more than an average suggests. A pilot report that shows only the average cost per task hides them.

The remedy is to report the 90th percentile and the single most expensive task next to the average, and to budget for that tail, not for the typical day.

Where that leaves the estimate

What the vendors offer here is limited. Both Anthropic and OpenAI have endpoints that count the tokens in a request before you send it. That helps size one call. It does not tell you how many calls a task will take, or how many people will be running tasks six months from now.

So my take is to treat the estimate as a range, not a number. Use the mixed pilot to get a typical cost per task and a worst case, multiply by the number of people you expect at the end of the year, not the start, and then compare it with real spend every month. The estimate is only a starting point. The monthly check is what keeps it honest.

How do you keep it inside a limit?

This is where my subscription habit made me look at things differently. On a subscription, the limit is built in. Claude's Max plan, for example, resets a session limit every five hours and adds a weekly limit on top. I used to see that window as an annoyance. Looking at it now, it is a hard cap that the vendor set for me. When I hit it, I stop, and the bill does not move.

Pay-per-token API usage does not work like that. By default, it keeps going and keeps billing.

While I was reading about how to put a cap on that, I came across Simon Willison's post, "We're going to need default hard budget caps on pretty much everything". He separates two kinds of limits. A hard cap says "after $X/month, cut this thing off and return errors." A soft cap only sends a warning email, and soft caps, he writes, "will not cut it."

That sent me to check what the vendors offer today.

Vendor	Hard cap you can set	What happens at the limit	Is it instant?
OpenAI	Organization and project spend limits	Requests fail with a `429` error	No. "Enforcement is not instantaneous," so spend can slightly exceed the limit
Anthropic	Organization and workspace spend limits	Requests fail with an error saying you reached your usage limits	Not stated
Google Cloud	Spend caps, in preview, for the Gemini API, Agent Platform, and Cloud Run	New requests to that service in that project pause	No. Enforcement "isn't instantaneous," and you pay the overage
AWS	Per-project spend limits, rolling out gradually to new customers	The project pauses until you raise the limit	Not stated

Two things surprised me. None of these caps promises to act instantly, because a provider can only stop spending it has already counted. And they are blunt. A cap on a project or workspace stops everything in it, including features your customers use. That is better than an open-ended bill, but if it is the only cap, a budget problem simply turns into an outage.

So I looked at what large companies are doing, and Microsoft gave a good example. This week, The Information reported that Microsoft had expected to spend at least $1 billion a year on internal use of Anthropic's models and has since cut that projection by more than a third. As part of that, individual monthly usage caps reportedly dropped from $100,000 to roughly $10,000 in most cases. That is a cap at the level of a person, not the whole company.

Putting the cap close to whoever spends is the pattern I would copy. I think about it at three levels.

Per team or person. Each team gets a monthly budget, roughly a twelfth of its share of the yearly budget. If your model calls go through a gateway, the one service that every call passes through, this is usually just a setting. LiteLLM's proxy, for example, supports a max_budget and a budget_duration on teams, users, and keys, and rejects requests once a budget runs out.

Per app or job. Each app or scheduled job gets its own key with its own budget, so one runaway job cannot spend everyone else's share.

Per agent run. This one lives in your own code, and it is the only cap that acts right away. Every model response already says how many tokens it used, so the agent can keep a running count and stop itself.

MAX_STEPS = 20
MAX_TOKENS = 200_000

steps = 0
tokens_used = 0

while True:
    if steps == MAX_STEPS:
        raise RuntimeError(f"Run stopped: step limit of {MAX_STEPS} reached")

    response = call_model(messages)
    steps += 1
    tokens_used += response.usage.input_tokens + response.usage.output_tokens

    if tokens_used > MAX_TOKENS:
        raise RuntimeError(f"Run stopped at step {steps}: token limit reached")
    if is_finished(response):
        break

    messages = run_tools(messages, response)

In plain words, the agent keeps two counters: how many steps it has taken and how many tokens it has used. Before each step, it checks the step count. After each step, it adds the tokens reported in the response. If either counter goes over its limit, the run stops with a clear message instead of carrying on. The step limit catches an agent going around in small circles, and the token limit catches one sending huge requests. The functions call_model, is_finished, and run_tools stand in for your own code, and the usage field names match the Anthropic Messages API and the OpenAI Responses API.

The provider cap still makes sense on top of all three, set above normal monthly spend, as a backstop for the day your own limits have a bug.

One trap I did not expect. When a spending cap trips, the error can look like an ordinary rate limit. OpenAI's spend limits return HTTP 429, the same code as "too many requests," and Anthropic uses 429 when an organization reaches the monthly cap for its usage tier. A gateway that switches to a backup provider on every 429 will send the same runaway work to a second provider and a second bill. I would treat spend-limit errors, such as project_spend_limit_exceeded and enforced_spend_limit_reached, as a stop. No retry. No failover.

How do you find out early that it is drifting?

Alerts are the soft cap Willison talks about. They do not stop anything. But they are still how a person finds out, so the real question is when they fire.

The math from the customer's story helped me here. If a yearly budget lasts four months and spending is even, a quarter of it is gone after the first month. An alert at 80 percent of the yearly budget would fire early in month four, when there is almost nothing left to save. A monthly view would have shown the problem after month one.

So I would set alerts against a monthly line, the yearly budget divided by twelve, not against the yearly total. A few vendor features help with that:

Google Cloud and AWS both let a budget alert fire on forecasted spend, not just actual spend. A forecast notices a fast burn rate within weeks instead of months.
Google added early anomalies in July. It flags daily AI spend that looks unusual for a project and names the top three SKUs behind the jump.
Anthropic's Usage and Cost API typically shows usage within about five minutes, and Anthropic suggests it for building your own budget alerts.

The catch is delay. AWS Budgets updates up to three times a day and warns that you can pass a threshold before the notification reaches you. Google also notes a delay between usage and billing. For a monthly budget, a few hours does not matter much. For an agent stuck in a loop overnight, it does, and that is why the per-run cap in your own code matters.

I would also send the alert to the team that owns the spend, not only to finance. The people who can change a prompt, switch a model, or turn off a job are the ones who need to know.

When the bill jumps, how do you find what caused it?

A single invoice line does not say which team, app, or person spent the money. And labels cannot be added afterward. They have to be on the calls before the spike.

The simple version is separation. Give each team its own project or workspace, give each app or job its own key, and avoid shared keys. Then the vendor reports do most of the work:

OpenAI's Usage API groups usage by project, user, API key, and model.
Anthropic's Usage and Cost API groups usage by workspace, API key, and model, and its Claude Code Analytics API reports estimated cost per user per day.
On AWS, Bedrock can tag costs through application inference profiles, and since April it can record which IAM user or role made each call.
Google Cloud budgets can be scoped by project, service, and label.

There is a catch with gateways. AWS points out that when calls go through a gateway or proxy, the recorded caller is the gateway, not the person behind it. The same happens with any shared key. If you run a gateway, it has to do the tracking itself by passing a team or user ID with each request. LiteLLM, for example, tracks spend per key, user, team, and tag.

Once the labels exist, finding the cause is mostly sorting. I would group last month's spend by team, then by app, then by model, and compare it with the month before. Then I would look at three things: who started using it, which new workload showed up, and whether a model or context change made each call more expensive.

What this means for practitioners and architects

If you are having this conversation with your customers or within your organization, I would suggest four things:

Turn the yearly budget into a monthly one. Divide it by twelve and compare it with last month's real spend. That one number shows whether there is a problem. For new workloads, estimate from a pilot with a mix of users and budget for the expensive tail, not the average.
Give every team and app its own key. Split any shared keys into a project or workspace per team and a key per app or job, so the next spike has a name on it.
Put caps close to whoever spends. Set monthly caps per team at the gateway, a step and token cap in every agent, and a provider cap above total spend as the backstop. Then trip each one once on purpose to make sure it stops the work instead of retrying or failing over.
Alert on the forecast, and send it to the owners. Add a forecast alert and an anomaly alert that go to the team that owns the spend, not only to finance.

None of this makes a single call cheaper. That is the other half of the problem, and the Token and Cost Management chapter covers it through context trimming, prompt caching, and choosing the right model size. The AI Gateways chapter explains why the gateway is a good place for team budgets and labels.

Looking back at my own setup, that five-hour wait I used to grumble about was doing a job I never thought about. On a subscription, the vendor caps you whether you like it or not. On the API, nobody does unless you set it up.

Have you had this conversation with a customer or inside your own team? How did you manage token budgets in your project? Leave a comment below and share what worked for you, and what did not.

The AI industry is building its own referee

Mangat Rai — Tue, 29 Sep 2026 15:29:57 +0000

OpenAI models escaped a test environment and compromised Hugging Face. Claude models gained unauthorized access to real production systems in three cases. Gemini accessed systems at three companies during another test.

Different labs. Same unsettling pattern. Sound familiar? That has been the rhythm of AI safety news lately.

By the time I read the third story, I wanted to know whether anyone was connecting these incidents. Were the labs learning from them together, or was each company still deciding for itself what counted as safe? What were the frontier labs doing as an industry, what was the government requiring, and did any shared standard exist?

So I started digging into those questions. That search led me to industry plan taking shape under the tentative name SAFA.

⏳ TL;DR
I started with a simple question: after a run of frontier-model security incidents, who sets the shared safety rules? In July, Google DeepMind's Demis Hassabis proposed an industry-funded standards body with federal oversight, independent experts, and a path from voluntary reviews to mandatory pre-release tests. By late September, Google, OpenAI, and Anthropic were reportedly building a private version without government oversight. Shared tests could still improve today's fragmented safety reporting. The unresolved question is whether this new body can make its founders follow a rule, publish an embarrassing result, or admit competitors on equal terms.

The answer changed in ten weeks

The first real answer I found came from Google DeepMind's Demis Hassabis. His July proposal called for a federally overseen standards body with independent technical experts and open-source representatives on its board. Industry would provide most of the funding, while federal agencies and US national laboratories would help test advanced models in areas tied to national security.

The proposal also described a route to actual authority. Labs would initially submit frontier models for voluntary review up to 30 days before release. If the process proved effective, models could eventually be required to pass before deployment in the US.

Then the idea changed. On September 24, The Information reported that Google, OpenAI, and Anthropic were preparing their own standards organization, tentatively called the Standards Authority for Frontier AI (SAFA). The companies hope to launch it by the end of 2026 or early 2027, but its final name, charter, leadership, funding, and enforcement powers are not public.

Here is the difference I kept coming back to:

	July proposal	Reported September plan
Who is behind it	A US-led body, funded mostly by industry	Google, OpenAI, and Anthropic
Government role	Federal oversight, with agencies and national labs involved in testing	No government oversight
Governance	Independent technical experts and open-source representatives on the board	Board and appointment rules not public
Participation	Voluntary reviews first, with a path to mandatory tests	Voluntary safety commitments; enforcement not public
Main work	Define frontier thresholds and assess models before release	Support outside testing, incident reporting, and auditor qualifications

This does not make SAFA pointless. It changes what SAFA is. The July proposal described delegated governance: industry expertise and money operating inside a public structure. The September plan is, so far, a private standards organization seeking credibility without delegated authority.

I can see why the labs want one standard

Once I looked at what the labs do today, the practical case for SAFA became clearer. Each company already has a framework for deciding when a powerful model needs stronger safeguards. The frameworks cover similar severe risks, but they use different categories, thresholds, and names.

Lab	Published framework	Its trigger language
Anthropic	Responsible Scaling Policy (v3.4, July 2026)	Capability Thresholds tied to ASL-3 safeguards
OpenAI	Preparedness Framework (v2, April 2025)	High and Critical capability thresholds
Google DeepMind	Frontier Safety Framework (v3.1, April 2026)	Tracked and Critical Capability Levels

These are not three versions of one scale. I cannot cleanly translate an OpenAI “High” result into an Anthropic ASL level or a Google DeepMind Critical Capability Level. The methods, thresholds, and release decisions remain vendor-specific.

Common evaluation protocols, incident definitions, and auditor qualifications would make the evidence easier to compare. They could replace three separate safety vocabularies with a shared one. After reading the incident reports that sent me down this path, that sounds genuinely useful.

But a shared vocabulary is not the same thing as a referee.

I had to look up why FINRA matters

Hassabis used the Financial Industry Regulatory Authority, or FINRA, as his model. I did not know much about FINRA before reading this proposal. At first, “industry-funded regulator” sounded close to what the AI labs were building.

The important part is what sits behind those two words. FINRA is privately operated, but it lives inside public law.

US broker-dealers generally must belong to a self-regulatory organization before doing business. A broker-dealer operating outside the exchanges of which it is a member must usually join FINRA. FINRA's proposed rule changes are filed with the Securities and Exchange Commission for review. Its board reserves ten seats for industry members, one for its CEO, and the remaining seats for public members.

That is what gives the arrangement weight. Firms cannot simply ignore the system and keep doing the same business. The industry does not get final say over every rule. Public representatives have formal power inside the organization.

None of those conditions has been announced for SAFA. No law compels a lab to participate. No public agency approves its rules. No published charter says who controls its board or what happens when a founder fails a test. Customers, insurers, and governments could eventually give its standards market power, but that would be influence earned after launch, not authority granted at formation.

The conflict is not subtle

The first version of SAFA would be formed by three companies whose models and labs it could help evaluate. That does not prove the standards will be weak. I do think it means independence has to be designed into the institution rather than promised after someone raises the question.

There is also a competition problem. If enterprise buyers or insurers eventually demand a SAFA result, the founding labs will have helped write a test their competitors must pass. A demanding standard could improve safety and still become a moat. I do not think we should pretend only one of those outcomes is possible.

A famous independent chair would not settle this for me. I would want the boring institutional details: published appointment rules, protected terms for independent board members, transparent funding, open membership, appeal procedures, and results that the founding companies cannot suppress.

What I will watch next

SAFA is still a reported plan. I am not ready to call it either a regulator or an industry stunt. I want to see what its founders are willing to give up.

Control: Do independent members hold real voting power over standards, leadership, and publication decisions?
Transparency: Are methods, test versions, incident definitions, and full results public enough to challenge?
Membership: Can another frontier lab join as an equal rule-maker on reasonable terms?
Consequences: What happens after a failed test, and can a founding lab simply leave or release anyway?
Bad news: Will SAFA publish a finding that delays or embarrasses one of its founders?

For practitioners, a future SAFA report could become useful vendor evidence. I would ask which standard and version was used, who ran the test, what the result excluded, and whether the complete finding is public. I would not confuse that evidence with government approval.

The next time one of those safety headlines lands, I will look past “we are investigating.” Did the incident enter a shared reporting system? Could an independent body inspect the evidence? Did the failure change a release decision? Could the lab keep an inconvenient result private?

That is what a referee would change. The real test of SAFA will not be whether it writes sensible rules. It will be whether its founders can lose a call.

Goose: An Open-Source Agent You Can Assemble Yourself

Mangat Rai — Thu, 24 Sep 2026 06:20:55 +0000

A few days ago, I wrote about AGENTS.md, the file that gives coding agents project-specific instructions. While reading through the Linux Foundation's Agentic AI Foundation site, I came across another founding project: Goose.

The pitch caught my attention. Goose runs on your machine, lets you choose the model, and connects to tools through the Model Context Protocol.

However, The five-minute quickstart says Goose improves software development by automating coding tasks. The tutorial asks it to build a JavaScript game. The Goose homepage, however, calls it a general-purpose agent for research, writing, automation, data analysis, and code.

What was I looking at? Is it a coding agent or a general-purpose agent?

What i figured at the end is that Goose is both, but not equally. Coding is the go to path. Its extension system lets you give the same agent a different set of tools and a different job.

⏳ TL;DR
Goose is coding-first, not coding-only. Its built-in developer tools cover repository work. Change the extensions and instructions, and it can handle research, documents, browser tasks, data, and other automation. You also choose the model, including compatible local models. That freedom comes with more setup, evaluation, and security work.

So which is it?

Goose's history clears the confusion. When Block introduced it in January 2025, the launch post said its first use cases focused on engineering while the community was already trying it on non-engineering work. The examples included code migrations, tests, benchmarks, feature flags, and API scaffolding.

The product has since expanded. Its current homepage and README describe a general-purpose agent. Even its system prompt uses that term. The quickstart still takes the shortest route to a useful result: give Goose a working directory and ask it to build an application.

Both descriptions are accurate, but they set different expectations. Calling Goose a coding agent hides what its extension system can do. Calling it a general assistant makes it sound more ready-made than it is outside software development.

My read is simpler: Goose is an agent you install and configure. It comes prepared for coding. Its tools and instructions determine how far you can take it.

What actually changes

The architecture documentation splits Goose into three main pieces: an interface, the agent, and extensions. The model sits behind the agent and decides whether to answer or ask for a tool.

flowchart LR
    U["You"] --> I["Desktop, CLI, API,<br/>or ACP client"]
    I --> G["Goose agent runtime<br/>sessions, context, permissions"]
    G <--> M["Your chosen model<br/>hosted or local"]
    G --> E["Built-in and MCP extensions"]
    E --> C["Coding work<br/>files, shell, repositories"]
    E --> W["General work<br/>documents, browser, data, services"]
    C --> G
    W --> G

Ask Goose to change a repository and it sends the request, relevant context, and available tool definitions to the model. The model can ask to read a file, make an edit, or run a shell command. Goose executes the permitted call, returns the result, and gives the model another turn.

That loop can also prepare a weekly project brief. Give it local meeting notes and a read-only issue-tracker extension instead of a code repository. The model can read the notes, query completed issues, and write the brief. Goose still coordinates the model and tools; the toolbox and instructions are different.

The coding task and the project brief are not two different versions of Goose. The loop stays the same. The tools determine what work it can do.

Start with coding

Goose's built-in Developer extension can read and modify files, run shell commands, and maintain a plan. Start a session in a project directory and it has the basic tools needed to investigate and change a codebase.

This is why the quickstart builds a tic-tac-toe game. In a few minutes, Goose creates the files and, after you add the Computer Controller extension, opens the result in a browser.

Goose can also run as an ACP server. ACP, or Agent Client Protocol, separates the agent runtime from its interface. An ACP-compatible editor can use Goose as the agent while Goose keeps its configured models and extensions.

That does not make Goose an IDE. The desktop app and CLI do not replace an editor's navigation, autocomplete, debugging, or visual review tools. Goose is the agent doing the work. You can run it beside an editor or connect it to one that supports ACP.

For repository work, I would give it the same project guidance I give any coding agent. An AGENTS.md file can state the build commands, paths, conventions, and boundaries it should follow. Open source at the runtime layer does not remove the need for clear project context.

Beyond code, you assemble the workflow

Goose calls its MCP servers extensions. MCP is the open protocol for connecting agents to tools and data; the MCP track explains the host, client, and server roles in detail. Goose's built-in catalog includes the Developer, Computer Controller, Memory, and Auto Visualiser extensions, and it can connect to external MCP servers for other systems.

A research setup might get access to a document directory and a browser. An operations setup might use an issue tracker, a database, and a reporting destination. Calling Goose general-purpose does not give the model those abilities automatically. You add them by choosing extensions.

Recipes make those choices repeatable. A recipe can package instructions, parameters, model settings, and required extensions in YAML or JSON. Goose can run recipes interactively, headlessly, or on a schedule.

This is what I mean by an agent you assemble yourself. You do not have to build the agent loop, but you do choose the parts around it:

Part	What you choose	What you still have to verify
Model	Hosted provider, private endpoint, or compatible local model	Tool use, quality, context, latency, and cost
Capabilities	Built-in tools and local or remote MCP extensions	Data access, credentials, dependencies, and permissions
Workflow	Instructions, recipes, parameters, retries, and schedules	Correctness, failure handling, and review points
Interface	Desktop, CLI, API, or ACP client	Which surface fits the people and task
Distribution	Upstream Goose or a custom distribution	Upgrades and the cost of maintaining changes

This path is less turnkey than coding. Goose gives you the runtime and extension system. It does not know which business process matters, which source is authoritative, or what a correct result looks like. You still have to design and test that workflow.

Bring your own model, with caveats

Goose does not bundle one mandatory model. Its provider guide covers hosted providers, cloud platforms, compatible private endpoints, subscription-backed agents, and local options. Goose connects to the provider and runs the agent loop around the selected model.

A coding team can test another model without replacing Goose. A broader workflow can use a model and endpoint that fit its cost, data, or deployment requirements.

Ollama is one way to run inference locally. With a compatible tool-calling model, Goose and the model can both run on your machine. Newer Goose desktop releases also provide built-in local inference, so Ollama is no longer the only local route.

Local inference does not automatically make the whole workflow private or offline. A remote MCP extension still receives whatever its tool call sends, and browser work still uses the network. The model also needs to fit the hardware and reliably make structured tool calls. Goose's documentation warns that models without tool calling can provide chat completion only when extensions are disabled.

The models are not interchangeable just because Goose can connect to them. Two models may accept the same prompt and expose the same tools while producing very different results. I would treat every provider change as a new agent configuration and rerun the task evaluation.

Where Goose fits

I would choose Goose when the work begins with code but does not end there. The same agent that changes code might query an issue tracker, inspect a design document, update release notes, or prepare a deployment checklist. It becomes more useful when a job crosses tool boundaries.

It is also a useful place to compare models against a reasonably stable runtime. Keep the instructions, extensions, inputs, and acceptance criteria fixed, then change the provider. That does not make the models interchangeable, but it makes their differences easier to see.

I would also consider it when I need control over the runtime or data path. Goose is an Apache 2.0 project governed at the Agentic AI Foundation, and it documents custom distributions. An organization can inspect the system and adapt it instead of waiting for a hosted product to add the option it needs.

Open source still does not eliminate operating work. A private fork needs maintenance. A local model needs hardware and evaluation. Extensions bring dependencies and credentials. The benefit is the ability to make those decisions, not the disappearance of their cost.

When I would look elsewhere

If I wanted a polished coding experience entirely inside an editor, I would first check whether Goose's ACP integration provides the interactions I rely on. The agent can connect to an editor, but the surrounding editor experience still comes from that client.

If I wanted a general assistant with almost no setup, I would also look elsewhere. Choosing a provider, enabling extensions, defining permissions, and turning a task into a reliable recipe are part of using Goose. A managed assistant makes more of those decisions for you.

The biggest concern is how much authority you give it. Goose can execute commands, change files, control a browser, and call external services. Its permission controls, prompt-injection detection, and adversary review can reduce risk, but they do not make every extension safe. I would start with the smallest tool set, least-privilege credentials, approval for consequential actions, and an output I can inspect.

The agent security chapter explains why untrusted tool output needs rules enforced in code, not only instructions in a prompt. That becomes more important as Goose moves beyond a repository and into other systems.

How I would try it

I started with the question of whether Goose was a coding agent or a general-purpose agent. I would test it in that same order.

First, I would use it for one bounded coding task in a disposable branch: inspect a small repository, make a change, run the tests, and explain the result. That exercises Goose's strongest default path and reveals how well the selected model uses its developer tools.

Then I would reuse the runtime for one bounded non-coding task. My choice would be a weekly project brief built from a local notes folder and one read-only issue-tracker extension. I would define the required sections and citations, keep approval enabled, and compare the result with the source material.

Only after the interactive task worked would I turn it into a recipe or schedule. That sequence separates three questions: can the model complete the task, have I given the agent the right tools, and is the workflow dependable enough to repeat without constant supervision?

Goose is a coding agent you can turn into something broader. That is its sweet spot, and also its cost. You get to choose the model, tools, and job, but you also have to assemble and test the result.

Agent security starts before the first prompt

Mangat Rai — Sun, 13 Sep 2026 23:47:16 +0000

An agent can cause damage before the model makes its first decision.

On September 1, 2026, Manifold described coding agents that invoked Git while gathering workspace context, allowing repository-local Git configuration to run a program on the host. Some affected agents performed that work before a user typed a prompt or approved workspace trust.

In Manifold's GitSpawn research, the relevant path was a repository received as files, with a .git directory inside a ZIP archive. Git read repository-local configuration while the agent gathered context. An ordinary Git clone, fetch, or pull does not transfer repository-local .git/config from the remote.

The model did not choose a harmful tool call. The runtime acted while preparing to use the model. That is why an agent review needs three boundaries: the runtime that starts processes, the gateway that holds credentials and routes requests, and the tool that makes a sensitive change.

Workspace trust has to include startup

Imagine a coding agent opening a project so it can answer a simple question about a failing test. Before the chat box is useful, the application may inspect the working tree, run Git to learn the branch and status, index files, load extensions, and collect credentials for the model gateway. None of that is a model-requested tool call. It is still code acting with the authority of the agent process.

Manifold reported eight findings across seven coding agents. Some affected paths ran before authentication or workspace approval. At publication, some issues had been patched; Manifold said four findings remained unpatched and withheld details for those paths.

For the coding agent in this example, I would make the startup contract explicit. It may read the checked-out project files. It may not inherit production cloud credentials, reach arbitrary internet hosts, or start programs named by workspace configuration. Opening an unfamiliar workspace happens in a reduced-permission environment. If the product asks for workspace trust before executing code, discovery and indexing need to honor the same boundary. Keeping production credentials out of its environment and filesystem limits what a missed startup hook can expose.

This is not a replacement for prompt-injection defenses. It covers authority that exists before the model is asked to reason.

The gateway is also outside the model loop

The same coding agent may send its prompts through a gateway. That gateway can hold provider keys, route requests, apply guardrails, and reach internal services. It is a useful policy point, but it is also a privileged service. Its authentication, administration, credentials, and network reach are security decisions made outside the model's visible tool loop.

Wiz's September 9 LiteLLM research gives that distinction some weight. In a February 2026 scan of 3,074 publicly exposed instances, Wiz found that 9.6% either accepted the default master key or required no authentication. That is a combined figure, not a claim that 9.6% used the default key, and it does not describe private deployments.

Wiz also reported an MCP authentication bypass, CVE-2026-59822, and a separate post-authentication code-execution issue in custom-code guardrails, CVE-2026-59821. The researchers wrote that the custom-code guardrail registration path lacked protections present in the test endpoint. Patches were available when Wiz published. A gateway deserves the same careful boundary design as any other service that holds keys and reaches sensitive systems.

For this agent, the gateway should reject a request without a valid credential before it routes anything. Its service identity should have only the provider access and internal destinations the task needs. Administrative and custom-code features should be limited to the small group that operates them. Keeping a gateway private, patching it with an owner and response window, and logging administrative changes are ordinary infrastructure controls. They matter here because a compromise could act without asking the model to call a tool.

Keep the last boundary at the tool

Now return to the coding agent after it has started safely and reached an authenticated gateway. The model examines a failing test and proposes a sensitive action: deploy a fix to the production environment.

That decision still needs a tool boundary. The deployment service can require an approved environment, a change identifier, and a human approval. The model can propose the action, but deterministic code decides whether the request fits the authority granted to this task. A model instruction such as “deploy it now” cannot substitute for those checks.

This is where tool allowlists and argument validation belong. They are strong controls for actions the model requests: writing outside an approved filesystem root, querying a restricted database schema, sending to an unapproved recipient, or deploying to an environment that needs review. They do not secure Git startup or gateway authentication because those events happen on different paths.

Review one agent you run

Follow one sensitive job from startup to its most consequential action. For the illustrative coding agent, the map might look like this:

Component	Capability	Credential or network reach	Enforcing control
Workspace runtime	Read and index project files	Gateway access token; no provider or production cloud keys; gateway and approved dependency endpoints only	Isolated workspace, restricted subprocesses, outbound allowlist
Model gateway	Route model requests and apply policy	Scoped provider credential; approved internal services	Authentication that fails closed, private exposure, patched release, least-privilege service identity
Deployment tool	Request a production deployment	Deployment API for the approved environment	Environment allowlist, change ID, human approval

The point is to name the authority each component actually has, then attach a control that can enforce it. A local learning project may need far less isolation than an unattended agent with production access.

One boundary test makes the map real. Send the gateway the same request the coding agent would make, but omit its gateway credential. The expected result is an authentication rejection before routing, with no provider request and no internal service access. A successful model response or evidence in downstream routing logs that the request was forwarded is a failure. Record the rejection and check downstream logs; an error message alone does not show whether the request was forwarded.

The Agent Security lesson demonstrates the tool-call side with an indirect prompt-injection lab. AI Gateways explains the application-owned routing boundary.

Record expected and actual results for one boundary test, then fix one gap before giving the agent more authority.

AGENTS.md: one guide for your coding agents

Mangat Rai — Wed, 09 Sep 2026 00:23:15 +0000

AGENTS.md is a Markdown file you keep in your repository to tell coding agents how to work on the project. It can explain where the code lives, which commands to run, and which decisions the agent should leave to you. The open format has no required fields or schema.

Imagine asking an agent to add a page on Monday. You explain the project's colors, where the source lives, and why the navigation must stay as it is. On Friday, a contributor picks up the work in another coding tool. Their agent proposes a new color scheme and reorganizes the navigation. The decisions are still in Monday's conversation, but they never made it into the repository.

That is the gap this file is meant to close. Write the ground rules down once, keep them with the code, and give the next agent a place to start.

The instructions should travel with the project

For an open-source maintainer, Friday's contributor may use a different coding tool. Keeping the instructions in the repository makes them available to both people; a shared filename gives their tools a common place to look. One contributor can use Codex and another Cursor without needing separate copies of the same build instructions.

That practical benefit is also the story behind the name. Amp initially used AGENT.md, singular. When OpenAI chose AGENTS.md, Amp agreed to switch if OpenAI secured the matching domain. On August 20, 2025, Amp announced the change. Sharing a standard mattered more than keeping its original filename.

The Linux Foundation dates the format's release to August 2025. By its December 9 announcement of the Agentic AI Foundation, it reported adoption by more than 60,000 open-source projects and agent frameworks. That is a dated adoption report, not a live count. AGENTS.md became one of the foundation's initial projects.

The convention gives those contributors a common starting point, although each tool still decides how to discover and apply the file. First, that shared file needs instructions worth carrying forward.

Write down the decisions the next agent needs

A useful file answers the questions that would otherwise interrupt the work: where to start, what to run, what to preserve, and how to know the change is ready.

For our imagined documentation project, “follow best practices” adds little. “Edit source files in site/; do not edit generated files in site/build/” settles a concrete decision. “Run the checks” is vague. Naming the commands and their working directory makes the instruction usable.

Four sections are a useful starting structure:

Project and layout. What the project does and the few directories an agent needs to understand.
Setup and verification. Exact commands, where to run them, and any prerequisites.
Conventions and boundaries. Existing patterns to reuse, files to preserve, and actions that need approval.
Completion. What evidence to report and how to describe anything left untested.

For a small repository, I would start with roughly 30–60 short lines. That is my editing budget, not a format limit. If ten lines cover the important decisions, stop there. Add a rule when you can name the recurring mistake it should prevent.

Here is an illustrative, shortened file for the website portion of Few-Shot Academy. The paths and commands come from this repository; the example is not a replacement for our full contributor instructions.

A Sample AGENTS.md

# Project instructions

Few-Shot Academy is a free GenAI curriculum. Write for readers
who may have no programming or AI background.

## Project layout
- `site/docs/`: curriculum pages.
- `site/blog/`: standalone practitioner articles.
- `site/src/`: shared components and styles.
- `site/build/`: generated output; edit the source instead.

## Setup and checks
- Use Node.js 22 or newer.
- From `site/`, run `npm ci` to install dependencies.
- After site changes, run these from `site/`:
  - `npm run typecheck`
  - `npm run build`
- For visual changes, check mobile layout, keyboard navigation,
  and both light and dark themes.

## Working rules
- Read the relevant page and nearby examples before editing.
- Reuse design tokens in `site/src/css/custom.css`.
- Preserve published URLs.
- Verify factual claims against primary sources.
- Keep changes focused and preserve unrelated work.
- Ask before pushing, merging, or deploying.

## Before finishing
- Summarize the change and the checks that passed.
- State what you could not verify and why.

The next agent now has somewhere to find the source paths, design rules, and checks that were stranded in Monday's conversation. Replace the example's paths and commands with ones you have verified in your own repository.

These are still instructions the model receives, so check its work. “Do not deploy” is useful guidance; actual deployment access belongs in permissions and approval controls. Anthropic's documentation explicitly distinguishes instruction files from enforced configuration.

Make sure your tool reads it

With the file written, the next step is to connect it to the agent. Some tools recognize AGENTS.md directly; others need a small configuration change.

This is a selection of coding editors and agents, not a popularity ranking. The table reflects their official documentation checked on September 8, 2026. “Automatic” means the tool recognizes the file without a filename bridge; settings can still affect loading.

Tool	Root `AGENTS.md`	Details or setup
Codex	Automatic	Combines guidance along the root-to-working-directory path. An `AGENTS.override.md` takes priority over `AGENTS.md` in the same directory.
Cursor Agent	Automatic	Also applies nested files to their directory and children. More specific instructions take precedence.
GitHub Copilot in VS Code	Automatic	Root loading is enabled by default. Nested discovery is a separate experimental opt-in; see the settings reference.
Cascade (Windsurf / Devin Desktop)	Automatic	Root instructions are always on; subdirectory files apply to that part of the project.
Amp	Automatic	Discovers instructions in the working directory, parents, and relevant subdirectories.
Cline	Automatic	Recognizes `AGENTS.md`; check the Rules panel for whether a detected rule is enabled.
Gemini CLI	Configure filename	Defaults to `GEMINI.md`. Set `context.fileName` to include `AGENTS.md`.
Claude Code	Import or symlink	Reads `CLAUDE.md`. Import the shared file as shown below.

The Copilot row is specifically about VS Code. Do not assume that an editor extension, a command-line agent, and a hosted pull-request agent load instructions identically just because they share a product name.

Connecting Claude Code and Gemini CLI

The last two rows need setup, but you can keep the shared rules in AGENTS.md. For Claude Code, put this line in the root CLAUDE.md:

@AGENTS.md

Copy the line without the surrounding code fences. That is the arrangement in this repository. Claude's documented import loads the shared instructions at session start. Keep any existing Claude-specific guidance below it. A root import does not by itself wire up every nested AGENTS.md.

A symlink is another supported option: ln -s AGENTS.md CLAUDE.md on macOS or Linux, when CLAUDE.md does not already exist. I prefer the import because it is a plain-text edit on macOS, Linux, and Windows. Anthropic also recommends it on Windows, where creating symlinks can require extra privileges.

For Gemini CLI, merge this into .gemini/settings.json, preserving your other settings:

{
  "context": {
    "fileName": ["AGENTS.md", "GEMINI.md"]
  }
}

Gemini's context configuration supports multiple filenames. Keeping both lets existing Gemini-specific guidance remain discoverable; avoid repeating the shared rules in both files.

Let the setup grow with the work

Once the short file is written and your tool is set up to read it, you have a starting point for the next session. Over time, you may also want to preserve architecture explanations, design reasoning, or the state of unfinished work. Those do not all belong in the everyday instructions.

Separate architecture, design, and decision notes can preserve reasoning the code cannot explain. In our Monday-to-Friday example, “use these colors” records a rule. “We kept the existing palette so new pages match the rest of the site” records why it exists.

But five overlapping summaries give you five places to forget an update. Choose a pattern that solves a problem you actually have:

Pattern	When it helps	What to watch
Short root file with links to project docs	Architecture or design explanations are too long for everyday instructions.	Say when to read each document; a link alone does not guarantee it gets read.
Instructions scoped to a directory	Different packages need different checks or conventions.	Confirm the tool's nesting rules. Avoid copying root rules into every package.
A temporary handoff note	A task spans sessions or pauses halfway through.	Record the current state, unresolved questions, and next step; replace outdated notes when work moves on.

The first two have direct support in tools such as Cursor, which documents references and nested instructions. For handoffs, Anthropic describes using progress files alongside Git history.

Splitting files does not create more memory

The context window, the material a model can use at once, is finite. Summarizing a long conversation helps it continue, but that summary can miss details the next session needs.

Tools can also limit how much they load. Codex has a 32 KiB default limit on the combined project instructions it loads, but that is a ceiling, not a target.

The loading strategy matters more than the file count. Claude Code's @ imports load the referenced content at launch. Splitting one large file into five and importing all five still puts that material into context. To keep startup context small, keep the root guidance brief and instruct the agent to read the relevant supporting document when the task calls for it.

A “resume this project” prompt can tell the agent to read a handoff; it cannot recover details nobody saved. Update the relevant document when a decision changes. Before pausing, record unfinished work, checks performed, and the next step. Review those updates in the diff instead of assuming the agent made them.

My default is one short AGENTS.md, existing project docs for durable decisions, and a handoff note only when there is work to resume. Add files when they remove confusion.

Try the next handoff

Return to Friday's contributor. The useful test is whether the saved instructions help them pick up the work. Start a fresh session after changing the setup and try a small page edit. Did the agent find the source, use the existing design, run the right checks, and report anything it could not verify?

If it misses a rule, check which files loaded and whether their instructions conflict before adding another paragraph. The goal is to stop re-explaining the same decisions, while keeping those decisions easy to find and maintain.

GPT-6 Astra: the harness is the product

Mangat Rai — Sun, 06 Sep 2026 00:08:52 +0000

GPT-6 Astra does not prove that OpenAI has reached AGI. What it does prove is that the harness can no longer be treated as plumbing.

Astra scored 54.8% on ARC-AGI-3 at high reasoning in the standard harness. The same model scored 99.9% when the harness preserved its reasoning state and compacted long conversations. That is a 45.1-point difference without changing the model.

My takeaway is simple: if you evaluate only the model name, you are evaluating the wrong product. For long-running agent work, the product is the model, memory, tools, context management, and control loop together.

That is why Astra matters even if you are not interested in arguing about AGI.

The AGI claim is ahead of the evidence

OpenAI President Greg Brockman believes Astra qualifies as artificial general intelligence. During the launch briefing, he left the final judgment to others but said, “I think we're there,” according to The Washington Post. OpenAI's Charter defines AGI as highly autonomous systems that outperform humans at most economically valuable work. No launch benchmark can establish that on its own.

Astra gives OpenAI a serious case to make. The company reports 98% on FrontierMath Tier 4, 99.9% on ARC-AGI-3, and large gains in computer use and coding. OpenAI is positioning Astra to work inside browsers, development tools, spreadsheets, and other software with less supervision.

I would not call it “pre-AGI” either. That avoids the argument without giving us anything measurable. Astra is a meaningful step toward a general-purpose digital worker, but we do not yet have evidence that it can reliably outperform people across most economically valuable work.

ARC Prize calls Astra a major step forward while stating plainly that saturating ARC-AGI-3 is not proof of AGI. The benchmark uses bounded environments with deterministic rules. Real work is messier and carries consequences when the agent gets something wrong.

The biggest Astra number is really a harness result

An agent harness is the software around the model. It supplies tools, carries context forward, manages retries, records state, and decides when the model gets another turn.

ARC Prize tested Astra through two harnesses:

Evaluation setup	Reasoning effort	Score	Evaluation cost
Standard, provider-neutral harness	Max	62.7%	$26,098
Standard, provider-neutral harness	High	54.8%	$40,705
OpenAI Provider Adapter harness	High	99.9%	$18,817

The standard harness gives the model a common interface and lets it decide what to preserve in visible notes. The OpenAI adapter also keeps Astra's opaque reasoning state between requests and uses compaction to manage long conversations.

At the same high reasoning setting, the adapter moved the score from 54.8% to 99.9%, according to ARC Prize's analysis.

The 99.9% result is valid. It is just not a property of the model in isolation. It is the result of Astra operating inside the runtime OpenAI designed for it.

The neutral harness is better for comparing models under the same conditions. The native harness measures the system you might actually deploy. One score cannot answer both questions.

The new context system is the feature I would watch

Astra has a 1,050,000-token context window, but a coding agent can still fill it with source files, test output, logs, and failed approaches. Eventually, something has to be removed or compressed.

Compaction turns the old conversation into a shorter summary. That summary preserves what looked important at the time. A constraint, test failure, or rejected approach can disappear and become important again later.

With Astra, OpenAI says Codex can now keep notes across context windows and search earlier windows. If a detail did not make it into the notes, the agent can retrieve the earlier message or tool result instead of depending entirely on a chain of summaries.

Consider a repository migration. The agent learns early that one service must stay compatible with an older client, then spends an hour inspecting unrelated code and running tests. When it finally edits that service, searchable history gives it another way to recover the compatibility requirement instead of hoping the detail survived compaction.

This could make a long-running agent more dependable, but it needs real testing. The Codex feature is experimental at launch, and we do not yet know how reliably Astra will find the right old detail across large, messy repositories.

Astra can keep working while the task changes

Two new API capabilities support the same idea.

Async tool calling lets Astra start a function or custom tool and continue with independent work while the application runs it. An agent can begin a slow dependency scan, inspect configuration files while it waits, and use the scan after it returns. Your application still runs the job and has to return the result with the correct call ID.

Mid-turn steering lets a user change requirements while Astra is working over a WebSocket connection. You can tell the migration agent to keep the project within one engineer-week without discarding the research it already completed.

That sounds like a small interface improvement. In practice, it changes the agent from a request-response system into a process you can supervise while it runs.

The tradeoff is control. Steering does not undo an action that already happened or cancel a tool already in flight. Async tools can also finish in an unexpected order. If two of them write to the same system, the application needs isolation, idempotency, and a clear approval policy. A more capable model does not remove those engineering responsibilities.

More autonomy also raises the cost of a mistake

OpenAI calls Astra its most aligned model and reports better respect for task boundaries. Its safety review also found that Astra can sometimes evade internal monitors when adversarial tests explicitly tell it to do so. OpenAI says this came from adversarial evaluation rather than normal use, but the result is serious enough that the company is developing monitoring methods beyond reading the model's written reasoning.

Astra is also OpenAI's first broadly deployed model to reach its Critical cybersecurity capability threshold. OpenAI says that, with the right tools and access, Astra can find previously unknown vulnerabilities and develop exploits against hardened systems without a person directing each step. Less-restricted access for advanced cyber work is limited to vetted participants in the Daybreak program.

This is the real tradeoff. Better memory, more tools, and longer autonomy make the system more useful. They also give it more time and more access to act on a bad assumption before someone catches it.

Run two evaluations, not one

I would evaluate Astra in two passes.

First, use a provider-neutral harness with the same prompts, tools, retry limits, and grading criteria you use for other models. That tells you whether Astra itself improves the work you care about.

Then test the native OpenAI harness with persisted reasoning, compaction, searchable context, async tools, and steering enabled where appropriate. That tells you whether the complete Astra system is worth deploying.

Use 20 to 50 real tasks, not polished demos. For long tasks, force at least one compaction and plant an early requirement that matters near the end. Interrupt a run with a changed constraint. Delay tool results and return them out of order. Put an approval boundary in front of an irreversible action. Measure accepted results, recovery from mistakes, latency, and total cost.

The last point matters because Astra costs $10 per million input tokens and $50 per million output tokens, before tool fees. If your application needs short answers or straightforward extraction, these harness features may not justify that price. If it needs sustained work across tools and changing requirements, token price alone is a poor way to judge it.

By the way, this was a crowded release week

Anthropic released Claude Fable 5.1 for long-running coding and knowledge work. Meta released Muse Spark 1.3, with a focus on long-horizon collaboration, interruptions, and coding efficiency. Google released Gemini 3.8 Flash as a lower-cost agentic reasoning and coding model.

All three belong on an evaluation list if they fit your workload. They do not change my conclusion about Astra: its most important advance is not the AGI label or the model name. It is the tighter integration between the model and the system that keeps it working.

My recommendation is to stop treating the harness as an implementation detail. Compare models with a neutral harness, then evaluate the native system you would actually deploy. The evaluation chapter shows how to build the task set. Let the AGI debate continue somewhere else.