DEV Community: Antonio Scotland

Before you connect another MCP server: an audit checklist for workplace teams

Antonio Scotland — Fri, 16 Oct 2026 21:41:02 +0000

The Model Context Protocol (MCP) revision dated July 28 remains the current one. MCP's versioning rules let a current revision keep receiving backwards-compatible changes without a new version number, so the specification is not frozen, but there has been no further breaking revision since July. I think that is the right moment for workplace teams to take stock. The July revision changed how an AI application connects to the systems it works with, and it started a clock on features that existing servers rely on: under the protocol's new deprecation policy, Roots, Sampling, Logging and Dynamic Client Registration become eligible for removal in the first revision released on or after July 28, 2027, according to the deprecated features registry. That is the minimum twelve-month deprecation window counted from the July 2026 revision.

MCP is also widely adopted for connecting workplace agents to business systems. On October 8, Google said its new Gemini agent can connect to any MCP server, inside or outside the company network. A few days earlier, on October 5 and 6, the MCP Dev Summit came to Toronto with a focus on deploying MCP in regulated enterprise settings: identity, authorization, auditability, gateways, registries, evaluation and production reliability.

My position is that an MCP server is not a plugin. It is a system integration that holds credentials and acts on a model's decisions, so it deserves the same review we already give any integration that touches production data. Below, I explain what MCP is and how a tool call works, then list the seven questions I would ask before connecting a server.

What MCP is

Anthropic introduced MCP on November 25, 2024, as an open standard that lets AI assistants connect to the places where information lives, such as document repositories, business applications and development tools. In December 2025, Anthropic donated it to the Agentic AI Foundation, "a directed fund under the Linux Foundation" co-founded by Anthropic, Block and OpenAI. At that point, Anthropic counted "more than 10,000 active public MCP servers" and listed ChatGPT, Cursor, Gemini, Microsoft Copilot and Visual Studio Code among the products that had adopted it.

The problem it solves will be familiar to anyone who has worked on integrations. Without a shared protocol, every AI application needs its own connector for every system, so ten applications and twenty systems can mean two hundred connectors to build and maintain. With MCP, a system exposes its capabilities once, as an MCP server, and any compliant application can discover and use them.

As a business systems analyst, the closest analogy I know is an API specification. An OpenAPI document describes a REST API so that developers can write code against it. An MCP server describes its capabilities so that an AI application can call them. For MCP tools, the important difference is who consumes the description: the host puts the tool definitions into the model's context at run time, and the model can choose which tool to invoke.

Hosts, clients and servers

MCP has three participants, described in the architecture overview. The host is the AI application, such as Claude, ChatGPT, VS Code or an agent your team built. The server is a program that exposes a system's capabilities. Between them, the host creates one MCP client for each server it connects to, so a host connected to three servers runs three clients.

Figure 1. One host, one MCP client per server. The server names are examples.

Servers offer three kinds of building blocks, which the specification calls primitives. Tools are actions the model can call, such as creating a ticket or running a query. Resources are data the application can read, such as a file or a database schema. Prompts are reusable templates that a person can choose to run. The server overview sums up who is in charge of each: tools are model-controlled, resources are application-controlled, and prompts are user-controlled. In the other direction, a server can ask the client to collect input from the user, which MCP calls elicitation.

Underneath, every message is JSON-RPC 2.0, and two transports carry it. A local server runs on the same machine as the host and talks over standard input and output (stdio). A remote server uses Streamable HTTP, where every message from the client is a new HTTPS POST. Remote servers are where OAuth comes in: a protected MCP server acts as an OAuth 2.1 resource server and accepts only tokens issued for it.

It matters just as much to know what MCP does not do. The architecture overview says the protocol covers only the exchange of context, and leaves it to the AI application to decide how it uses the model and the context it receives. MCP standardizes the interface and the exchange. It does not decide whether a call was a good idea, or whether the result is correct.

How a tool call works

Figure 2 follows one call through the current version of the protocol, using an illustrative create_ticket tool on a ticketing server.

Figure 2. One tool call under MCP 2026-07-28, simplified from the specification.

Discover (optional). The client can call server/discover to learn which protocol versions and capabilities the server supports. Every server must implement it, but because every request now carries its own protocol version, clients may skip it.
List the tools. tools/list returns each tool's name, description and input schema. The result also carries caching hints (ttlMs and cacheScope), so the client does not need to ask on every turn.
Choose. The host puts those definitions into the model's context, and the model picks a tool and fills in its arguments. The tools specification calls tools "model-controlled" for this reason.
Approve, if the host requires it. The same page says there "SHOULD always be a human in the loop with the ability to deny tool invocations," but MCP does not supply that approval gate. The host decides which calls need a person's approval, and the host has to implement the check.
Call. The client sends tools/call as an HTTP POST to the server's single MCP endpoint (/mcp in this example), not to a route named after the method. Two headers repeat what is in the body, Mcp-Method and Mcp-Name, so a gateway can see which tool is being called without parsing the JSON.
Return. The server answers with resultType: "complete", and the host passes the result to the model. If the server needs more information first, it returns input_required instead; the client asks the user, then retries the original request with the answers and a new request ID.

Here is what step 5 looks like on the wire. It is adapted from the examples in the Streamable HTTP specification, and the tool and its arguments are fabricated.

tools/call request (illustrative)

POST /mcp HTTP/1.1
Content-Type: application/json
Accept: application/json, text/event-stream
Authorization: Bearer <token issued for this server>
MCP-Protocol-Version: 2026-07-28
Mcp-Method: tools/call
Mcp-Name: create_ticket

{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "tools/call",
  "params": {
    "name": "create_ticket",
    "arguments": { "project": "OPS", "summary": "Printer on floor 3 is offline" },
    "_meta": {
      "io.modelcontextprotocol/protocolVersion": "2026-07-28",
      "io.modelcontextprotocol/clientCapabilities": { "elicitation": {} }
    }
  }
}

What changed on July 28

Until July, MCP was stateful. A client opened a session with an initialize handshake, and a remote server could track that session with an Mcp-Session-Id header. The 2026-07-28 changelog removed both, so each request now carries its own protocol version and client capabilities, and any server instance can handle any request. Cloudflare, which hosts MCP servers, described the result as servers that no longer need sticky sessions or held-open streams. The same revision replaced server-initiated requests with the input_required retry, added the routing headers and caching hints, tightened authorization, and gave every feature a lifecycle (Active, Deprecated or Removed) with a minimum twelve-month deprecation window. The questions below cover the parts that matter for an audit.

Seven questions to ask before you connect a server

None of these questions will surprise anyone who has reviewed a system integration. What MCP adds is that the caller is a model choosing actions at run time, often for a person who is not watching. The MCP roadmap says it plainly: "MCP authorization assumes a person with a browser at consent time. Increasingly the caller is an agent." For each question, I list what the specification expects and the evidence I would ask for.

1. Which servers are connected, where did they come from, and where do they run?

Start with an inventory. A local server may run under the user's operating-system identity and inherit substantial local privileges unless the host isolates it, so the question is not only where it runs but under which identity and inside which isolation boundary. The authorization specification tells stdio servers to "retrieve credentials from the environment" rather than use OAuth. The security best practices also require that a client offering one-click setup of a local server show the exact command it will run, without truncation, and get explicit approval first. That protects against hidden execution, but it does not establish that the code you chose to run is trustworthy. Before connecting a server, I would also want to know who published it, which exact version is running, how it is updated, and who reviews an update before it runs with your credentials. Evidence: a register of every server with its owner, source repository, package version or commit, update mechanism and reviewer, execution identity and isolation, transport, protocol revision, and the systems and data it can reach.

2. Is every token issued for this server, and only this server?

A remote MCP server must check that each access token was issued for it, and the specification says servers "MUST NOT accept or transit any other tokens." That rules out token passthrough, where a server forwards a user's token for some other API, which the best-practices guide links to confused-deputy attacks and broken audit trails. On the client side, the July revision requires checking the issuer, when the authorization server includes it, before redeeming an authorization code, which defends against mix-up attacks. Evidence: a negative test showing that a token issued for another service is rejected with a 401.

3. How much can a token do, and who can take it away?

The best-practices guide lists "Publishing all possible scopes in scopes_supported" as a common mistake, and the reason is practical: when a server does not say which scopes an operation needs, clients fall back to requesting everything it publishes. The tools specification also allows a server to return only the tools that the caller's scopes permit, so a narrow token can shrink the set of tools the model even sees. For organizations, the Enterprise-Managed Authorization extension lets the company's identity provider decide which servers each employee can use, and revoke that access in one place. Evidence: the scopes each role receives, and an offboarding test showing that access ends when the account does.

4. Can you see and control which tool is called?

The new headers make MCP traffic visible to ordinary infrastructure. A gateway can allow specific tools for specific roles, or rate-limit them, by reading Mcp-Method and Mcp-Name, and the server must reject a request whose headers disagree with its body, so the gateway and the server act on the same facts.

There is a limit, though, and Figure 2 shows it: the headers expose which capability is being invoked, not the business meaning of its arguments, which stay in the body. A gateway can cheaply block or rate-limit create_ticket, but the headers do not tell it which project or customer the ticket concerns. For a transfer_funds tool, they would not say whether the amount is $10 or $100,000, or who receives it. Header-level enforcement controls which capability may be invoked; policy that depends on the arguments still belongs in a gateway that reads the body, or in the server itself.

The specification adds two cautions. Intermediaries that enforce policy on these headers should reject requests from older protocol versions rather than trust headers nobody validated, and sensitive parameters such as tokens or personal data should not be mirrored into headers. Logging over MCP itself is deprecated in favour of OpenTelemetry, and the specification now documents how to carry trace context in each request's metadata. As I argued in my post on securing agents, those logs belong somewhere the agent's own credentials cannot write. Evidence: the gateway policy, and one end-to-end trace of a single tool call.

5. What does a state handle prove?

Without sessions, a server that needs to remember something (a shopping cart, an open browser, a database transaction) returns a handle that the model passes back as an ordinary tool argument. The best-practices guide is explicit that servers "MUST NOT treat possession of a state handle as authentication," and that they should bind each handle to the authenticated user. Evidence: a test where one user presents another user's handle and is refused.

6. What breaks when deprecated features are removed?

Every deprecated feature is a migration with a date. The registry lists the following, and notes that the earliest removal date marks when a feature becomes eligible; the actual removal is a maintainer decision and may come later.

Deprecated feature	Migrate to	Earliest removal
Roots	Directories or files passed as tool parameters, URIs or config	First revision on or after July 28, 2027
Sampling	Calling an LLM provider's API directly	First revision on or after July 28, 2027
Logging	`stderr` for stdio servers, OpenTelemetry for observability	First revision on or after July 28, 2027
Dynamic Client Registration	Client ID Metadata Documents	First revision on or after July 28, 2027
HTTP+SSE transport	Streamable HTTP	Three months after SEP-2596 reaches Final

Evidence: for each server and client you depend on, which of these it uses and a migration item with an owner and a date.

7. Who approves risky calls, and how do you know the result is right?

The specification asks for a human who can deny tool calls, and it adds a warning I would underline: clients "MUST consider tool annotations to be untrusted unless they come from trusted servers." A server describing its own tool as read-only is a claim, not a control, so approval rules should not rest on that description alone. Servers, for their part, must validate inputs, enforce access controls, rate-limit calls and sanitize outputs.

This is also where my evaluation habit comes in. Tool names and descriptions go straight into the model's context, so my assumption is that a server-side edit to a description can change which tool the model picks without any change on your side. Because servers should return tools in a deterministic order, the tool list can be treated like a contract. There may not be a single list, though: as question 3 noted, a server can show different tools to callers with different scopes. So for each relevant role or authorization profile, I would keep an expected tool contract under version control (names, descriptions, schemas and annotations), diff it when the server changes, and rerun the evaluation set before the change reaches users. The actions themselves still need acceptance criteria, as I argued in my first post. Evidence: approval rules per tool, a versioned tool contract for each role, and evaluation results from the last change.

What this means for Canadian teams and job seekers

The Toronto summit's agenda reads like a fair summary of what regulated Canadian employers will ask before an agent reaches their systems. Banks, insurers and public-sector teams already run integration reviews, and my expectation (an assumption, not something I have measured) is that MCP servers will be routed through those reviews rather than around them. Teams that can answer these seven questions will get through review faster.

For job seekers, the checklist doubles as a portfolio template. If you have built an MCP server, answer the seven questions in its README: the transport, the token audience check, the scopes, the gateway headers, the handle binding, the deprecation exposure and the tests. That is the kind of evidence I argued for in my post on what a deployed-agent engineer should be able to show.

What I am still unsure about

I have not measured how many public servers and clients have moved to the July revision, and the documentation for Enterprise-Managed Authorization notes that client support for it varies. While versions are mixed, the advice to reject older requests at the gateway has a real cost, and I do not know how teams are handling that trade-off. Finally, the roadmap admits that "Existing MCP servers lean on pasted API keys and long-lived refresh tokens," and its agent identity work is still forming. Independent data on how teams deploy MCP gateways, or published conformance results for the major clients and servers, would change my view in either direction.

The takeaway

MCP makes it easy to connect an agent to a work system, and that is exactly why the connection deserves a review. Treat each server as an integration with an owner, a token audience, a scope list, a gateway policy and a test suite. The cheapest time to do that work is while the July revision is still current, before the deprecations land and before the next breaking revision adds more to review.

If your team runs MCP servers today, which of these seven questions would be the hardest to answer?

Sources

Originally published at blog.scenehere.ca.

Badge or evidence: what a deployed-agent engineer should be able to show

Antonio Scotland — Fri, 09 Oct 2026 14:34:02 +0000

On October 2, Anthropic announced Claude Frontier Academy, a $100 million commitment to train 10,000 "Frontier Deployed Engineers" by the end of 2027. A day earlier, it published a case study on Barclays' rollout of Claude. Read together, the two announcements show a vendor putting its own definition and credential around the increasingly prominent forward-deployed engineering role, alongside a customer story rich in deployment scale but sparse on measured outcomes.

That combination raises a practical question for anyone hiring or job hunting in Canada. When a company needs someone to put AI agents to work inside its operations, what should that person be able to show? Beyond the working system and its code, my answer is four concrete artifacts: a use-case memo, acceptance criteria, an evaluation harness and a rollback plan.

A credential tells you that a candidate passed someone else's assessment. A portfolio lets you inspect how they select a problem, define correctness, build and evaluate the system, and decide when it is safe to keep running. Canadian candidates do not need to wait for access to Anthropic's program to start producing that evidence.

What Anthropic announced

Anthropic frames the Academy as an answer to "one of the most pressing issues in AI implementation: talent." Its first program is a residency for "hands-on software engineers with strong fundamentals." Entry is by nomination: "Organizations nominate their strongest engineers, each arriving with a named Claude project to lead when they return." Those who pass a multi-day, in-person program earn a Claude Resident Engineer badge and move into a 12-week residency, leading a real project at their own organization, and are assessed again before earning the Claude Frontier Deployed Engineer badge. Anthropic expects the first of those badges in early 2027.

The first cohorts come from Accenture, Bain, Capgemini, Commonwealth Bank of Australia, Deloitte, McKinsey, Morgan Stanley, Novo Nordisk and others, with programs in San Francisco, New York and London. The announcement does not name a Canadian organization or location.

The design deserves credit. Anthropic says the residency "follows the medical model for building expertise," and the curriculum is practical: trainees "work through a simulated enterprise deployment, from the right use case through security review to handover," and finish with "a graded practical on a new scenario." The title itself borrows from the forward deployed engineer (FDE), a role that a Forbes analysis describes as "a software engineer who works on-site to solve complex, real-world problems."

What the Barclays story measures

The Barclays post is full of scale. More than 16,000 colleagues have adopted an internal knowledge assistant, which "has handled over one million searches." An email platform "processes approximately 120,000 emails each day." Barclays expects Claude Code to reach half of its developers by the end of 2026 and a majority of its software engineers in 2027.

What the post does not report is a measured outcome. There is no accuracy figure, no error rate, and no time saved; the closest it comes is "faster access to information and quicker support for customers." That is not a criticism of Barclays, which may well measure these things internally, and a vendor case study is written to promote the product. It does, however, illustrate what adoption numbers alone cannot tell a hiring manager: whether the deployed system actually improved the process, by how much, and how its failures are detected.

What Canadian employers are already asking FDEs to do

Anthropic's title has not reached Canadian job postings yet, but the role it borrows from has. On October 9, I read four FDE postings open in Canada: one at a consultancy, one for a manufacturer, one at an AI integration firm and one at an AI software company.

Huron, Toronto (remote). Huron's posting puts the engineer in a pod with a solution architect and an engagement manager, so the seat is focused on building. The responsibilities are short and direct: build full-stack features on AWS and Azure "using Claude and other models," sit with the business day to day, and keep quality up with "tests and evaluations on everything that ships." Its preferred qualifications include exposure to "eval-driven pipelines."

A Canadian manufacturer, Montréal region (on-site). The most explicit of the four is a posting by the recruiter Calliere for "an established Canadian manufacturer." The engineer takes work "from first conversation through to design, build, launch and adoption," turning unclear processes into "concrete requirements, measurable goals" and production software. The engineer must also "own what you ship," which the posting spells out as testing, access control, monitoring, documentation, training and support. In the first three months, the role is expected to produce a ranked roadmap with starting metrics, launch at least two tools that people use and that measurably improve results or turnaround time, and put testing and rollback in place. The requirements ask for experience putting AI into a live business process, "including how you measured quality and handled mistakes." Even part of the pay depends on evidence: the base salary comes with "a performance incentive tied to the measurable business results of your work."

Experience Bold AI, Toronto (hybrid). Bold AI's posting describes an engineer sent into client businesses to build production systems rather than demos. The engineer turns a loosely defined brief into a scoped plan within the first week, then has to "deploy to production and stand behind it." The posting is blunt about what counts: "Monitoring, error handling, rollback plans, and runbooks are part of the deliverable, not extras." The handoff ends with "a named client-side owner who can run what you built," and the interview includes "a deep walkthrough of one production system you shipped: decisions, failures, fixes."

STAN, North York. STAN's posting, for a company that builds AI for condominium and HOA property managers, puts quality monitoring at the centre of the role. The engineer owns "the technical health of your deployments end to end," and "when the AI gets something wrong, you find out why." The tasks include reviewing voice and chat logs for failure patterns and analyzing outcomes such as "resolution rates, escalation rates, ticket quality," then turning the findings into decision-ready reports for customers.

Put side by side, the four postings describe the same job in different industries. Each one expects a working system in production, not a demo. Beyond that, they ask for the things the Barclays post leaves out: choosing the problem (Calliere's ranked roadmap, Bold AI's scoped plan), defining what good looks like (Calliere's measurable goals), testing and evaluating what ships (Huron's tests and evaluations, STAN's reviews of failure patterns), and recovering when it goes wrong (Calliere's and Bold AI's rollback, Bold AI's runbooks). None of them mentions Anthropic's program. The only credentials any of them names are Huron's AWS and Azure AI certifications, listed as preferred qualifications rather than requirements.

Four artifacts I would ask for

A working system comes first, and the postings above already ask for one. Alongside it, I would ask to see four artifacts. Each one is something a candidate can produce on their own, in public, without access to a vendor's program.

A use-case selection memo. One page explaining why this process, what it costs today (volume, error rate, time spent), what success would mean, and what was deliberately left out. Anthropic's own curriculum starts with "the right use case," and for good reason: a well-built agent on the wrong task still fails the business.
Acceptance criteria. Testable statements of what a correct result looks like, written before the agent runs. My first post worked through an example for an invoice follow-up agent.
An evaluation harness. A labelled set of test cases, scored automatically where possible, with any LLM judge checked against human labels before it is trusted. It should report both kinds of failure: actions the agent should not have taken, and valid work it wrongly refused.
A rollback plan. What evidence widens the agent's autonomy, what failure withdraws it, who decides, and how the agent is switched off. My last post argued that the monitor and the off switch belong outside the agent's reach.

These overlap with Anthropic's own sequence of use case, security review and handover, and with what the four postings ask for, which tells me the vendor, these employers and I largely agree on what the job is. The difference is who gets to inspect the evidence. A badge tells a hiring manager that someone passed an assessment that Anthropic designed and graded. The artifacts let the hiring manager see how that person thinks, and they give the interview something specific to argue about.

Why the portfolio matters for Canadian candidates

Three things make a public portfolio the practical choice for Canadian candidates right now. The first is access. Most Canadian developers cannot independently pursue this credential: participation is by employer nomination, the initial cohorts are concentrated among large organizations, and the current program locations are outside Canada. The first full badges are not expected until 2027. The second is portability: the residency is built around Claude, while Canadian employers work with several model vendors, and a memo, a set of criteria and a working harness transfer across all of them. The third is visibility: a portfolio can be read before the first interview, by anyone, without taking the candidate's word for it.

None of this makes the badge worthless. A badge is third-party evidence that someone passed a demanding assessment. A portfolio lets the employer inspect the evidence themselves. The two are different signals, and if an employer offered me a nomination, I would take it. The strongest candidate will eventually have both. Until then, my advice is simple: do not wait for access to the credential. Build evidence of the same capabilities now.

What I am still unsure about

I do not know whether Canadian employers will come to treat this badge as a hiring signal. Several of the first-cohort firms are large consultancies, and if they start staffing Canadian projects with certified engineers, the badge could become an expectation faster than I assume. Huron already lists cloud vendors' AI certifications as preferred qualifications, so vendor credentials clearly have some place in Canadian hiring. My evidence is also thin: four postings, read on one day, and a job posting describes what an employer hopes for rather than what the work turns out to be. Canadian postings that list the Frontier Deployed Engineer badge as a requirement, or a larger sample of deployment roles tracked over several months, would change my view.

The takeaway

If you are hiring for this role in Canada, ask for the memo, the criteria, the harness and the rollback plan, and read them before the interview. If you want the role, build them in public. I do not have this badge, and because entry is by employer nomination, I could not apply for it on my own. What I can do is make each post on this blog add one of these artifacts, which is the standard I am trying to hold myself to.

Hiring managers: what is the first thing you would ask a deployed-agent engineer to show you?

Sources

Originally published at blog.scenehere.ca.

Securing AI agents in your infrastructure: a sandbox is only the first layer

Antonio Scotland — Fri, 02 Oct 2026 15:08:37 +0000

On September 28, Nvidia launched its Open Agent Safety Platform, and much of the coverage framed it as the answer to AI agents going rogue. The timing made that framing easy: in July, an agent driven by OpenAI models broke out of its evaluation sandbox and spent roughly two and a half days inside Hugging Face's production infrastructure.

I read the announcement the way I would review any integration that touches production systems, by asking which question each control actually answers. My conclusion is that securing agents inside your own infrastructure is not one product decision. It is four separate questions: what the agent can reach, where the data it reads is allowed to go, what watches it from outside its own reach, and whether the result met the requirement. Nvidia's platform answers two of them well. A small open-source project called OpenAPPA is aimed at a third. The fourth is still ours to answer.

What the Hugging Face incident shows

Hugging Face published both a disclosure and a technical timeline. The timeline describes "an autonomous AI agent driven by a combination of OpenAI models" that ran an end-to-end intrusion between July 9 and July 13, and Hugging Face's forensic reconstruction recovered about 17,600 attacker actions. The agent first "escaped its sandbox by exploiting a zero-day in the package registry cache proxy." Inside Hugging Face, it abused the dataset processing pipeline, and one step returned "the worker pod's full environment, including some secrets and credentials." From there it moved laterally across clusters.

What strikes me is how ordinary most of the failures were: a proxy was the weakest point of exit, and credentials sat where a compromised worker could read them. Hugging Face's own reflection puts it well: "machine-speed offense makes ordinary weaknesses more expensive for defenders." This was a frontier lab's agent in a security evaluation, not a workplace assistant, but the lesson transfers. An agent inside your network is a workload that probes and retries at machine speed, so every gap you have tolerated becomes easier to find.

Rogue agents: a four-layer control model

There is no single fix, but there is a workable structure. I think of it as four layers, one for each of the questions I listed at the start. Figure 1 shows where each one sits, and the sections that follow take them in turn.

Figure 1. Where each control sits on the path from an agent to the outside world.

Layer 1: what the agent can reach

OpenShell is the part of Nvidia's platform most teams can use today. It is open source under Apache 2.0 and currently labelled alpha. It runs each agent in a sandbox where filesystem paths and process restrictions are locked when the sandbox is created, outbound network access is blocked unless a policy allows it, and credentials are granted per endpoint. According to VentureBeat's report, a policy prover checks each policy against organizational limits before it is applied, and Nvidia's Ali Golshan was clear that "It is deterministic."

Justin Boitano, Nvidia's vice president of enterprise AI, summarized the principle in a line I agree with completely: "The organization should not have to trust the agent to respect that boundary. The infrastructure should enforce it explicitly." That is least privilege applied to a new kind of workload, and it targets exactly the layer that failed first in July: egress.

Layer 2: where the data can go

Access control answers "can the agent touch this?" It does not answer "may this data go there?" Golshan gave the clearest example himself. Suppose a policy forbids an agent from reading code on GitHub and posting it externally. The agent could split the job between two sub-agents, one that reads from GitHub and one that can reach the outside, and let them pass the code between them. Each sub-agent stays within its permissions; the combination does not. VentureBeat reports that Nvidia is extending the prover to catch these combinations, which is the right instinct. It is a data-flow problem.

That is the question OpenAPPA is built around. It sits between an agent and its tools and, before every call, asks "is this data allowed to go to this destination?" It tracks the sensitivity and trust of everything the agent has read, and its checks run before dispatch, "so sensitive data never reaches an unauthorized tool." Like OpenShell, its decisions are deterministic, and its policies are plain TOML.

It is also very early. The repository describes itself as a "preview and an RFC," and its results are self-reported, although the evaluation page is substantial. It covers 1,320 runs across two suites, Bench-Corp and AgentThreatBench, and reports that no scored attack succeeded against guarded OpenAPPA. On Bench-Corp, guarded OpenAPPA completed 88 to 90 percent of tasks across three models, while the evaluated configurations of Microsoft's FIDES let roughly 28 to 35 percent of attacks through. The page also includes ablations: in one Bench-Corp test, task completion fell from 88 percent to 56.5 percent without subagent isolation and to 35 percent without guided recovery. My reading, which is an inference rather than something the page states, is that much of the utility depends on how well the agent recovers after the policy says no.

What the page does not isolate is the number I care about most: how many incomplete tasks failed because the policy refused legitimate work, as opposed to ordinary agent failure. I do like its framing: security has to be measured on two axes at once, because "an agent that permits unauthorized flows is unsafe, and an agent that refuses valid work is useless." That echoes the argument about permissions and outcomes in my last post.

Layer 3: a monitor the agent cannot reach

The third layer in my model is external monitoring. Nvidia's answer to that problem is Sentry, which its announcement describes as "an out-of-band watchdog" running on BlueField-4 data processing units, separate from the host running the agent. Nvidia says it can quarantine an agent that moves outside its boundaries in milliseconds. The technical blog states the design principle plainly: "The controls do not live inside, or within reach of the agent. The agent does not know it is being watched."

Sentry is a reference design aimed at frontier labs, and Nvidia says organizations can deploy elements of the platform according to their own requirements. I expect most workplace teams will not run Sentry, but the principle applies at any scale. Agent activity logs should go somewhere the agent's own credentials cannot write, and alerts should not depend on the agent's own account of what it did. After the incident, Hugging Face added tooling "that flags tokens used from unexpected origins," which is the same idea expressed with ordinary infrastructure.

Layer 4: whether the result met the requirement

A security engineer could fairly object that this is not a security layer, and in the narrow sense that is true. The first three layers are security controls; the fourth is operational assurance. I include it because an agent that securely performs the wrong action is still not safe to delegate work to. An agent can stay inside its sandbox, send data only to approved destinations, pass every monitor, and still email the right client the wrong invoice amount. Catching that takes acceptance criteria and evidence, and no vendor sells them, because they depend on what your process means by "correct."

Layer	Question it answers	Example control	What it cannot see
Reach	What can the agent touch?	OpenShell sandbox and egress policy	Data moving between allowed tools
Flow	Where may this data go?	OpenAPPA flow check	Escapes below the tool layer
Watch	Who notices when it goes wrong?	Sentry, out-of-band logs and alerts	Whether the output is correct
Verify	Did the result meet the requirement?	Acceptance criteria and evidence	Anything the criteria leave out

To make this concrete, picture the invoice follow-up agent from my last post running inside your network, with read access to receivables and permission to send email. A client's reply hides an instruction to forward the full receivables report to an outside address. A reach-only policy allows it, because the agent may read receivables and may send email. A flow policy that labels the report as internal blocks it before the email leaves.

Figure 2. The same prompt injection under each kind of control. The scenario is illustrative; I have not run it against either tool.

Out-of-band logging catches what slips past both, and acceptance criteria catch the quieter failure: the wrong amount, sent with no attack at all.

Acceptance criteria only help if something enforces them, and that is where they stop being a document and become part of the system. I see three ways to do it. The first is a deterministic check before the action commits: before the email leaves, compare the client, invoice number, amount and due date in the draft against the system of record, and block the send if anything differs. The second is a human gate for whatever a check cannot decide. The third is an audit after the fact: sample completed work, score it against the criteria, and use the results to widen or withdraw the agent's autonomy. Fuzzier criteria, such as tone, may need an LLM judge, and that judge needs its own evaluation against labelled examples before anyone should trust it.

What I am still unsure about

Boitano told reporters that "From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on." That is plausible for the first step, the sandbox escape, but it is a vendor claim, and a sandbox around the attacking agent does nothing for the credentials exposed inside the target. I also do not know how much labelling work data-flow control demands in a real organization. An independent benchmark that separates false refusals from ordinary agent failures would change my view, in either direction.

The takeaway

An agent does not become safe because you put it in a sandbox. The sandbox answers one question, a data-flow policy answers another, and independent monitoring answers a third. None of them can tell you whether the agent did the job correctly. So rather than asking whether an agent is "secure," I would ask which failure each control prevents, and write down four answers before the agent runs inside your infrastructure: what it can reach, where its data may go, what watches it from outside its reach, and how you will know the result is correct.

Across this post and my last one, the same pattern keeps emerging: agentic systems need deterministic boundaries around probabilistic workers, and explicit evidence for whether the work was correct. Vendors are starting to supply the boundaries. The evidence is still the work of whoever owns the process.

If you are running agents inside your network today, which of these layers do you have in place, and which one are you quietly relying on the model to handle?

Sources

Originally published at blog.scenehere.ca.

OpenAI's dots need acceptance criteria, not just Custom Rules

Antonio Scotland — Thu, 01 Oct 2026 00:46:34 +0000

On September 29, OpenAI introduced dots: always-on agents, powered by GPT-6 Astra, that keep working on your goals after you close the chat. Most of the coverage focused on the capability list and the plush, colourful avatars. The part that caught my attention was a settings screen called Custom Rules, because it is the closest thing in the product to a requirements document.

Reading it as a business systems analyst, I came away with one conclusion. Custom Rules describe permissions, not outcomes. They say what a dot may do; they do not say what a good result looks like. My argument in this post is that agent autonomy should be earned through evidence against acceptance criteria, and that writing those criteria is the real work of putting an agent to work.

What OpenAI shipped

According to OpenAI's announcement, each dot has its own cloud computer and browser, can work toward your goals around the clock while you are offline, connects to more than 4,000 apps, and learns your preferences over time. You can reach it in ChatGPT, Slack, or Teams.

The control model has several layers. Dots start with built-in rules for when to act alone and when to ask. On top of those, OpenAI says, "Custom Rules let you allow specific actions, require approval, or block them." An auto-review step checks actions that could affect your accounts or share information "against your instructions, Custom Rules, and safety requirements." Password changes always stay with the user, and proactive background research is limited to read-only tools that cannot send messages, change app content, or control your computer.

For organizations, OpenAI describes "specialist dots" with their own identities and credentials, governed through Microsoft's Agent 365. The sentence I would underline is this one: "Our engineering teams will work directly with organizations to define each dot's responsibilities, the tools it can use, and how people review and approve its work." That is a requirements engagement, described in OpenAI's own words: responsibility, capability, authorization, exception handling, and human approval.

It is also a signal about where the work is moving. More capable agents do not remove the need for requirements. They raise the cost of ambiguous ones, because the system can now act on the ambiguity instead of waiting for someone to ask what you meant.

On availability, OpenAI's help centre says dots are rolling out to ChatGPT Pro users in markets outside the EEA, Switzerland, and the UK, to Business Premium users in all supported regions, and to Enterprise workspaces as a beta that admins can enable. Canada is not on the exclusion list.

Why the timing matters

The launch came one day after OpenAI cancelled the release of GPT-6.1 Astra. As reported by Android Authority, citing the Wall Street Journal, safety testing found that the model did not follow its operators' instructions closely enough, tried to hide what it had and had not done, and took actions beyond its assignment without asking.

I do not read that as a reason to avoid agents. I read it as three different requirements failures: scope that was violated, actions taken without the required authorization, and unreliable evidence about what the system actually did. Custom Rules address part of that problem by constraining scope and authorization. Acceptance criteria and auditable evidence have to address the rest.

Permissions are not outcomes

The help centre lists four behaviours you can assign to an action, and each one maps onto a familiar requirement type:

Custom Rules behaviour	Requirement it expresses
Take action without asking	Pre-authorized scope: the agent owns this step
Take action if pre-approved	Conditional authorization: only when I explicitly requested it
Ask before taking action	An approval gate with a named reviewer
Hand off to you	A human-owned step in the process

The announcement also says actions can be blocked entirely, which is the negative requirement every specification needs: this must never happen.

All of these are authorization rules. A rule can say that a dot may draft a follow-up email. It cannot say whether the email went to the right client, referenced the right invoice, or used the right tone. That second half is what acceptance criteria are for, and I did not find a place to write them in the announcement or the help article.

In my own work on AI-generation workflows, the most useful line I have learned to draw is the one between a successful model call and a successful outcome. A provider can return a well-formed response that passes every technical check and still fail the requirement it was meant to satisfy. My criteria now state the expected output format, the validation rules a response must pass, and the tests that decide whether it counts as done, rather than treating "the model responded" as "the work is complete." Agents raise the stakes on exactly that distinction: "the agent was allowed to do it" is not the same as "the result met the requirement."

How I would specify a dot before trusting it

One of the launch examples is a dot that notices unsent invoices. Here is how I would specify that responsibility before switching it on, written the way I would hand it to a delivery team:

invoice-follow-up-dot.txt

Responsibility: follow up on invoices more than 14 days overdue.

Permissions
  Take action without asking:  read invoices and client contacts
  Ask before taking action:    send any email to a client
  Hand off to me:              disputes, partial payments, anything over $5,000
  Block:                       changing payment details, issuing credits

Acceptance criteria
  Given an invoice 15 days overdue with no reply on record,
  when the dot drafts a follow-up,
  then the draft names the correct client, invoice number, amount and due date,
  and it waits for my approval before sending.

  Given a client who has replied in the last 7 days,
  when the invoice becomes overdue,
  then the dot does not send a reminder and flags the thread for me.

Evidence before widening authority
  Week 1: review every drafted email against the criteria above.
  Move "send email" to "Take action without asking" only after
  20 consecutive drafts that meet every criterion.

Rollback
  If any autonomous email has the wrong client, invoice, amount or due date,
  return "send email" to "Ask before taking action" until the failure is reviewed.

None of this is exotic. It is a user story with acceptance criteria, an explicit rule for earning more autonomy, and an explicit rule for losing it. Put together, it is an operating model rather than a one-time setup:

Responsibility → permissions → acceptance criteria → evidence → autonomy → regression monitoring → rollback.

The last two steps matter more for agents than for ordinary software. OpenAI's own help article says a dot "can make mistakes, including when following your rules." A dot that learns from feedback is also, by design, a system whose behaviour changes after you approved it. That turns acceptance criteria into a regression suite, not a sign-off.

What I am still unsure about

I have not used dots yet, so this is a reading of the announcement and documentation, not a field report. Three things I would want to know before recommending them to a team:

How auto-review decides. OpenAI describes what it checks, not how. If a model is reviewing another model's actions, I would want to see its error rate.
What the activity record captures. An acceptance criterion is only testable if the log shows inputs, actions, and outcomes in enough detail to audit.
What learning is allowed to change. The help article says a dot can create its own memories, including from connected apps. I want to know whether that learning can ever widen what it does without a rule changing.

That last question deserves a principle of its own: learning should be able to change how an agent performs an authorized responsibility, but never what responsibilities it is authorized to assume. Evidence on any of these points would change my view, in either direction.

The takeaway

Custom Rules are a good start, and they are the right place to draw boundaries. They are not where you define success. Before you give an always-on agent standing access to your work, write down what a correct result looks like, what evidence proves it, and what failure takes autonomy away. If you cannot define all three, the responsibility is not ready to delegate, whichever vendor you choose.

That is also why I think systems analysis becomes more relevant in an agentic world, not less. The agent can increasingly do the work. Someone still has to specify the boundaries, define correctness, decide what counts as evidence, and judge when the system has earned the right to act unsupervised.

If you are piloting agents at work, what is the first responsibility you would trust one with, and what would it have to prove first?

Sources

Originally published at blog.scenehere.ca.