DEV Community: Tirth Gajjar

How to set AI agent permissions and access control

Tirth Gajjar — Tue, 10 Nov 2026 18:45:44 +0000

Set AI agent permissions one action at a time, for one workflow, and never by approving the agent as a whole. Write down each change the workflow makes to another system, give each change its own tool with typed arguments, and check the asking person, the target record and its current state in code before the call runs. Keep the credential in a vault or broker that attaches it at call time, so the model never sees it.

We build agents this way. On Skionis the agent reaches a live AWS account only through purpose-built tools. On Solarpunk the credentials stay in an encrypted vault on the user's machine, outside anything the model reads, so neither design asks anyone to trust the model's judgement about what it is allowed to do.

The usual access-control tools answer a different question, which is why this is harder than it looks. An OAuth scope says what an application may do at a server, such as files:write. It does not say that the agent may move one onboarding case to ready_for_provisioning, for the person who asked, while that case is still verified.

Identity vendors cover tokens and delegation well, and our guide on MCP authorization covers the token a run should hold. This guide covers the step before that: listing what one workflow changes, deciding a scope for each change, and putting the authorization check and the credential where the model cannot talk its way past them.

Why not approve the agent as a whole?

Because nobody can review an approval that broad. An AI agent reads data, picks a tool, prepares a change, makes it and checks the result, and each of those steps carries a different amount of authority. Reading a customer record is not changing it, and preparing a refund is not sending one. An approval that covers the agent covers all of those steps at once, including the ones nobody has looked at.

What the agent can attempt also grows without anyone deciding it should. A new MCP server, a browser tool or a model update adds capability, and an approval written as "the onboarding agent may manage cases" covers each of those the day it arrives.

Approve an action instead, written so that a reviewer can say yes or no to it:

Update the status field on eligible onboarding cases from verified to ready_for_provisioning in the EU production tenant, after a named operator approves, within 15 minutes of that approval.

That sentence names a verb, a field, a starting state, an end state, a tenant, an approver and a time limit. Security, product and operations can each object to one part of it, which they cannot do with "manage cases". The time limit matters as much as the rest, because an approval with no expiry becomes standing permission for every later run that finds it.

How do you list the actions in one workflow?

List the changes before you write a single tool, because the list decides which tools exist. Walk the workflow the way a person does it today, and write down every point where something outside the agent changes. Phrases such as "handle the ticket", "process the claim" or "update the CRM" each hide several changes behind one verb. Split them until each line has one verb and one object, then decide line by line whether the agent gets a tool for it.

For the onboarding workflow above, the list comes out like this:

Step in the workflow	What it changes	Decision for the agent
Read the case and its documents	Nothing	Allowed, for cases the asking user can already open
Draft the next status and a reason	Nothing until someone submits it	Allowed, as a proposal a person reads
Move the case from `verified` to `ready_for_provisioning`	`status` and `status_reason` on one case	Allowed after a named operator approves, for 15 minutes, and only while the case is still `verified`
Correct identity, payment or compliance fields	Regulated fields on the customer record	No tool. A person does it in the existing screen
Close the case	The case, plus a provisioning job in another system	Not yet. The agent cannot confirm the provisioning job finished
Change alerting, audit or monitoring settings	What the team watches the agent with	No tool, and no credential that reaches them

The last three rows matter most. A field the agent should never touch is safest with no tool at all, rather than a tool with a rule against it. A rule can be misapplied by the code that enforces it, while a tool that does not exist cannot be called at all.

An action whose result the agent cannot confirm stays with a person until something independent can check it, as our guide on AI agent verification explains. In the onboarding list, closing the case waits until the provisioning job can be read back from the system that runs it, rather than taken from the agent's word.

Give the monitor no tool and no credential, even for an agent you trust, because an agent can read routine instructions as permission to silence it. OpenAI's misalignment report of 20 October 2026 has coding agents on internal tasks doing exactly that to the alerts watching them. An agent with no write path to the monitor cannot do it, whatever it concludes. Guardrails in the prompt cannot promise the same, because they rely on the same reading of instructions that went wrong.

Why give each action its own tool?

A tool that names one operation can carry its own rule. update_case_status, taking a case id, a target status and a reason, can be allowed for one status change and refused for every other, because the check can read each argument.

We do not give an agent a general tool such as run_sql or http_request, because it cannot be scoped like that. The argument decides the operation, so allowing the tool allows every statement its credential can run. That is the excessive functionality OWASP files under excessive agency, third in its 2026 Top 10 for LLM applications: the agent holds functions its task does not need.

We built Skionis on the opposite rule. Its agent applies changes to live AWS accounts, and the tools we gave it are its whole action set. They read account state, generate a fixed template format, validate it, draw an architecture diagram, and apply through CloudFormation change sets. There is no general shell, and no path by which a step in the plan becomes an arbitrary API call.

Solarpunk works across a leader's email, calendar, documents, CRM and ERP, and there each tool is a typed contract with declared inputs, outputs and side effects. The agent can act only through tools someone registered, and it cannot invent access it was not given. With more than a hundred registered tools, the list is still one a reviewer can read from top to bottom. Our guide on MCP tool design covers how large each tool should be.

Where should AI agent authorization run?

Authorization runs in code, at the tool layer, before the call leaves your process. It compares the request with the person who asked, the task, the target record and its current state, and with any approval the action needs, and it records the version of the rule it applied. A prompt telling the model to stay inside its permissions does not count, because the model can misread it and a prompt injection can argue against it.

Security engineers call this complete mediation: the system that holds the data decides on every request, rather than trusting whatever called it. AVOX, an insurance voice agent we built, puts that check in the tool contract. The model's output is a request to call a tool and never an action in itself, and each policy, claim, payment and endorsement tool carries its own permission check, required disclosures and escalation rule.

A request that fails any of those does not run. Nobody can promise that a model will always remember a check, but anyone can confirm that a tool will not run without one, by reading its code or by calling it with a request that should fail. That difference is why the check lives in the tool and not in the instructions the model is given.

Run the check against the record as it is now, rather than as the agent saw it when it planned. A plan can be several steps and several minutes old by the time the call happens, and the case may have been edited in between. We met this on Skionis with cloud accounts, where a template can pass schema validation and still be refused by AWS because of a quota, a region without the service, or a missing resource.

So the Skionis agent reads the live account before it plans, since only the account can say whether a change will apply. For a database record the same rule is simpler: compare the version the agent read with the current version when the call arrives, and refuse and plan again when they differ. A refusal there costs one more planning step, while a write against stale data can overwrite an edit a person made a minute earlier.

Should an AI agent have its own identity?

Yes, as long as it also carries the identity of the person it acts for. The permission it uses should be that person's, narrowed to the task. On a shared service account, the system the agent calls can see that the service connected, but it cannot see who asked, or whether that person could have made the change in the ordinary screen.

RFC 8693, the OAuth token exchange standard, names the two options. Under impersonation the agent is treated as the user and gets all of the user's rights, while under delegation it keeps its own identity and acts on the user's behalf. A delegated token records this in its act claim. The token's subject is the user, and the claim names the agent as the party acting now. For an agent that changes things, delegation is the version you can audit, because both names reach the system that records the change.

We build on token exchange because it exists today. NIST has opened a project on how to identify, authorize and audit AI agents, but it is a proposal, not a standard, and an agent shipping this quarter cannot wait for it.

On the telecom marketing platform we are building agents for now, the analytics agent reads through a data layer that applies the client's access rules. It sees only the tables, columns and rows that the asking user is entitled to, and personal data is masked or left out by the client's policy, rather than by an instruction in the prompt. In this phase of the work, every agent action is proposed by the agent and approved by a person.

Record who asked on the thing that changed, as well as in a log. Skionis tags every resource it creates with the owner, the environment, the tool that made it and the request behind it. Attribution that is not written at creation will not be pieced together later.

The tags pay off six months on, when someone has to decide whether a resource is safe to delete. When one agent hands work to another, the person who asked is still the one whose permission applies, and the second agent's identity shows which process sent the request, not who wanted the change. Our guide on prompt injection in multi-agent systems checks that at each hop, because a message from a trusted agent can carry text it only read.

How do AI agents handle authentication?

Through the tool layer, which attaches the credential when a tool runs, so the agent uses a capability without holding the credential behind it. Once a key is in the model's context, a prompt injection can ask for it, the run log can store it, and a later run can reuse it. Rotating the key closes the copy you found, but it does not tell you which calls used it. A credential the model never sees cannot leak by any of those routes.

Solarpunk runs as a desktop application rather than a hosted service. The credentials, the tool registry and execution all stay on the user's machine, and the model is called only as a reasoning service. Secrets live in an encrypted local vault and are injected into the tool adapter, the code that calls the outside system, when a tool runs, so they never enter the prompt.

The agent can then act inside an application where the user is signed in, and it drives that session without ever learning a password. We treat this as the condition for letting the agent act in a signed-in browser at all, since a password the model has seen can turn up in any later output.

The same split works on a server. Keep the long-lived secret in a broker. Have the tool layer issue a short-lived token for each run, scoped to the tools that run may call, and never pass the secret itself to the agent. Our guide on MCP authorization covers that token in detail, including why it is bound to one server.

Local MCP servers need a second look. The MCP authorization specification, as of its 28 July 2026 revision, uses OAuth 2.1 for servers reached over HTTP. It tells servers that run over stdio to read credentials from the environment instead. So a local server often starts with a long-lived key in an environment variable, on the same machine as the agent.

Run that process where the agent cannot read its environment, or put the key behind a broker. Our guide on AI agent sandboxing covers the network limits and stop rules around a broker. They matter for the case where the model tries a call it was never given a tool for.

When is this more than you need?

An assistant that only reads, over data the asking user could already open, needs that user's entitlements and little else, because nothing changes and there is nothing to approve. Check those entitlements in code all the same. An agent that reads through a shared service account can show one user another user's records without anything looking wrong.

A prototype that runs against a copy of the data, with no production credential anywhere in reach, can leave this exercise until it gets near real systems. Do it the first time a tool would change a record that another person or system depends on, and from then on the list grows one line at a time.

Each line raises a second question: does a person still approve it, and what would let you stop asking them? That is the subject of our guide on human in the loop AI agents. Our guide on AI agent governance covers moving one action at a time to less oversight, and the evidence each move needs.

Common questions

What are AI agent permissions?

AI agent permissions are the actions an agent may take on other systems. Write each one as a verb, an object and the conditions for the change. A usable permission names the field that changes, the states that allow it, and the tenant and environment. It also names who must approve it, if anyone, and for how long. "Manage the account" is not a permission, because it hides several changes behind one word.

What authorization does an AI agent need before it acts?

A check in code at the tool layer, before the call leaves your process. It compares the request with the person who asked, the target record and its current state, and any approval the action needs. Then it records the version of the rule it applied. A prompt telling the model to stay inside its permissions does not count, because a prompt injection can argue against it.

How do AI agents handle authentication?

Through the tool layer, which attaches the credential when a tool runs, so the model never reads it. Keep the long-lived secret in a vault or broker and give each run a short-lived token scoped to the tools it may call. Remote MCP servers use OAuth 2.1 over HTTP. Local stdio servers read credentials from the environment, so run those where the agent cannot read it.

Should an AI agent have its own identity?

Yes, alongside the identity of the person it acts for. Under OAuth token exchange, RFC 8693, a delegated token keeps the user as its subject and names the agent in an act claim. The system that records the change then sees both. A shared service account shows only that the service connected, not who asked or whether they could have made the change themselves.

Is an OAuth scope enough to limit what an agent does?

Not on its own. A scope such as files:write says what an application may do at a server. It does not say which record may change, which state change is allowed, or which person the change is for. Those checks belong in the tool layer, in code, before the call runs, against the current state of the record.

Why not approve the whole agent?

The agent's tools, models and workflow paths change after the approval, and an approval of the agent covers every change that arrives later. Approving one action keeps the decision attached to a consequence someone has already read. Security, product and operations can each object to a specific part of it, such as the target, the time limit or the approver.

Why RAG systems fail in production

Tirth Gajjar — Sun, 08 Nov 2026 18:47:36 +0000

RAG in production fails because retrieval, context assembly and generation break for different reasons, and one accuracy number hides which of them broke. The answer sounds plausible, cites something nearby, and still misses the fact the user needed. Teams then swap models, change prompts, rebuild indexes or add more chunks, hoping one change fixes everything at once, and that rarely works.

A production RAG system is three systems that fail separately. Retrieval has to find the right evidence, context assembly has to put usable evidence in the prompt, and generation has to stay faithful to it. Around those three sit the corpus, permissions, freshness, logging and what the product does when the evidence is not there. If you do not measure them separately, every bug looks like "the model hallucinated".

This guide draws on three retrieval systems we built: an ESG filings platform in production, our own prospect research engine, Prospex, and a trademark drafting platform that has been live since 2023. In all three, the failure that mattered most was not the model. It is a correct-looking answer built on the wrong entity, the wrong year or the wrong jurisdiction, which every downstream check passes. The sections below take each layer in pipeline order: what breaks, how you would see it, and what to try first.

How does retrieval work for RAG in production?

At query time the question is turned into an embedding, and the index returns the passages whose embeddings sit closest to it. Filters remove what the user may not see, and the best of those passages go into the prompt beside the question. Around that query path the system has four stages, and you debug them in the order the data flows through them. The corpus has to be parsed, chunked and turned into embeddings, with permissions attached. Retrieval, usually vector search plus filters, has to put the right passage in the candidate set. Context assembly has to fit a usable, ordered slice of that evidence into the context window. Generation has to stay faithful to the evidence or refuse.

Each stage hides choices that matter more than which vector database you bought. How are tables represented when a document is parsed, and are permissions applied inside the search or after it? How many candidates are retrieved before reranking, and does hybrid search catch exact strings? Does the answer cite a passage or a whole document? A weak pipeline with a strong model still fails when the right passage never reaches the prompt, and a strong pipeline still fails when the model treats evidence as optional.

Here is that pipeline, with the trace that records each stage underneath it:

Corpus: parse, chunk, embed; version, permissions.
Retrieval: rewrite, filter, search; recall before precision.
Context: rerank, dedupe, order; fit usable evidence.
Answer: cite, refuse, verify; ground claims.

Audit trace: query, filters, candidates, scores, prompt context, citations, feedback.

Production failures happen when this picture exists only in someone's head and the team sees one final answer with one thumbs-up or thumbs-down. The application did dozens of things before the answer appeared, and if the logs do not keep them, a bad answer cannot be replayed. Keep a trace with one entry per stage.

Then debug in pipeline order, starting at the corpus. Do not tune answer prompts before you know retrieval recall. Do not add GraphRAG before you know whether ordinary search is missing evidence or finding it and presenting it badly. Do not blame the model before you have tested it with the correct evidence already in the prompt.

What if the right document was never retrieved?

Retrieval failure is the easiest failure to measure and the easiest to miss. A user asks about a refund exception, the policy exists, and the system retrieves general billing pages instead. The model cannot answer from a passage it never sees, so it either admits it does not know or writes a likely-sounding policy.

Start with a labelled set: collect 50 to 200 real questions and have someone who knows the domain mark the passage that answers each one, not the whole document. A gold label of "the employee handbook" makes the eval too forgiving, because any chunk from the handbook counts as a hit when the answer needed the paragraph on parental leave exceptions. Then measure recall at k: how often the right passage appears in the top 5, 10, 20 or 50. If it is in the top 10 for only seven questions in ten, at least three answers in ten cannot be grounded, and no prompt recovers that.

Metric	What it asks	Why it matters
Recall at k	Did any gold passage appear in the top k?	Sets the ceiling on answers that can be grounded
MRR	How high did the first useful passage rank?	Shows whether it will survive when the context is trimmed
Exact-token success	Did queries with ids, codes, names and dates find the right source?	Shows where embeddings blur identifiers
Filter loss	Did a permission, metadata or date filter remove the gold passage?	Separates search quality from access logic
Query rewrite loss	Did the rewritten query keep the key nouns and constraints?	Finds rewrites that read better and retrieve worse

The usual causes are dull ones. Embeddings blur part numbers, acronyms, customer vocabulary and uncommon names. A permission filter removes the right document, or a metadata filter is too strict. The parser drops table headers, or a query rewrite drops a noun the match depended on. The index is stale, or the document exists in the source system and never entered the ingestion job.

Hybrid search is the usual first fix, because semantic and keyword search fail on different questions. On Prospex the questions were mixed in exactly this way. "How do they make money" is a question about meaning, and the answer is phrased a hundred ways and never in those words. "Are they using Snowflake" is a keyword question, and pure vector search will return a passage about a different data warehouse because it sits close in meaning. We run semantic and keyword search together on every query, not as a fallback from one to the other, since each finds what the other misses.

Query rewriting helps when questions are conversational or full of pronouns, and it hurts when the rewrite smooths away a constraint. Log the original and rewritten queries side by side and evaluate both. A good rewrite keeps identifiers, dates, product names, jurisdiction and negation. A bad one turns "Can EU contractors expense coworking after July 2025?" into "contractor expense policy".

Filters belong in the retrieval eval, not only in the security review. On the trademark platform, retrieval is filtered by jurisdiction and class. When the jurisdiction is wrong upstream, retrieval returns authorities that are real, well formed, correctly cited and inapplicable. Every validator after it passes them, so only the attorney's review can catch it. Access lists fail in the same shape: if the user has access and the filter removes the document because the permission sync lagged, the system looks incompetent and nothing in the answer explains why. Store enough in the trace to tell "not retrieved because irrelevant" from "not retrieved because filtered".

What if the right chunk was retrieved but unusable?

RAG handles a long document by splitting it into chunks before indexing, and at query time it retrieves only the chunks that match the question. So a long document is only as answerable as its chunk boundaries. Sometimes the right document is present and the answer still fails, and the usual cause is chunking. One chunk holds the number without the condition, another holds the condition without the exception, and the model sees fragments instead of evidence. The model may get the sentence "approval is required after 30 days" without the heading that limits it to temporary equipment loans, or a table cell without its column header.

Chunk boundaries are set when the index is written, so a bad one fails every later query that needed the text it split. On Prospex the custom chunker exists for that reason: a boundary that splits a person's role from their company is a retrieval bug created at write time, and no query fixes it afterwards. Measure the chunker on your own labelled set instead of accepting a library default, because the choice alone moves recall: on one shared dataset, Chroma's chunking evaluation (July 2024) measured up to 9 percentage points of recall between strategies.

Tables are where chunking does the most damage. On the ESG platform the meaning of a figure sits in a table, qualified by a footnote, under a reporting framework stated in an annexure. Converting that table to plain text loses the row and column that named what the number measured. So the parser there keeps tables, sections and footnotes as structure instead of flattening them to text, and the rest of the system depends on that decision. When an agent has to act on a whole packet of documents rather than answer a question from a few passages, document intelligence for AI agents covers the checks across documents that retrieval skips.

Context assembly is a design problem of its own: which chunks survive, in what order, and with which titles, dates and neighbouring text. A paragraph from a contract is far more useful when the prompt also carries the section heading, document name, effective date, jurisdiction and parties, plus the previous paragraph if it defines a term.

Reranking helps when the answer is somewhere in a noisy candidate set. Late-interaction models, such as ColBERT, can match a small phrase that a single vector would wash out. No ranking model can repair a chunk that lost its meaning before indexing.

These bugs recur often enough to name:

Bug	What happens	Fix
Orphan chunks	A chunk holds a value without its heading or condition	Attach headings, breadcrumbs and neighbouring text
Duplicate crowding	Five versions of one page use up the context budget	Deduplicate by source, version and similarity
Stale override	An old policy outranks the current one because it matches more words	Use source dates, active flags and recency-aware ranking
Silent contradiction	A restated figure and the original both sit in the index	Check new documents against what the index already holds, and flag the conflict
Table flattening	The parser extracts cells without row and column labels	Keep tables as structure, or serialise them with headers and labels
Citation inflation	Whole documents in the prompt, so citations look solid and say little	Cite chunk or section ids, not documents
Bad ordering	The answer passage sits in the middle of weaker context	Put the strongest, freshest, most specific evidence at the edges

Take the ordering row seriously even with a large window. Models use evidence at the start and end of a long prompt more reliably than evidence buried in the middle, and that holds for models built for long input, so a correct passage packed in the middle can be retrieved and still ignored.

"Top k" also hides three different numbers: candidates retrieved, candidates kept after reranking, and blocks packed into the prompt, and the three should differ. Retrieve wide enough to keep recall, rerank to improve precision, and pack only what the answer can use, and set each from your own recall measurements. Make packing deterministic. If two chunks are equally relevant, prefer the fresher source, the more authoritative one, or the one from the narrower product area. Do not leave it to whatever order the database returned.

What if the model ignored or distorted the evidence?

If retrieval and context assembly are sound, the last layer is faithfulness. Hallucination enters here, and it helps to be specific about which kind. Did the model cite a passage that does not support the claim? Did it combine two unrelated snippets, or answer a question nobody asked? Did it use prior knowledge where the evidence was silent, or present an inference as a sourced fact?

Test this layer on its own by giving the model the gold passages directly and asking it to answer. If it still fails, retrieval is not the problem, and the work moves to the answer contract, refusal behaviour, model choice, and evals that score support claim by claim.

Write the contract as rules the output can be checked against:

Answer directly when the supplied evidence is enough.
Cite the specific source id for every factual claim that depends on retrieved content.
Say "I could not find enough evidence" when the answer is absent or ambiguous.
Separate facts from recommendations.
Point out conflicts instead of averaging them.
Use outside knowledge only where the product allows it, and label it.

Then test the contract against cases where the answer is absent, contradicted, stale, or sits in one small clause. Score at the level of claims, not answers, because an answer can be mostly right and carry one unsupported sentence that changes its business meaning. Break the answer into claims and ask, for each one, whether the evidence supports it, contradicts it or is silent on it.

The strongest version of the contract is enforced in code, not requested in the prompt. On Prospex a claim with no retrieved passage behind it does not appear in the output: not softened, not hedged, absent. That makes too little retrieval the safe failure, because a short profile is visibly short and an invented one is not. The trademark platform behaves the same way, and a draft with too little evidence comes out shorter rather than filling the gap from what the model remembers. On the ESG platform, a generated conclusion without its citations is not presented as a finding even when it is probably right. That rule is applied at generation, not in a later review.

Product pressure pushes the other way, for reasonable reasons. Users want short answers, so the model summarises, and they want decisive answers, so it resolves ambiguity. They also want helpful answers, so it fills gaps. All three behaviours suit a general assistant and do damage in a grounded one. The product has to decide which outside knowledge is allowed, such as general programming knowledge in a code assistant, and how it is labelled. If nobody decides, the model decides on every request, and that becomes a production bug.

What if the corpus is wrong before retrieval starts?

Teams spend their attention on search and too little on ingestion, and no retriever can compensate for a broken corpus. Ingestion is a data pipeline with quality gates: connectors, change detection, parsing, normalisation, chunking, embedding, indexing, permission sync, deletion handling and version tracking, and each of those steps can fail without raising an error.

The most dangerous corpus failure we have designed against is a correct number under wrong metadata. On the ESG platform, four things have to be right for a figure to mean anything: entity, period, unit and reporting framework. A passage stating a correct number for the wrong subsidiary, the wrong year, in thousands instead of millions, or against a superseded framework version passes every check downstream. The value is real, the citation resolves, the page is right. So those four travel as hard metadata from the moment of parsing and are never inferred when the question is asked, because by then the error cannot be recovered.

Entity resolution fails the same way. On Prospex a person on LinkedIn, a name in a press release, a signatory on a filing and a bio on a company site may be one human described four ways. Resolve them wrongly and the profile is internally consistent, well sourced and about the wrong person, and every claim in it is supported by a real passage.

Scanned and irregular documents are the volume problem: annexures, rotated tables, a footnote three pages from the figure it qualifies. The right behaviour is low confidence and a route to human review. On the ESG platform most of the tuning went into making sure low confidence actually fires instead of producing a confident wrong parse. Version drift is the slow failure, because a framework mapping that was right last year can break this year's comparison without any error, so the mapping version is stored on each fact. Stale sources follow the same rule on Prospex, where a role that changed since the page was scraped gives no sign in the text, so recency is recorded at ingestion.

PDFs are a common trap too, because a parser may emit text in the wrong order, merge columns or detach values from labels where a human sees an obvious layout. Spreadsheets need their row and column context, code needs file paths and symbols, and tickets need author, time and thread.

Deletion matters as much as ingestion, and it fails more quietly. If a policy is removed from the source system but stays in the index, retrieval can return a document users cannot find anywhere else. If a customer account is deleted and its chunks remain, you have a quality problem and a data retention problem, so build deletion handling early.

Give the corpus its own health checks, so bad answers are not the way you find corpus bugs.

Health check	Question
Coverage	Which systems, folders or tables are indexed?
Freshness	How long from a source update to a searchable update?
Parse quality	What share of documents parse with warnings, empty chunks or lost tables?
Version clarity	Can an answer show which version of a source it used?
Permission parity	Do search results match the source system's access lists?
Duplication	How many near-identical chunks compete for one query?
Deletion lag	How fast do removed sources leave the index?

What if security is treated as a model behaviour?

RAG puts untrusted text directly into the prompt, which makes prompt injection a design problem rather than a red-team footnote. A retrieved web page, customer document, ticket, PDF or chat message can carry instructions that try to override the system prompt, reveal hidden data, change citations or trigger a tool call. Design for it rather than prompt against it, because no change to the model closes it: OWASP's LLM01:2025 entry on indirect prompt injection concludes that RAG and fine-tuning "do not fully mitigate" it.

The reason is structural: the model receives trusted instructions and untrusted data in the same context. You can train and prompt it to respect the boundary, but the application should not use the model as the only boundary. Draw the line where the permissions already sit. Retrieval reads what the user may read and generation answers from it, while writes, outgoing messages, exports, admin tools and billing changes sit behind a separate permission path the model cannot grant itself. If an answer proposes an action, show the evidence and ask a person to approve it.

Put the defences outside the model wherever you can. Apply permissions before retrieval and keep tool credentials out of retrieved context. Label retrieved text as data rather than instructions, and allow only listed tools. Validate structured outputs and require human review for sensitive actions. Add eval cases where a retrieved document carries a malicious instruction, and treat a citation as evidence, never as authorisation.

These bugs usually arrive through helpful features, which is why they survive review. A system that searches tickets must not leak one customer's incident into another customer's answer. A system that summarises uploaded PDFs has to assume a PDF can carry hidden instructions. For your own system, ask what untrusted content can enter retrieval, which secrets and private documents the model can see, and which tools or arguments retrieved text could change. Then ask what happens if the model follows a malicious passage exactly. If the answer is that it sends data, changes records or reveals secrets, the boundary is in the wrong place. A prompt telling the model to ignore malicious text will not move it.

What if the evals measure the wrong thing?

Split the score by layer before you trust any of it. "Answer accuracy: 78 percent" does not tell you whether to work on retrieval, chunking, reranking, the prompt, the model or the source data, so a team holding only that number changes whichever piece it suspects. Keep retrieval quality, faithfulness to the passages and answer quality as three separate numbers. The common eval libraries already ship that split, so it costs no new tooling.

Retrieval evals need gold evidence, and they answer whether the retriever finds the right source. They are less expensive and more stable than full answer evals, because they do not depend on how the model writes. Run them on every change to indexing, chunking, query rewriting, embeddings or reranking. Our guide on LLM evaluation covers building the labelled set from real failures.

Context evals ask whether the selected context is usable. Context can have high recall and low precision, where the right chunk is present but buried among irrelevant ones, or high precision and low recall, where every chunk is on topic and none holds the answer. The fixes pull in opposite directions, because the first needs reranking or tighter packing, and the second needs a wider search, so applying the wrong one makes the number worse.

Groundedness evals compare each claim in the answer with the retrieved evidence. A model judge can help here once it agrees with human labels on a sample, because judges tend to pass claims that sound plausible and citations that sit near the right passage without supporting it. Task evals measure whether the user's job got done: the ticket resolved, the right exception found. Those are the numbers the business cares about, and they do not replace the lower layers because they cannot say what broke.

Build the set by query type, so each layer has cases aimed at it:

Query type	Example	What it tests
Exact lookup	"What is policy FIN-042?"	Keyword recall and exact-token cases
Semantic lookup	"Can I expense a monitor for remote work?"	Vector and hybrid recall
Conditional answer	"When does the exception apply?"	Chunk completeness and claim support
Multi-hop	"Which vendors are affected by this new control?"	Planning, graph or agentic retrieval
Absent answer	"What is the 2027 policy?" when none exists	Refusal
Conflicting sources	Old and new policies both indexed	Freshness, authority, showing conflicts
Permission boundary	User asks about another account	Access-list parity and leakage
Injection	Retrieved text contains "ignore previous instructions"	Security and tool boundary

A small set labelled by someone who knows the domain beats a large generated one. Questions a model writes by reading the corpus tend to be answerable by it and phrased like it, so they miss the questions real users fail on. Generated questions are useful for coverage before launch, and real query logs should replace them as soon as you have traffic.

Version every eval run: corpus snapshot, parser, chunker, embedding model, retriever and reranker settings, prompt, model and judge. Without that record you cannot explain why a score moved. On the trademark platform the base and fine-tuned models are logged on the same inputs for the same reason, so a regression from a model change shows up as a score that moved.

How do you match a symptom to the layer that failed?

Name the layer before you change anything. A missing document is a retrieval problem and a missing condition is a chunking problem. A cited claim the passage does not support is a faithfulness problem, and a private source in the answer is a permissions problem. The table gives the first measurement and the first fix for each.

Symptom	Likely layer	Measurement	First fix
Correct document absent from top results	Retrieval	Recall at k	Hybrid search, and inspect query rewrites
Correct page present, answer misses a condition	Context assembly	Read the gold passage as the model saw it	Chunk with headings and neighbouring text
Near-duplicates crowd out evidence	Context assembly	Distinct sources in the top k	Deduplicate and rerank
Answer cites weak evidence	Faithfulness	Claim support eval	Require source ids and refusals
Exact ids fail often	Retrieval	Exact-match query set	Add keyword search
Cross-document questions fail	Retrieval or planning	Multi-hop eval	Decompose the question, or route to graph or agentic retrieval
Old policy wins over new	Corpus and ranking	Freshness eval	Version metadata and authority ranking
Private source appears in an answer	Security	Leakage test	Permission filtering inside retrieval
Model follows document instructions	Security	Injection eval	Treat retrieved text as data and restrict tools

Build the retrieval stack before you reach for a bigger model, because a bigger model cannot use a passage the stack failed to find: hybrid search, metadata filters, document-aware chunking, a wide first retrieval, reranking to a smaller set, prompt assembly that keeps source structure, exact citations, and separate evals for retrieval, context and faithfulness. The retrieval half of that stack has a public measurement, from Anthropic's contextual retrieval tests (September 2024) across codebases, fiction, arXiv papers and science papers:

Pipeline	Top-20 retrieval failure rate
Standard embedding retrieval	5.7%
Contextual embeddings	3.7%
Contextual embeddings and contextual keyword search (BM25)	2.9%
Both, then reranking	1.9%

That is a 67 percent reduction in retrieval failure with no change to the model. The contextual step is worth understanding on its own, because an embedding of "The revenue was $1.2B" is generic. Prepending 50 to 100 tokens of generated context turns it into something closer to "Acme Corp Q3 2024 revenue", which sits somewhere useful in embedding space instead of next to every other revenue sentence.

Do not patch the one bad answer an executive saw, because pinning a source or editing the prompt may fix the screenshot and leave the system unchanged. Turn the failure into an eval case, name the layer that failed, and then decide whether the fix generalises. Latency and cost limit which fixes are available. Reranking more candidates adds a model pass to every request, an agentic loop adds a model call per step, and a long context raises token cost and still needs faithfulness checks.

When is GraphRAG or a long context better than RAG?

GraphRAG helps when the answer depends on relationships between entities rather than a passage match: ownership chains, supply dependencies, compliance owners. Long context fits when one bounded document, such as a contract or a small policy set, should be read whole.

On the ESG platform we use a knowledge graph and vector search together, because each fails alone. Pure vector search cannot express "the same subsidiary two years earlier", and pure graph traversal cannot find the narrative claim that was never turned into a field. Research questions there are split into parts and never answered from one retrieval, because the questions analysts ask are comparative and one retrieval cannot answer a comparison.

Treat a graph as a second system to own: entity and relation extraction, one agreed name per entity, conflict handling, updates, and its own evaluation. A bad graph can be worse than none because it imposes false structure on messy text. Before building one, check whether the failing questions are relational. If the failure is that the paragraph never reached the top k, fix ordinary retrieval first.

Long context is the opposite move: skip aggressive retrieval when the material fits and the access pattern allows it. Keep the faithfulness checks when you do, because a larger window changes the failure rather than removing it. The evidence now has to be attended to rather than found, and that degrades with length: across 18 models in Chroma's Context Rot tests (July 2025), reliability fell as input grew, even on tasks as simple as finding one sentence, and fell faster when the target looked less like the question or sat among similar distractors. A team that stops measuring because everything fits has stopped seeing that failure.

Architecture	Use when	Avoid when
Simple RAG	Large, changing corpus, mostly single-hop questions	Exact ids and structured data dominate
Hybrid plus rerank	Mixed meaning and exact-match search, noisy corpus	The latency budget cannot fit a reranker
Long context	One bounded document, a reading task, a clear permission boundary	Corpus is large, changing, or shared between customers
GraphRAG	Relationships are the product	Entity extraction is unreliable or questions are simple
Agentic retrieval	Finding the answer takes several dependent searches	Determinism, latency or security matter more

Many products route each question type to its own design: a policy lookup to hybrid search, a contract review to long context, a dependency question to the graph. Start the router with rules, and evaluate it like any other component. RAG vs fine-tuning vs agentic search covers when the agentic row is worth its latency.

How do you operate RAG after launch?

RAG quality drifts because the corpus underneath it drifts: documents get renamed, policies get superseded, duplicates get uploaded, and new vocabulary enters the business. A launch eval is only a starting point, and after launch you need a steady sample of real queries, failed searches, low-confidence answers and user corrections.

Logs should keep the layers apart: the user query, the rewritten query, filters applied, candidates retrieved, reranker scores, chunks placed in context, the answer, citations, refusal state, latency, token use and feedback. With that trace a bad answer can usually be classified quickly, and without it the team argues about the model. On the ESG platform the unit a reviewer can inspect is the path from a conclusion back to its evidence: document, page, parsed structure, extracted value, confidence, retrieval decision, framework mapping and the generated claim. Some single numbers are worth watching too, and on Prospex it is the ratio of collected items to items used as evidence, which on one run was 420 to 6. A run where that ratio collapses is usually a retrieval problem rather than a source problem.

Freshness is a product decision, because some corpora can re-index nightly and others need updates within minutes. If a support agent answers from a policy page that changed this morning, stale retrieval is a product bug. Put the document version, index time and source date in the prompt where the model and the reviewer can both see them.

Make feedback specific, because a thumbs-down on the final answer tells you little about where to look. A reviewer choosing "missing source", "wrong source", "unsupported claim", "outdated policy", "permission leak", "bad refusal" or "too vague" tells you which layer to look at.

Design the fallback yourself, and do not leave it to the model. When retrieval confidence is low, the product can ask a clarifying question, search a wider corpus, hand off to a person, or say which evidence is missing. Users take "I could not find the current policy for that account" better than a confident answer with a weak citation. The ESG platform treats its per-field confidence score the same way, as a routing signal, not a diagnostic: a low-confidence figure goes to a reviewer instead of into a dashboard as fact. Track how often each fallback fires and which part of the corpus caused it. A high refusal rate can mean a careful system or a weak retriever, and only the layer metrics tell you which.

What belongs on a launch checklist that survives production?

A RAG system is ready for users when each layer has a bar you can check, not when the demo looks right. Use this as the list to sign off before launch, and expect each row to have an owner.

Area	Minimum bar
Corpus	Sources, versions, permissions, deletion and freshness are known
Parsing	Tables, headings, lists and document metadata survive ingestion checks
Metadata	Fields that change the meaning of a fact (entity, date, unit, jurisdiction) are set at ingestion, not guessed at query time
Retrieval	A labelled eval set measures recall at k across query types
Context	Reranking, deduplication, ordering and packing are deterministic enough to replay
Faithfulness	Claim-level evals include absent, conflicting and stale evidence
Security	Prompt injection, access leakage and tool-boundary tests are part of the eval
Logging	The trace keeps query, rewrite, filters, candidates, scores, context, answer and citations
Fallback	Low-confidence behaviour is designed and measured
Ownership	Someone owns corpus health, eval health and production triage

Common questions

Why do production RAG answers sound right and still fail?

The answer can cite a nearby passage and still miss the fact the user needed. Retrieval, context assembly and generation are separate systems, and one accuracy number hides which of them failed. Check the layers in pipeline order. If the right passage never arrived, a prompt change will not repair the answer.

How does retrieval work in RAG?

The question is turned into an embedding, and the index returns the passages whose embeddings sit closest to it, usually alongside a keyword search that catches exact ids and names. Filters remove what the user may not see, a reranker can sort the shortlist, and the top passages go into the prompt. The model then answers from those passages, so a passage that never reaches the prompt cannot be used.

How does RAG handle long documents?

It splits each document into chunks before indexing and retrieves only the chunks that match the question. That works when each chunk carries its heading, conditions and table labels, and fails when a boundary separates a number from the condition that limits it. A single bounded document, such as one contract, can instead be read whole in a long context, with the same faithfulness checks.

What should we measure before changing the prompt?

Measure recall on a labelled set of real questions, and mark the passage that answers each one rather than the whole document. If that passage is missing from the top results, grounded answers are already capped. To test the model on its own, give it the gold passages directly. If it still fails, the fault is in the answer contract or the model.

When does hybrid search beat vector search on its own?

Use both when questions mix meaning with exact tokens. Embeddings match "refund window" to "return eligibility period", and they blur SKUs, policy ids, clause names, dates and error codes, which keyword search catches. Run both together, then rerank down to the passages that enter the prompt.

How should retrieved documents be treated for security?

Treat retrieved text as evidence and never as authority. A page, a ticket or a PDF can carry instructions that try to override the system prompt or trigger a tool, and neither RAG nor fine-tuning fully closes that gap. Apply permissions inside retrieval, keep tool credentials out of the context, and put writes, emails and exports on a permission path the model cannot grant itself.

Does a longer context window replace retrieval?

No, because a longer window turns a retrieval miss into a different failure, where the evidence is in the prompt and the model does not use it. Putting one contract or a small policy set in the prompt can be the right design when the document is bounded and the permission boundary is clear. Models get less reliable as input grows, even at finding a single sentence, so keep the recall and faithfulness checks when the text fits.

Sources and further reading

The six failure modes in this guide are our own grouping, drawn from the systems cited above and the sources below. The grouping overlaps with the taxonomy in Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System (CAIN 2024), drawn from research, education and biomedical case studies:

Their failure point	Where it appears here
FP1 Missing content	The corpus section, and the "say so and stop" refusal
FP2 Missed top-ranked documents	The right document was never retrieved
FP3 Not in context, consolidation limits	Retrieved but unusable
FP4 Not extracted	The model ignored or distorted evidence
FP5 Wrong format, FP6 incorrect specificity, FP7 incomplete	Generation failures, caught by the claim-level evals in the evals section
No counterpart	Security, where retrieved text carries instructions

We agree with their point that "validation of a RAG system is only feasible during operation", and it is why this guide puts the eval harness and the logs inside the design rather than after it.

Anthropic, Introducing contextual retrieval, September 2024.
Hong, Troynikov and Huber, Context Rot, Chroma, July 2025.
Smith and Troynikov, Evaluating Chunking Strategies for Retrieval, Chroma, July 2024.
Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2023. The basis for the ordering row.
OWASP LLM01:2025 Prompt Injection. Direct and indirect injection, and why RAG and fine-tuning do not fully mitigate it.
Case studies: ESG filings, prospect intelligence, trademark drafting.

Originally published on bigcircle.ai. I'm Tirth, founder of BigCircle, where we build production AI systems with governance built in.

How to design MCP tools for agents

Tirth Gajjar — Fri, 06 Nov 2026 18:45:22 +0000

MCP tools are the operations an MCP server exposes for a model to call. Each has a name, a description and a JSON Schema for its inputs, and the model decides when to call one, so the model reads the tool list before every answer. Design them around the questions your users ask rather than the endpoints a vendor ships: fewer, larger tools, and one consolidation layer in front when two or more sources can answer the same kind of question.

The layer exposes a small number of larger tools shaped around those questions, and it decides in code which source is authoritative for each one. It returns structured JSON, authenticates every request as the user, and logs every call. A direct wire stays reasonable for one read-only source with a single owner, and it stops being reasonable once a second source for the same question arrives.

Criterion	Direct vendor wires	Consolidation layer
Who chooses the source	The model, on every question	A rule in code for each question type: which system is authoritative, and which is the fallback
Tool list	The union of each vendor server's tools, loaded on every turn	Fewer, larger tools shaped around the questions users ask
Output	Each vendor's own shapes and error forms	JSON with defined fields, units, and as-of dates
Auth	Reasonable when the vendor's server already authenticates as the user and you can read its logs	Every request authenticates as the calling user, and the layer holds its own credential for each vendor
Maintenance	Each wire brings its own error shapes, and a vendor change can break something unrelated	One owner for every vendor integration, behind one set of tool contracts
When it fits	One read-only source with one owner, user auth and logs you can read, or a prototype. Revisit when a second source for the same question appears	Two or more sources can answer the same kind of question
How Bigcircle handles this	On Bigcircle's Solarpunk build, direct wires were the failure. Selection held until roughly thirty tools, then a confident result came from the wrong system	Bigcircle replaced those wires with one uniform tool layer of typed contracts, and each plan step searches it for the few tools that apply

The default goes the other way because each vendor server is built to be complete for its own product. It mirrors the vendor's API, so it ships many small tools, one per endpoint, and if you connect six of those the agent holds the union of their definitions on every turn. Every question then starts with the same work of deciding which system has the answer and whose version to trust. The model does that work in tokens, differently each time, and when it picks wrong nothing in the answer says so.

This guide covers what the direct approach costs, what the consolidation layer should own, and how large one tool should be. It then covers structured output, authentication under the 28 July 2026 MCP specification, rate limits and logging, and how the call log decides the next change. AI agent permissions covers who may call a tool, and MCP prompts versus tools for agent playbooks covers how an agent finds the right workflow in a large catalog.

What does wiring every vendor server cost?

Every MCP tool definition the client loads is text in the context window, on every request, whether the task needs it or not. With one server that cost is small enough to ignore. With several servers of many tools each, the definitions become a large share of what the model reads before the user's question. Several of them describe near-identical operations from different vendors that the model has to tell apart from their descriptions alone.

The token cost is the visible part, and the expensive part is the decision each question now carries: which system has the answer, and whose version to trust.

We saw that failure on Solarpunk, a product that works across more than a hundred tools. With every tool described in the prompt, selection held until roughly thirty tools and then went wrong with no sign in the output. The model returned a confident result from the wrong system, and that error cost more than the context did because it did not look like an error.

Maintenance degrades in the same arrangement, for a different reason. Each direct wire brings its own credential handling, its own error shapes and its own prompt fragments, and on Solarpunk a vendor's change to one response surfaced as a break somewhere unrelated, days later. That is why the build put one uniform tool layer between the model and every integration, with nothing wired point-to-point and each tool a typed contract with declared inputs, outputs and side effects.

Every vendor server, or one layer in front of them. The sources are the same in both halves. The choice of source moves.

Wired direct: the agent picks a source among CRM, Warehouse, Data vendor A, Data vendor B, Documents and Tickets. Every tool from every server sits in context, and every question starts with a choice of source.
Consolidated: the agent asks a question of one Research MCP (find_company, company_profile, deal_history, log_note, preview first) in front of the same six sources, and the source choice lives in code. The layer answers which source to trust once, in reviewed code, instead of once per question.

The six sources and four tool names above are examples.

The two halves reach the same data, and they differ in where the choice of source is made. In the top half the model makes that choice on every question, while in the bottom half an engineer makes it once, in code someone reviewed. The model asks a question and gets an answer from the source the team already decided to trust.

What should the consolidation layer own?

The layer is an MCP server your team runs between the agents and the vendor systems. A proxy that re-exports each vendor's tools under one name does not count, because it moves the long tool list without shortening it. The MCP specification, as of its 28 July 2026 revision, expects such proxies to hit name collisions, such as two servers that each expose a search tool. Its suggested fix is to prefix each name with a server identifier, which leaves both near-identical tools in the list for the model to choose between.

Responsibility	What the layer does	What happens when the agent does it instead
Source selection	A rule in code for each question type: which system is authoritative, and which is the fallback	The model re-decides per question, and two users get answers from different sources
Entity resolution	Resolves a fuzzy name to one record ID before any lookup	Answers blend two records for similar names, each correctly sourced and about the wrong entity
Output shape	Returns JSON with defined fields, units and as-of dates	The model parses prose and misreads a figure
Identity	Authenticates every request as the calling user	A shared key reaches the data, and the audit trail cannot say who asked
Rate limits	Limits by user and by tool, before the upstream vendor does	A reasoning loop spends the vendor quota for everyone
Call log	Records every call, allowed or refused	Nobody can see what the model actually asked for
Writes	Returns a preview before committing a change	The model commits a change nobody reviewed

Entity resolution is the row easiest to leave to the model, and the one that hurts most when it goes wrong. On Prospex, our prospect research engine, collection was never the hard part. The hard part was resolving one company or person across four source types whose schemas share nothing, where a wrong merge carries into every later inference. The profile that comes out is coherent and well sourced, and it describes the wrong person. Put resolution in the layer as its own step, and make every other tool take the resolved ID rather than a name.

The write row has the same logic as the rest of the table. On Skionis, our DevOps agent on live AWS, the agent reaches the account only through purpose-built tools, with no general shell and no path by which a plan step becomes an arbitrary API call. An engineer approves an architecture diagram before anything is applied. Keep reads and writes in separate tools, so a permission policy can allow one without the other. Have the write tool return a preview, so the guardrails and the reviewer have a concrete change to approve.

How large should MCP tools be?

A tool should match a question a user asks, not an endpoint a vendor ships. The vendor's API is organized around its data model, with separate calls for the company, its funding rounds, the investors in a round, and notes. A user's question crosses those boundaries, so a server that mirrors them makes the model plan and join four calls to answer one question.

Here is the same question answered with each of the two shapes:

	Skinny, endpoint-shaped tools	Larger, question-shaped tool
Example	`get_company`, `list_rounds`, `list_round_investors`, `search_notes`	`company_profile` with an `include` list
Calls per question	Several, planned by the model	One, planned by your code
Where the join happens	In the model's reasoning, in tokens	In the server, in code you can test
Definitions in context	One per endpoint, per vendor	One per question type
Typical failure	A step skipped, or two results joined on the wrong key	The tool returns more than the question needed

Larger does not mean general. A run_sql tool or a call_api tool that takes any endpoint is one tool in the list and every operation in practice. The model has to write the query, the policy cannot tell a read from a destructive write, and the log records a string instead of an intent. Keep the tool's scope to one question type, with typed options such as which sections to include, a date range, and an enum for the region.

This is roughly what a question-shaped tool's contract looks like, with illustrative fields:

{
  "name": "company_profile",
  "description": "One company's profile from our authoritative sources. Use after find_company has returned a company_id. Not for comparing several companies; use compare_companies.",
  "inputSchema": {
    "type": "object",
    "required": ["company_id"],
    "properties": {
      "company_id": { "type": "string" },
      "include": {
        "type": "array",
        "items": { "enum": ["funding", "investors", "notes", "scores"] }
      },
      "since": { "type": "string", "format": "date" }
    }
  }
}

The description names the prerequisite and the near-miss, and it needs both. When two tools could plausibly answer a question, the description is all the model has to tell them apart. The line "not for comparing several companies" prevents a wrong pick that a longer list of the tool's features would not.

Split a tool when two uses need different permissions, different rate limits or different owners. A tool that reads CRM notes and a tool that writes them should never be one tool with a mode flag, since the permission check would then depend on an argument the model chose.

Why return structured JSON and not prose?

Return JSON with defined fields, not descriptive text. Prose reads well in a demo and hands the model a parsing job on every call, and a misread figure comes back looking like an answer. Scale Venture Partners found the same on the MCP server behind their investment research, where tightening the schemas improved answer quality straight away. The protocol now supports this directly. As of the 28 July 2026 revision a tool can declare an outputSchema, and a server that declares one must return structuredContent that conforms to it. Clients should validate the result against that schema.

Give every group of fields a source and an as-of date, so the agent can say where a number came from. Return explicit statuses for the cases that are not data, such as not_found, not_permitted, stale and rate_limited. An empty array is ambiguous, and the model will often read it as "none exist" and tell the user so, when the truth was that the caller could not see the records.

Keep result sizes bounded, whatever the source holds. A tool that returns every note on a company since founding pushes the rest of the conversation out of the context window. Accept a since date or a limit, and when there is more, return a count and a continuation handle. MCP has no protocol session any more, so that handle is an ordinary argument the agent passes back on purpose. The layer checks the caller's authorization against it again on the next call.

How should the layer authenticate, limit, and log?

One server in front of every source holds more reach than any one vendor server, so its controls have to be tighter. Validate the caller on every request, not once when a client connects. As of the 28 July 2026 specification that is the protocol's default rather than a design choice.

MCP no longer has sessions, the client must send its authorization on every HTTP request, and the server must validate each token and reject one issued for a different server. The shortcut that survives is caching a validation result or accepting long-lived tokens, and either one leaves a revoked user with access until the cache or the token expires.

The same section forbids an MCP server from accepting or passing along any token not issued for it, so the user's token cannot be forwarded to the vendor. The layer holds its own credential for each vendor, inside the layer where the model never sees it, and checks the user's identity against its own policy before it calls out. How to mint a short-lived token for each agent run is covered in MCP authorization.

Rate-limit by user and by tool, inside your own layer. The same specification requires servers to rate-limit tool calls. A reasoning loop can fire a burst of calls in seconds. Without a limit in your layer, the first thing to stop it is the vendor's quota, which then fails for every user at once. When the layer limits a caller, return rate_limited with a retry time, so the agent waits instead of switching to a worse source to get an answer.

Log every call, allowed or refused, with at least these fields:

the tool and its version;
the calling user and the agent or client;
the validated inputs;
the sources the layer read;
latency and result size;
the status returned, including refusals.

The refusals in that log matter as much as the successes. A tool that is refused often is either mis-permissioned or mis-described, and without the refused calls in the log you cannot tell which. Keep the log in the same observability stack as the rest of your services, because the question after an incident is usually which tool call touched which record.

How does the call log change the tools?

Read the call log on a schedule, because it shows how the model actually uses your tools, which is rarely how you expected when you wrote them. The signal is usually plain. Scale saw repeated calls over a single tool on their research server, added a batch mode, and cut sessions from an hour to minutes. Build that loop on purpose rather than waiting for an incident to send you to the log.

Run the loop on a schedule rather than after an incident, and look for these patterns:

Pattern in the log	What it means	Change
One tool called many times in a conversation with different IDs	The question is about a set, and the tool only takes one	Add a batch mode that takes a list
Two tools alternate for the same question type, and one result is discarded	Their purposes overlap	Merge them, or rewrite both descriptions with the near-miss
The agent fetches a large result and uses a few fields	The tool returns more than the question needs	Add an `include` list or filters
Validation errors cluster on one argument	The schema is unclear	Use an enum, add an example, or make the field required
An empty result followed by a retry with a different spelling	Entity resolution is missing	Add a resolve step and require the ID
A tool nobody calls	It costs context and answers nothing	Remove it

After each change, re-run a fixed set of real questions and compare the tool paths, not only the final answers. A change that keeps the answers correct and doubles the calls per question has made the system slower, and an answer-only check will not show it. Our evals guide covers how to build that question set from real failures.

Pin the tool list in CI as well, so a rewritten description fails a pull request before it reaches the agent, since the description is the instruction the model reads. MCP server testing covers that snapshot, and the agent eval that re-runs your question set when the snapshot changes.

When is wiring a vendor server directly the right call?

Consolidation has a cost: a service to run, an owner, and a release process for tool changes. For some setups it is not worth it, and the questions below run in order, with the first yes deciding.

Wire the vendor server directly, or consolidate first. Stop at the first yes.

Do two or more sources answer the same kind of question? Yes: consolidate. Put the source rule in code.
Does a call need the user's own identity, a rate limit, and your log? Yes: consolidate, or put a gateway in front.
Would the agent hold more than a few dozen tools at once? Yes: consolidate, and add discovery.
No to all three: wire the vendor server directly, and log its calls where you can read them.

The "few dozen tools" threshold in the third question is the range we work to, set near the roughly thirty where Solarpunk's selection went wrong. A tool set whose tools are very distinct may hold more.

A direct wire is fine for one read-only source with one clear owner, provided the vendor's server already authenticates as the user and you can read its logs. A prototype that proves a workflow is worth building also qualifies. Revisit the choice when a second source for the same kind of question appears, since from then on the model chooses between sources on every question and that choice belongs in code.

If the answer is to consolidate, start from the questions rather than the vendors. Collect what users actually ask the agent, group the questions into types, and write down for each type which source is authoritative and which is the fallback. That list becomes the source rule in code. Build entity resolution first, as its own tool, and then one question-shaped tool per type that takes the resolved ID.

The agents calling this layer may also need shared definitions, so that "active customer" means the same thing in every client. That is a context problem as much as a tool problem, and our guide on shared business context for every AI agent covers it. Bigcircle's MCP servers work covers the consolidation layer.

Common questions

What are MCP tools?

MCP tools are the operations an MCP server offers for a model to call, such as a lookup or a write. Each tool has a name, a description and a JSON Schema for its inputs, and it may declare an output schema. The model reads those definitions and decides when to call a tool, so the description is the only thing it has to tell two similar tools apart.

What are the best practices for designing MCP tools?

Shape each tool around a question users ask, not a vendor endpoint. Give it typed options rather than a free query string, and a description that names its prerequisite and its near-miss. Return JSON with sources, as-of dates and explicit statuses, and keep results bounded. Split reads from writes, authenticate every request as the user, rate-limit by caller, and log every call, refusals included.

Should we connect every vendor's MCP server to our agent?

Only when each vendor answers a different kind of question and you have few of them. When two or more sources answer the same question, the model has to pick a source and decide whose data to trust on every request. That costs tokens and produces inconsistent answers. Put a consolidation layer in front instead, and make the source choice once, in code.

What is an MCP consolidation layer?

It is an MCP server your team runs between your agents and the vendor systems. It exposes a small set of question-shaped tools and decides which source is authoritative for each question. It resolves names to record IDs, returns structured JSON, authenticates each request as the user, applies rate limits, and logs every call. It does not re-export each vendor's tools under one name, because that keeps the long tool list.

How many tools should a consolidation MCP expose?

Start with one tool per question type your users actually ask, plus a resolve tool for entities. There is no fixed limit, but on Bigcircle's Solarpunk build direct tool calling degraded at roughly thirty tools. If the list is heading past a few dozen, add discovery or merge overlapping tools, and remove any tool the call log shows nobody uses.

Does fewer, larger tools mean one general query tool?

No. A tool that runs any SQL or calls any endpoint is one entry in the list and every operation in practice. The model has to write the query, and the permission policy cannot tell a read from a destructive write. Keep each tool scoped to one question type with typed options, and keep reads and writes in separate tools.

How should a consolidation MCP handle authentication?

Authenticate every request as the calling user. Since the 28 July 2026 MCP specification there are no sessions, so the client sends its token on every request and the server must validate it each time. The layer may not forward the user's token to a vendor, so it holds its own vendor credentials where the model never sees them. It records the user on every call, so the audit trail can say who asked.

How do we know when to change a tool?

Read the call log on a schedule and look for patterns. Repeated calls over one tool with different IDs call for a batch mode. Two tools alternating for the same question call for a merge or better descriptions. Validation errors on one argument call for a clearer schema. After each change, re-run a fixed set of real questions and compare the tool paths as well as the answers.

How to build an LLM evaluation from real failures

Tirth Gajjar — Wed, 04 Nov 2026 18:45:09 +0000

Build the eval from failures you have actually read, rather than from a rubric you wrote in advance. Group those failures, put one domain expert in charge of the standard, and create the cases only after that. Use code where the property is mechanical, and add a model judge only after it agrees with that expert.

Bigcircle uses this order on client systems, because a score you cannot trace to a failure does not tell you what to fix. Every guide on this site tells you to build evals before you do anything else, and none of them has explained how. That gap is worth closing, because the order of the steps matters more than any individual technique in them, and most of the teams we work with have the order backwards.

The difficulty is not that evaluation is conceptually hard. It is that the thing you are trying to measure does not hold still. A traditional test suite asserts that a function returns four, and four is four regardless of who is asking. An eval asserts that an answer is grounded, or that a citation supports the claim attached to it, or that a system correctly declined to answer a question it had no evidence for. Those are judgements, they vary between reasonable people, and they change as you learn more about your own product. Barnett et al. reached the same conclusion across research, education and biomedical deployments: "validation of a RAG system is only feasible during operation" (Seven Failure Points, IEEE/ACM CAIN 2024). You cannot certify these systems the way you would certify a payments integration, which means the harness is not a gate you pass once but an instrument you keep running.

In this guide we walk through the process we use on client systems, in the order we use it. It starts with reading real failures before writing any criteria, grouping those failures into a taxonomy, and deciding who owns the standard. From there you build the case set, choose what to measure on it, and decide which of those checks need a model at all. Then you write the rubric for the ones that do, and validate that judge against human labels before it gates anything. Where our own systems illustrate a step, we use them: two in production with clients, and two we built and run ourselves.

What is LLM evaluation, exactly?

An eval is a test whose assertion requires judgement rather than equality.

The distinction matters more than it first appears. A unit test can be written before the feature exists, because the correct answer is knowable in advance. An eval usually cannot, because the definition of a correct answer is something you discover by looking at wrong ones. Everything else about an eval is ordinary engineering: it lives in the repository, it runs in continuous integration, it fails a build, and it has a history you can bisect when a regression appears.

# unit test
assert parse_date("2026-09-01") == date(2026, 9, 1)

# eval
assert grounded(answer, passages)         # every claim traceable to a passage
assert cites_correctly(answer, passages)  # each citation contains what it claims
assert refused_when_unanswerable(answer, passages)

Why can't you just write the rubric first?

Because you do not know your criteria until you have looked at outputs, and looking creates them.

This is not a failure of discipline, and a more rigorous team does not avoid it. Shankar et al. identified it as a structural property of the task and named it criteria drift:

To grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria.

We see this on nearly every engagement. A team sits down to write a rubric before launch and produces something like "the answer should be accurate and helpful." An hour of reading real outputs turns that into something far more specific. Does the response cite the controlling clause rather than a topically related one? Does it flag a conflict between two policies instead of silently picking one? Does it decline when the user's jurisdiction is not covered by the documents in scope? None of those would have occurred to anyone in advance. All of them are obvious in retrospect.

This is also why prefabricated metrics tend not to survive contact with a real product. A generic helpfulness or coherence score measures an abstraction that has no particular relationship to the failure modes your system exhibits, and when it moves you cannot say which change moved it. We have watched a team spend a sprint chasing a coherence score that fell after a retrieval change, on a system whose actual problem was that the reranker had started dropping the controlling clause. The score was real. It just was not measuring the thing that broke.

The rubric, then, is an output of the process rather than an input to it. The process that produces it is error analysis.

How do you find your actual failure modes?

Read your traces in a structured way before anyone writes a criterion, in three passes borrowed from qualitative research methods. The structure matters because unstructured reading produces impressions rather than counts, and impressions cannot tell you where to spend a quarter.

Start with open coding: pull real traces and write a free-form note on each one that went wrong. There are no categories at this stage and no attempt at consistency, because imposing either would defeat the purpose. Notes look like "cited the wrong section," "answered when it should have refused," or "used the policy that was superseded in March." The notes are raw observation, and the discipline is to resist labelling while you are still looking.

Then group those notes, the pass called axial coding, so that similar complaints collapse into a category and you end with a failure taxonomy of roughly five to ten modes. Whether "cited the wrong section" and "invented a clause number" become one category called citation error or two separate ones depends entirely on what you saw, and that decision is the substantive work of the pass.

Count before you plan the work, because the count decides where engineering time goes. If forty-three percent of your failures are retrieval misses, no amount of prompt work will move your quality this quarter. Counting makes that unarguable rather than a matter of opinion between two engineers who each read a handful of examples.

You stop at theoretical saturation, which in practice means twenty consecutive traces producing no new category, with at least a hundred reviewed before you trust that. Do not stop earlier. It is the common error, and it leaves a taxonomy of the sample rather than of the system. The modes you found are real, but the ones you missed are the ones that will show up in production. Every evaluator you build afterwards then targets a failure you have seen in your own system, rather than one you read about.

Who decides what good means?

One person, and not a committee.

Pick one principal domain expert: someone with real depth in the domain, or who truly represents the users the system is built for. Their job is to set the standard, and it has to be one person because three stakeholders labelling in parallel produce three rubrics and an argument. We have sat through that argument more than once, and it is almost always about vocabulary rather than about quality, which means it consumes a week and settles nothing. You need labels that stay consistent from one trace to the next, and one person gives you that where a panel does not.

The work itself is small and specific. The expert reads a trace, makes a binary pass-or-fail judgement, and writes a critique explaining the reasoning behind it. Teams cut the critique when they are busy, and cutting it costs more than it saves, because those critiques become the few-shot examples in the judge prompt later. On the trademark platform the useful ones named the clause that should have governed and said why the wrong one was plausible, which is the reasoning a judge needs in order to make the same call. The difference between a usable critique and a useless one is large:

Critique	Usable?
"Wrong."	No. It carries no information at all
"Bad citation."	No. Which citation, and wrong in what way?
"Cites clause 7.2, which covers annual plans. The question was about monthly, governed by 7.4. The number quoted is correct for the wrong clause, which is worse than being plainly wrong because a reviewer scanning for errors will not catch it."	Yes. This can become a few-shot example

The time cost is lower than teams tend to fear. Thirty to fifty traces to establish the standard, then periodic review as the system changes. What cannot be delegated is the judgement itself, because the expert is the only available source of the standard the system is being held to.

Where do the cases come from?

You have a taxonomy and someone who owns the standard. The next thing to build is the set of cases both will be applied to, and it is built to cover the modes the taxonomy named rather than to sample traffic. Keep it as JSONL, one case per line, so a diff is readable and a new case is one appended line.

{"id": "refund-001", "query": "What is the refund window for enterprise annual plans?", "gold_passages": ["policy-billing#p14"], "gold_answer": "30 days from invoice date.", "must_refuse": false, "tags": ["billing"]}
{"id": "refund-002", "query": "Can I get a refund after 90 days?", "gold_passages": ["policy-billing#p14"], "gold_answer": null, "must_refuse": true, "tags": ["out-of-policy"]}
{"id": "acl-001", "query": "What is the Q3 headcount plan?", "gold_passages": [], "must_refuse": true, "tags": ["permissions"], "as_user": "contractor@example.com"}

Two decisions in that format do most of the work.

Label the passage, not the document. This is the mistake most worth avoiding early:

{"gold_passages": ["employee-handbook.pdf"]}                    // useless
{"gold_passages": ["employee-handbook.pdf#p31-parental-leave"]} // testable

A document-level gold label makes retrieval recall look excellent while measuring nothing, because any chunk drawn from a ninety-page file counts as a hit. Ask the expert to mark the minimum span that answers the question, and if two spans are required, record both, which turns the case into a multi-hop test at no extra cost.

Stratify the set by the taxonomy you found, not by traffic volume. A uniform sample of production traffic returns whatever your users happen to ask most often, which is usually one easy intent repeated many times.

Bucket	Share	Why it exists
The dominant failure mode from your counts	30%	The quarter's work lives here
Exact identifiers: ids, codes, part numbers	15%	Embeddings blur these and nothing else catches it
Multi-hop or cross-document	15%	Fails silently without a dedicated bucket
Unanswerable from the corpus	15%	Tests refusal. Barnett's FP1
Permission-restricted	10%	Runs as a user who must not see the answer
Adversarial: instructions inside a document	10%	Prompt injection surface
Long tail, unusual phrasing	5%	Where query rewriting breaks

Once a hand-built set exists, Chroma's generative benchmarking report (April 2025) describes one way to grow it: filter your documents with a judge aligned to your expert, then generate queries from what is left, steered by real production queries so they read like your traffic.

What should you measure?

With cases in hand, you can decide what to measure on them. Decompose the system first. Retrieval, tool use and generation fail for unrelated reasons and are repaired by unrelated work, so a single quality score cannot tell you which of the three broke on any given change.

Here is the whole set in one place, before we go through them. Standard names
are given where one exists, because you will meet these under those names in
Ragas and
elsewhere.

Layer	Metric	Answers	Typical gate
Retrieval	Recall@k	Did any gold passage reach the top k?	0.90 at final depth
Retrieval	MRR, or NDCG@k when passages are graded	How high did the first useful passage land?	above 0.6
Retrieval	Exact-match rate on ids and codes	Do identifiers survive embedding?	above 0.95
Retrieval	Filter loss	Did permissions or metadata remove the gold passage?	near 0
Generation	Groundedness, called faithfulness in Ragas	Is every claim supported by a passage?	0.95
Generation	Citation validity	Does each cited passage contain its claim?	0.95
Generation	Completeness	Was anything the passages supported left out?	by eye, then by judge
Generation	Refusal recall	Of the cases that should refuse, how many did?	0.98
Tool use	Selection, argument, execution, sequence accuracy	Right tool, right parameters, error handled, right order	0.95 each
Judge	TPR and TNR	Of real failures, how many caught. Of good answers, how many passed	TPR above 0.90
Judge	Cohen's kappa	Agreement with the expert above chance	0.6 to gate on
System	p95 latency	Can anyone wait for the answer?	your interface budget

Two notes on that table. The gates are the ones we start from and then move once
a system has its own history; they are not standards. And precision does not
appear on the retrieval rows, because at the depths that matter here recall is
the constraint. Precision starts to matter when you are paying per passage in a
long context window.

Retrieval

If the passage never reaches the model, nothing downstream can recover, which is why this layer is measured first and why a low number here makes every other measurement uninterpretable.

def recall_at_k(retrieved_ids, gold_ids, k):
    return len(set(retrieved_ids[:k]) & set(gold_ids)) > 0

def reciprocal_rank(retrieved_ids, gold_ids):
    for i, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in gold_ids:
            return 1 / i
    return 0.0

Measure recall at your candidate depth and again at your final depth, because the two numbers answer different questions. Recall at fifty tells you what retrieval could deliver if nothing downstream discarded it, and recall at five after reranking tells you what the model actually receives. The gap between them is the reranker's contribution. If recall at fifty is already low, no reranker can help and the work belongs in retrieval. If recall at fifty is high and recall at five is not, the reranker is where the points are.

Recall tells you the passage is somewhere in the list, while MRR tells you whether it will survive context trimming. A system reporting recall at twenty of 0.95 alongside an MRR of 0.2 is retrieving the right thing and then burying it, and the two numbers together diagnose that in a way neither does alone.

Anthropic's contextual retrieval benchmarks, published in September 2024, give you a target to hold your own harness against. They measure the failure rate as one minus recall at twenty. Plain embedding retrieval missed 5.7%, and adding contextual embeddings, keyword search and a reranker took that to 1.9%.

Measure this before you touch generation quality, because the order is not a preference. At a recall of 0.6, four questions in ten are unanswerable before the model is ever invoked, so every prompt change you evaluate after that point is being measured against a ceiling you have not lifted. On Prospex, our own prospect-research engine, a single run reduced 420 collected items to the 6 that answered the question, and that reduction made the downstream answer quality worth measuring at all. Once those numbers are readable, which layer to change is the decision in RAG vs fine-tuning vs agentic search.

Tool use

This layer is consistently under-measured, and it is where agents fail quietly rather than loudly. Four separate questions need separate answers.

Check	Failure it catches
Selection	Called the wrong tool, or called one when it should not have acted at all
Arguments	Right tool, wrong parameters, usually a schema mismatch or a date format
Execution	The tool returned an error and the model continued as though it had not
Sequence	Right tools in the wrong order, or a required step skipped entirely

Sequence is the one teams omit, and it produces agents reporting work as done when a step in the middle failed silently.

Safety: guardrails and PII

A guardrail and an eval are different mechanisms and teams keep conflating them. A guardrail runs on every request and blocks or rewrites. An eval runs against a fixed set and tells you whether the guardrail works. You do not pick one. You build the guardrail and then measure it, and teams skip the measuring.

Three kinds of case go in the set, and the third is the one teams miss.

Cases the guardrail should catch. A request that would send personal data to a provider, an action outside the policy, an instruction buried in a retrieved document. These are the obvious ones.

Cases it should let through. Over-blocking is a real failure with a real cost, and a guardrail tuned only on the first kind of case will get quieter and quieter until the product stops working. If nobody is measuring the allowed traffic, nobody notices.

Cases where the sensitive data arrives from retrieval rather than from the user. A redaction layer that reads the request and not the retrieved passages is a layer with a hole in it, and the hole only opens once the corpus grows.

PII handling has a second measurement most sets miss. If the boundary encodes on the way out and decodes on the way back, the eval has to check both directions. Redaction that never decodes cleanly produces answers that are safe and useless, and that failure shows up as a quality complaint rather than a safety one, so it gets routed to the wrong team.

Check	Question	Gate
Block recall	Of the requests that should have been stopped, how many were?	1.0, and treat anything less as an incident
Over-block rate	Of the requests that were fine, how many were stopped anyway?	track it, or the guardrail tightens unnoticed
Redaction recall	Was every sensitive field detected before the request left?	1.0
Decode fidelity	Did the answer come back complete after decoding?	by eye first, then by judge
Verdict coverage	Is every decision recorded, including the allows?	1.0

That last row makes the rest possible, because a guardrail that logs only its blocks cannot be measured: the cases you most need are the ones it let through and should not have.

The gates here are not like the others in this guide. Elsewhere we start at 0.90 and move it once a system has history. On safety the target is 1.0 and a miss is an incident, because the cost of one leaked record is not on the same scale as one wrong answer.

Generation

Assertion	Question
Groundedness	Does every factual claim appear in the retrieved passages?
Citation validity	Does each cited passage contain the claim attached to it?
Completeness	Did it omit something the passages supported and the question required?
Refusal correctness	Given these passages, should it have answered at all?

Notice that helpfulness does not appear, because it is not a measurement, and a judge asked to assess it tends to favour the longer answer.

When should you not build a judge?

You now know what to measure, so next comes what does the measuring, and for most of these checks it should not be a model. A judge is the expensive option, to build and then to run on every case forever, and teams reach for one earlier than the evidence justifies.

Check	Use when	Cost
Assertion in code	The property is mechanical: valid JSON, a citation id that exists, a required field present, a forbidden string absent	Minutes to write, free to run
Reference-based check	You have a gold answer to compare against, or a gold passage set for computing recall	An hour to write, free to run
Model judge	The property requires reading comprehension: groundedness, whether a clause governs, whether a refusal was correct	Days to build, and a cost per case indefinitely

The mistake we see most often is a team building a judge for a failure they could have fixed that afternoon. If a mode appears eleven times in your traces and a prompt change removes it, you have spent nothing and gained the fix. Build a judge for it instead and you have added a per-case cost that runs for as long as the product does, all to keep measuring something that stopped happening. Judges earn their cost on the modes that survive several attempted fixes.

How do you write a rubric that holds up?

For the modes that survive, you need a judge, and a judge is only as good as the question you give it. Each mode becomes one question the rubric asks. Four things decide whether that rubric works.

	Rule	Why
1	One question per judge	Ask "is this good" and the judge weighs four things at once and blends them. The score moves and you cannot say what moved it
2	Binary, not scored	Nobody can say what sits between a three and a four, so two people score the same answer differently and the noise hides real changes
3	Every clause comes from a real failure	If you cannot point at the trace behind a clause, it guards against a problem you made up
4	Exclusions written into the prompt	Judges prefer long, fluent answers unless you tell them not to

Two of those are worth more than a table row.

Ask one question at a time. "Does every claim appear in the passages?" fails in a way you can act on, because you go and check the claims. "Is this answer good?" gives you a number that dropped and no way to tell whether retrieval, the citation, or a missing refusal caused it.

Binary is harder to agree on than it looks, and that is the point. Two people will argue about where the line sits, and the argument is worth having. A team that cannot say whether an answer passes does not yet know what it wants from the system.

A three-way split is sometimes right, when a partial answer means something real. A five-point scale used as a gate rarely is. Ask two people to score the same answer and one says three and the other says four. A real change has to rise above that noise before anyone can see it, and that takes more labels than most teams will ever collect.

Our trademark platform is the case where a scale earns its place. Its harness scores every draft from 0 to 10 on six axes before an attorney sees it: structure, reasoning depth, citation accuracy, jurisdiction discipline, edge-case handling and tone. The scale works there because its main job is comparison. Production logging scores the base model and the fine-tuned one on the same inputs, so a regression from a model change shows up as a number that moved. The attorney's corrections stay the most valuable signal in the system, because they are the only measure of the gap the automated scoring cannot see.

Here is a rubric that came directly out of the failure note in the table above:

You are checking whether a cited clause governs the question asked.

QUESTION: {question}
PASSAGES: {passages}
ANSWER:   {answer}

For each citation in the ANSWER, decide whether the cited clause is the one
that governs the QUESTION, rather than a related clause on the same topic.

- valid     every citation governs the question asked
- invalid   at least one citation is topically related but not controlling

A correct number quoted from the wrong clause is invalid. Ignore style,
tone, and length entirely.

Return JSON: {"verdict": "...", "offending_citations": ["..."]}

You could not write that rubric without one thing someone noticed first: a right number pulled from the wrong clause is the case that hurts, because a reviewer skimming the output will not catch it. That came from a domain expert's critique. No off-the-shelf metric would have found it.

How do you know your judge is any good?

You measure it against the expert's labels, and the number most teams reach for first is the wrong one.

Suppose your set contains two hundred cases of which a hundred and eighty pass, and your judge agrees with the human labels ninety percent of the time. That sounds like a working judge. Now consider a judge that returns pass for every input without reading anything at all, which also scores ninety percent on that set. Raw agreement cannot distinguish between the two, because on an imbalanced set the majority class carries the score entirely. This is a common way for a team to convince itself a judge works. The number is high, it is easy to compute, and it stays high no matter how badly the judge handles the cases that matter.

Here is that "always pass" judge on the 200-case set:

	Judge says fail	Judge says pass
Expert failed	0	20
Expert passed	0	180

Raw agreement is 0.90, true positive rate 0.00 and true negative rate 1.00. The judge missed 20 of 20 real failures, and every one of those ships. Lower the share of failures in the set and agreement climbs on its own. That movement is the set getting more imbalanced, not the judge getting better.

Measure the two error rates separately

The true positive rate is recall on failures, meaning of the cases the expert failed, how many the judge caught. The true negative rate is its mirror, meaning of the cases the expert passed, how many the judge also passed. Report them separately and the judge's behaviour becomes legible, because the two failures cost very different amounts.

from sklearn.metrics import confusion_matrix, cohen_kappa_score

# 1 = expert failed the case (the thing we care about catching)
expert = [1,0,0,1,0,0,0,1,0,0]
judge  = [1,0,0,0,0,0,0,1,0,1]

tn, fp, fn, tp = confusion_matrix(expert, judge).ravel()
tpr = tp / (tp + fn)     # 0.67  caught 2 of 3 real failures
tnr = tn / (tn + fp)     # 0.86  passed 6 of 7 good answers
print(cohen_kappa_score(expert, judge))   # 0.52

	Judge says pass	Judge says fail
Expert failed it	ships the bug	correct
Expert passed it	correct	noisy CI, wasted engineer time

A low true positive rate is the expensive failure. A judge at 0.9 TPR and 0.7 TNR costs an engineer ten minutes on a false alarm, while the reverse lets a real defect reach a user. When the two trade against each other, we bias toward catching failures.

Kappa is still worth computing as a single summary, because it corrects for chance agreement in a way raw accuracy does not.

Îº	Reading
below 0.4	The judge is not measuring your rubric
0.4 to 0.6	Usable for direction, not for a CI gate
0.6 to 0.8	Fine for a CI gate with a margin
above 0.8	As good as a second labeller

How many labels, and when to stop

Start at thirty and keep going until you stop learning. For validating a judge specifically, a hundred to a hundred and fifty labels with failures deliberately oversampled is the working range, and the oversampling is not optional. If ten percent of your traffic fails, a uniform sample of a hundred gives you ten negative examples. That puts an interval around your true positive rate wide enough that the number cannot support a release decision.

Below 0.4, check the humans before the judge

Do not begin tuning the judge prompt. Have a second person label thirty cases and compute human-to-human agreement first. If two domain experts only agree at 0.5, no judge will beat that, and what you have found is an underspecified task rather than a broken evaluator. That is a product problem, and no amount of prompt engineering resolves it.

When it fails, split the rubric before touching the model

A judge scoring around 0.45 is usually being asked a compound question, and the fix is to decompose it into separate calls that each ask one thing:

1. Does every factual claim appear in the passages?      grounded | not_grounded
2. Does each citation govern the question asked?         valid | invalid
3. Given the passages, should this have refused?         should_answer | should_refuse

Each of those is close to mechanical, and the results are compounded afterwards in code. This is usually the largest single improvement available, and it requires no prompt tuning beyond the split itself.

Three biases to expect

Judges prefer longer answers, so if a change made your outputs longer, some of the apparent improvement is an artefact; bucket by response length and check that the score holds within each bucket. In pairwise comparison one position, often the first, wins more often than it should, so run both orders and average. And a judge prefers text from its own model family, so if one provider both writes and judges, score a sample of cases with a judge from a different provider and look for disagreement concentrated on your own model's output.

Re-run the alignment check whenever you change the judge model, the rubric, or the task itself. Provider updates move judge behaviour silently, and nothing in your pipeline will announce that it has happened.

On RepliCoach, our own sales-roleplay build, we kept the model out of the number altogether. The model writes the coaching feedback and a versioned rubric sets the score. If the model both writes the feedback and sets the number, a model upgrade shifts every historical comparison, and nobody can tell whether reps improved or the scale moved. Raw model scores also drift between personas and between conversation lengths, so the build normalises them and watches the score distribution for each persona as the sign that the normalisation needs revisiting.

How do you stop overfitting the set?

You will overfit it, because the set is small, you read the failures, and you fix precisely those. Splitting it is the standard defence and it works here for the same reason it works anywhere else.

Split	Size	Use
Dev	70%	Read these. Iterate against them freely
Test	30%	Run before a release. Do not read the failures

When the dev split keeps improving and the test split does not move, you are tuning to the cases rather than to the task, and that divergence is invisible without the split. Beyond that, retire cases the system has passed for three months and replace them with fresh production failures, because a set that never changes is measuring a system that stopped changing.

Is the difference real?

Usually not, at the set sizes teams actually have. Eval sets are small and small sets are noisy, and a hundred cases scoring 0.90 carry a confidence interval of about six points either side. A run at 0.91 and a run at 0.89 are therefore the same result as far as the evidence goes, and teams ship on that difference every week.

Comparing the two runs as separate samples makes the problem worse than it has to be. Detecting a move from 0.89 to 0.91 that way needs somewhere near three and a half thousand cases, which no team building a product is going to label.

Both runs went over the same cases, though, so they were never separate samples to begin with. Throw away every case that scored the same in both and count only the ones that changed verdict.

Cases that flipped	Reading
5 better, 0 worse	Not enough yet
6 better, 0 worse	Real
8 better, 1 worse	Real
9 better, 3 worse	Not enough yet
12 better, 3 worse	Real

Six cases moving the same way is enough at the usual five percent threshold, because a change that did nothing would have split them roughly evenly. That is also why nine better against three worse is weaker evidence than six against none, though both leave you six cases ahead. This is a sign test and any statistics library will run it. It is also the second reason to write per-case results instead of an aggregate, since the aggregate is the one number that cannot tell you whether the move was real.

Two limits are worth holding on to. The test tells you the direction of the change is real, and says nothing about whether the size of it is worth shipping. And thirty cases can only produce six flips if the change touched a fifth of them, so a small set still hides small improvements. A two-point move needs at least three hundred cases even when every flip goes the same way, and more when some go the other way, so the better question is why a two-point move is a release decision at all.

We hold our own case studies to the same standard. We put a figure on a case study when it was measured the way this guide describes.

What does this look like in CI?

Deliberately ordinary, which is the point: a harness nobody runs is worth nothing, so the eval suite goes where every other check goes and fails a build the same way.

# .github/workflows/evals.yml
on:
  pull_request:
    paths: ["prompts/**", "retrieval/**", "chunking/**", "tools/**", "evals/**"]

jobs:
  evals:
    steps:
      - run: uv run evals --set evals/dev.jsonl --report out/report.json
      - run: uv run evals-gate out/report.json
        # fails if:
        #   recall@20        < 0.90
        #   groundedness_tpr < 0.90
        #   refusal_recall   < 0.98   (missed refusals are the expensive failure)
        #   p95_latency      > 2500ms

The path filter is the part worth paying attention to, because chunking configuration and retrieval settings change output quality as much as the prompt does. A filter that only fires on prompts/** will let the most expensive changes through ungated.

Three practices matter more than which LLM evaluation framework you choose. Version the eval set in the repository, because it is source: it gets reviewed, and you can bisect it. Write per-case results rather than only an aggregate, because you need to diff two runs and see which five cases flipped, and a score moving from 0.91 to 0.89 tells you nothing about where to look. And pin everything, meaning temperature at zero where the model still accepts the setting, pinned model versions for both the system and the judge, and those versions recorded in the report, since otherwise you cannot attribute a regression to your own change.

Temperature zero plus pinned versions reduces variation and does not remove it, so a case that has to come back the same still needs the repeat-run check in why LLM temperature 0 is not deterministic.

Our DevOps agent runs the same idea inside the product. A validation loop checks every generated template before it reaches AWS, and template errors before deploy fell from around ten percent to four across that loop's introduction. Every validation failure and its revision is recorded, and those records are the input to improving the loop rather than the prompt. On the trademark platform the harness keeps a fine-tuned model changeable at all. Without it nobody would touch a model an attorney has signed off on, and the system would sit frozen at whatever quality it launched with. When the model id itself changes, as it does with the GPT-4o deprecation, the migration runs this same harness on the current model and the replacement before traffic moves.

Where does online evaluation fit?

Online evaluation tells you whether your offline score means anything. Everything up to here is offline evaluation. You wrote the cases, an expert wrote down the right answers, and CI scores one against the other on every pull request. The other half is online evaluation, which watches the shipped system answer questions nobody wrote an answer key for.

	Offline	Online
Runs against	Cases you wrote	Live traffic
Right answer	Written down first	Never written down
Answers	Did this change break something we already know about?	Did something break that we never thought of?
Cadence	Every pull request	Continuously
On a failure	Blocks the release	Opens an investigation

Most teams build the first column and stop there, which is where this guide has left you, and it sits awkwardly against the claim the guide opened with: validation of these systems is only feasible during operation.

Your offline score is a proxy for how the system behaves in front of users, and a proxy needs checking. The check is short to state: when the eval improves and nothing moves in production, your cases do not represent your traffic. That is a finding about the set rather than about the model, and the response is to go back to error analysis with fresh traces instead of tuning the prompt again.

So you need a production signal to compare against, and ratings will not give you one. Almost nobody clicks the thumb, and the ones who do are the angriest and the most delighted. A study of the WildChat and LMSYS conversation logs found this feedback works as a lens on your users rather than as a clean training target (User Feedback in Human-LLM Dialogues, EMNLP 2025).

Behaviour arrives at volume instead: did the user ask the same question again in different words, abandon the session, hit retry, edit the answer before sending it, or escalate to a human? Any one of them is ambiguous, since someone copying an answer might be satisfied or might be collecting evidence for a complaint. Read against a baseline and in aggregate they move when quality moves, which is the only property you need from them.

Those signals also decide which traces you read. You get thousands a week and can read a hundred, so something has to choose which hundred. Take the traces carrying a negative signal, the outliers with unusual retry counts or session lengths, and the ones your judge scored badly. Then keep a slice of plain random traces in every batch, because the random slice is the only part that can find a failure mode your current signals do not describe. Ten to twenty traces a week keeps the reading going between rounds, plus a hundred fresh ones each time you run full error analysis.

On the DevOps agent this was easier than it usually is. The signal was mechanical: the validation loop checks each generated template before deploy and records every failure with the request that produced it, so the system keeps its own record of what the model gets wrong without anyone deciding a failure was one. Most products do not get that, so pick two behavioural signals and instrument them before you need them. Treat the gap between those and your eval score as the thing you are managing.

The loop this all becomes

Error analysis is not a phase you complete and move past. Production traces produce new failure notes, those notes cluster into modes, the modes become cases, the cases gate deployments, and the deployments produce more traces. Run it once and the set describes the failures you had when you built it. Keep the loop going and new failures enter the set as production finds them.

What will evals not tell you?

Question	Why the harness cannot answer it
Is the product good?	It measures a task against labels you wrote. Label the wrong task and you will pass confidently while users leave
Is it safe against a motivated attacker?	The adversarial bucket contains the attacks you thought of. Prompt injection needs ongoing red-teaming
Do public numbers transfer?	Contamination makes benchmark results weak evidence about your corpus
How does it behave under load?	p95 with a single user says nothing about p95 under concurrency

A correct transcript can still be a failed call, so a voice product needs checks on the audio beside this text harness. Turn-taking, interruptions, intelligibility, and silence are scored on their own, and voice AI evals keeps those verdicts off the text score. Treat that set as its own cases rather than another row in the table above.

A fan-out needs the same kind of separate check. Score the merged result against the expected state, which when to fan out to many agents sets out, rather than the path each worker happened to take.

How do we run this, in order?

Collect a hundred real traces, from production if you have it and from whoever asked for the feature if you do not.
Open code them, writing a free-form note on each failure with no categories.
Axial code, clustering into five to ten modes, then count them.
Stop at saturation, meaning twenty fresh traces produce no new category.
Name a principal domain expert, one person, making binary judgements with written critiques.
Build the case set from the taxonomy, labelling the passage rather than the document.
Fix what a prompt change can fix, and build no judge for it.
Write assertions and reference checks for everything mechanical.
Build a judge only for the modes that persist, one question each, binary, with the exclusions in the prompt.
Validate it against a hundred labels with failures oversampled, reporting TPR and TNR. Below Îº 0.4, check whether two people agree before you split the rubric.
Split dev and test seventy-thirty.
Compare runs by counting flipped cases, not by reading the aggregate.
Gate CI on the wide path filter, chunking and retrieval included.
Instrument two behavioural signals in the product, and check the eval moves with them.
Feed every production failure back into the set.

Steps one through four take a day, and they decide whether everything after them is worth building. The first hour of it is thirty real queries copied into evals/dev.jsonl, with the person who knows the domain marking the paragraph that answers each one.

Common questions

How do I build evals for an LLM app?

Start from real traces and write a note on each failure before you name a category. Cluster the notes and have one expert mark pass or fail, with a critique, then build cases for those modes. Use code for anything mechanical, and let a model judge block a release only after it agrees with the expert.

What is an LLM eval?

An eval is a test whose pass or fail takes a judgement rather than an equality check. Grounded, correctly cited, and correctly refused are judgements, and two careful readers can disagree until the line is written down. The harness still lives in the repository, runs in continuous integration, and leaves a history you can bisect when a score moves.

Why can't I write the rubric first?

You learn the criteria by reading outputs, and that reading makes them specific. A rubric written before launch tends to ask for an answer that is accurate and helpful, which does not tell you what to change. An hour with real traces turns that into a concrete question, such as whether the response cited the clause that governs.

How many traces should I read?

Review at least a hundred traces, and stop when twenty fresh ones add no new failure category. Stopping earlier describes the sample you happened to open, and the modes you missed are the ones that show up later in production. Count after the categories exist, since a pile of retrieval misses will not move if the quarter goes to the prompt.

When should I not use an LLM as a judge?

Use code when the check is mechanical, such as valid structure, a citation that exists, or a forbidden string. Use a reference check when you already have a gold answer or a gold set of passages. A model judge takes days to build and a cost on every later case, so reserve it for modes that survive several fixes.

What is an LLM evaluation framework?

An LLM evaluation framework is the harness that runs a fixed set of cases against your system and scores each one, with code checks where the property is mechanical and a model judge only where it agrees with your expert. The tool you pick matters less than three practices: the case set versioned in the repository, per-case results, and pinned model versions recorded in every report.

How do I know the judge is good enough to gate a release?

Have the expert label about a hundred cases, with failures oversampled, and then see how many of those failures the judge caught. Also record how many good answers it passed, because raw agreement looks strong when most cases pass. Below a kappa of 0.4, check whether two people agree before you blame the judge, and split the question if they do not.

Sources and further reading

Hamel Husain, Creating a LLM-as-a-Judge That Drives Business Results and the LLM Evals FAQ.
Shankar et al., Who Validates the Validators?, UIST 2024, on criteria drift.
Barnett et al., Seven Failure Points When Engineering a RAG System, IEEE/ACM CAIN 2024.
Eugene Yan, Task-Specific LLM Evals that Do and Don't Work.
User Feedback in Human-LLM Dialogues, EMNLP 2025, on what implicit signals can and cannot carry.
Jason Liu, RAG series.
Anthropic, Contextual Retrieval.
Chroma, Generative Benchmarking, Evaluating Chunking Strategies and Context Rot.
Ragas metrics for standard definitions.
Ours: why RAG systems fail in production, why AI agents report unfinished work as done, when to fan out to many agents, why LLM temperature 0 is not deterministic, and GPT-4o deprecation: dates, replacement and how to migrate.
Prompt and evaluation engineers. The engagement that turns a quality feeling into a suite that can block a merge.

Originally published on bigcircle.ai. I'm Tirth, founder of BigCircle, where we build production AI systems with governance built in.

How to build an AI agent for production

Tirth Gajjar — Mon, 02 Nov 2026 18:41:17 +0000

To build an AI agent for production, decide first whether the task needs one, then give it a short list of typed tools, check every step that changes state against the system it changed, stop after a bounded number of retries, and keep a trace of each run that a person can read. We have built agents this way against a live AWS account, on an executive's desktop, and on our own research pipeline. The AWS agent gave us the rule we start from: nothing in the system depends on the model being correct.

The first decision is the shape of the loop. The table sets five common shapes against the place each one appears in our own work.

Criterion	Workflow	ReAct	Plan and execute	CodeAct	Computer use
When it fits	The steps are known and can be written down	The next step depends on what the last one returned	The task can be planned up front and checked before it runs	The work is data or a calculation the model can write code for	The system has a screen and no usable API
Main risk	Too rigid when the path is not known	One bad observation carries into every later step	The plan goes stale when the environment changes underneath it	Generated code can reach anything the interpreter can	Slow, and the end state is hard to check
How Bigcircle handles this	FinSight runs reconciliation as one governed path with versioned matching rules and no agent loop	The Solarpunk agent thinks, acts and observes one step at a time, and a critic decides after each step whether to continue, retry or escalate	Skionis turns a request into a structured spec and an explicit chain of tool calls, and a validation loop checks it before AWS runs anything	Solarpunk does spreadsheet analysis by writing and running code, and hands back a notebook that can be rerun	Solarpunk drives signed-in browser sessions for systems behind SSO with no usable API, and checks the claimed result against the application's own state

How do you build an AI agent, step by step?

This page gives the short version of each step, with the guide that covers it in full, and the sections below take them in this order:

Check that the task needs an agent rather than a workflow.
Pick the model by measuring it on your own cases.
Give the agent a short list of typed tools, and no shell.
Keep credentials out of the model's context.
Check every step that changes state against the system it changed.
Bound the retries, and hand a stuck step to a person with its trace.
Put the approval where a reviewer can judge it.
Add memory, caching or more agents only where each one pays for itself.

Does the task need an AI agent at all?

An agent earns its loop only when the next tool call depends on a result you could not name in advance. Everything else is a workflow, and a workflow is easier to test because every run takes the same path. Agentic workflows vs AI agents gives our test in full.

FinSight, a fund-operations reconciliation system we built, shows what a known path looks like. Each custodian gets its own mapping, and the matching rules are explicit and versioned because they are the domain knowledge. Every break becomes a record that a person resolves. We kept it one governed path because every hand-off through something outside the system is a place the audit trail breaks. When the steps are that well known, they belong in code.

Which model should the agent run on?

Pick the model by measuring it on your own cases, after the eval set exists. The AWS agent's design rule is the reason the choice weighs less than it does in a demo: nothing in the system depends on the model being correct. An engineer then does not have to form an opinion about model quality to decide whether to run it, because every step is checked by a machine or reviewed by a person.

Set the baseline on the most capable model you can use, then try smaller ones where they still meet your accuracy target, because the smaller model is only worth having if the eval says it holds. That only works if the cases exist before the swap, which is why LLM evaluation comes before the model choice. When the model under a live agent changes, LLM model migration covers the retest.

What should the agent be allowed to touch?

Give the agent narrow, typed tools and nothing broader. On Skionis, a DevOps agent that runs against a real AWS account, the action surface is a set of purpose-built tools that wrap the provider's APIs. There is no general shell, and no path by which a plan step becomes an arbitrary API call. The agent also reads the account's quotas, limits and existing resources before it plans, because a template can pass schema validation and still be refused by the account.

A tool list written into the prompt stops scaling early. On Solarpunk, an AI chief of staff that runs on the user's desktop, putting every tool in the system prompt worked until roughly thirty tools. Past that the model began choosing plausible but wrong tools, and nothing in the output showed it. We changed the design so the model is never given the full list of tools. It searches a registry of them and chooses among a handful of relevant tool contracts, and the product now has more than a hundred tools registered.

Overlap between tools matters more than the count, so the point where selection breaks will sit somewhere else on a different tool list. OpenAI's practical guide to building agents gives the same warning with numbers: some systems handle more than 15 distinct tools while others struggle with fewer than 10 that overlap. Measure selection on your own tool list. MCP tools makes the case for fewer, larger tools that do not overlap.

How should an AI agent handle authentication?

Keep credentials out of the model's context, and let the agent act only through tools that hold them. Solarpunk keeps credentials in an encrypted vault on the user's machine and injects them into the tool adapter when the call runs. The agent drives a signed-in session without ever reading a password, and it can act only through systems that were registered for it, so it cannot invent access it was not given.

The same rule protects your own monitoring. If no registered tool reaches the alert path or its configuration, the agent cannot switch it off, whatever the prompt or a tool result tells it. AI agent permissions covers binding each action to a user, a target and an approval, and AI agent security covers what the agent can still reach if it ignores a permission.

Give tools served over MCP the same contract, and check which revision of the protocol your SDK targets. As of October 2026 the current one is the 2026-07-28 revision, which dropped the session a server used to hold open for each client, so a tutorial built around MCP sessions predates it. The token, the check before each call and the audit line are in MCP authorization.

How does the loop check its own work?

Every step that changes state needs a check that does not come from the model. Without one, the agent claims success on work it never finished, and one bad observation carries into every step after it. The loop below is the shape we build, with the permission gate in application code rather than in the prompt.

On Skionis the check is a validation loop of structural and semantic tests that revises the template until the provider will accept it. Most errors are caught there, before AWS is asked to start anything. We measured template errors before deploy falling from around 10% to 4% across the loop's introduction. On Solarpunk a critic reads the raw result of each step, not a summary of it, and decides whether to continue, retry or escalate.

The critic judges one step at a time, so Solarpunk also compares the agent's claimed completion against the actual state of the target application. That comparison catches work that never took effect. Read AI agent verification for that expected-state check, and score whole runs offline before a change ships, as LLM evaluation covers.

What breaks once it is live?

Once an agent is live, the failures we see are about the environment it works in. On Skionis they are provider and state failures. A change set fails partway, a resource the plan depends on changes after the account was read, or a quota moves between plan and apply. On Solarpunk they are a login that fails, a page that moved since the plan was made, an action the application does not support, or a partial tool result. We file these as failures of the environment rather than of generation, and that is where the retry and escalation design has to point.

Retries need a limit and a place. Skionis retries in two places, inside the validation loop and around execution, because one recovers from a malformed template and the other from a transient or partial provider failure. A template that will not come clean after a bounded number of revisions goes to a person as a request the agent cannot safely fulfil. On Solarpunk an agent that retried forever against a broken vendor would burn cost and time while looking busy. It stops instead, and hands back the unresolved step with its trace attached.

That trace is the record you debug from. On Skionis one record links the request, the account facts, the generated template, every validation failure and revision, the provider's response and the resources created. The validation failures are the part worth keeping, because they are the system's own log of what the model gets wrong. AI agent monitoring splits observability into quality and infrastructure, and AI audit trail covers the record a challenged decision needs.

Where does a person approve?

Put the approval where a reviewer can judge it. On Skionis the human gate is an architecture diagram of the planned change. The alternative was raw template diffs, which engineers do not read carefully at the tenth change of the week. Engineers who had refused to let an agent near production came to let it own routine cloud changes, and the review surface earned that, not the model.

An agent is a budget of autonomy in one practical sense: each action it takes without a person is one a reviewer has agreed to stop checking. The staged rollout in AI agent governance is the method we use to move one. Human in the loop AI agents covers the smallest authority the next release should ask for.

Where do memory, cost and multi-agent designs fit?

Memory moved the completion rate on our desktop agent. On Solarpunk, plans completed with no human intervention rose from roughly 55% to 80% once local retrieval and planner memory grounded the agent in the user's own workspace. We measured that rate on the same agent, once before those two pieces were added and once after, and a before-and-after rate like that is the number to ask for whenever someone says memory helps. Agent memory that improves the next plan also carries one user's data into it. AI agent memory and a semantic layer for AI cover keying it to the right person.

Repeated work is where caching pays. On Prospex, our own prospect research engine, cache-first execution reuses earlier extraction and retrieval when a profile is rerun, instead of calling external services again. That cut repeated external API calls by an estimated 50 to 60 percent, and the cache hit rate is the cost metric we watch on it. AI agent cost splits the bill into the build, the run and the keep-alive, with a worked token calculation.

Split work across several agents only when the slices can run without reading each other, or the roles need different tools or authority, because each hand-off is one more place a partial result can read as a finished one. The fit test is in multi-agent systems. Prompt injection travels with the text one agent forwards to another, and prompt injection in multi-agent systems covers re-checking trust at each hop.

Common questions

How do you build an AI agent?

Check that the task needs an agent rather than a workflow, pick the model on your own eval cases, and give the agent a short list of typed tools with no shell. Keep credentials in the tool layer, check every step that changes state against the system it changed, bound the retries, and put the approval where a reviewer can judge it. Keep a trace of every run.

When should we build an agent instead of a workflow?

Build a workflow when you can write the steps down, and an agent only when the next tool call depends on a result you could not name in advance. A workflow takes the same path on every run, so it is easier to test and to audit. Bigcircle built the FinSight reconciliation system as one governed path with versioned matching rules for that reason.

What has to sit outside the model?

Permission checks, credentials, spend limits, the approval for anything you cannot undo, and the alert path that watches the agent. The model may request a tool call, and application code decides whether the call is allowed and whether a person must approve it first. If no registered tool reaches the monitor or its configuration, the agent cannot switch it off.

How do AI agents handle authentication?

The agent should never hold the credential. On the Bigcircle desktop agent, credentials sit in an encrypted vault on the user's machine and are injected into the tool adapter when a call runs, so the model drives a signed-in session without reading a password. It can act only through systems registered for it, so it cannot invent access it was not given.

How many tools can one agent choose from?

On the Bigcircle desktop agent, listing every tool in the system prompt worked up to roughly thirty tools. Past that the model began choosing plausible but wrong ones, with no sign of it in the output. Letting the model query a tool registry, and choose from a handful of relevant tools, took the same product past a hundred registered tools. Overlap between tools matters more than the count, so measure selection on your own tools.

How do you know an agent run actually finished?

Compare what the agent says it did with the actual state of the system it changed, using a read that does not come from the agent. A successful API response is not a finished task. Bigcircle's desktop agent checks its claimed completion against the target application's state, and work it cannot finish comes back as a flagged exception rather than as done.

What is a safe way to give an agent more authority?

Give the reviewer something they can judge, then widen one action at a time with evidence from the traces. On the Bigcircle AWS agent the approval step is an architecture diagram rather than a template diff. Engineers who had refused the agent came to let it own routine cloud changes. Store the prompt, the tool schemas and the model version with every run so a regression can be traced.

Sources

OpenAI, A practical guide to building agents, April 2025. The tool-overlap figures.
Model Context Protocol, 2026-07-28 specification changelog.

Originally published on bigcircle.ai. I'm Tirth, founder of BigCircle, where we build production AI systems with governance built in.

RAG vs fine-tuning vs agentic search, and when to use each

Tirth Gajjar — Sat, 31 Oct 2026 18:40:36 +0000

Choose between RAG and fine-tuning by the failure you can see. Use retrieval when the model is missing a fact, and fine-tuning when the facts are right and a specialist says the output reads wrong. Use agentic search, also called agentic RAG, only when no single query can find the answer.

Prompting comes before all three, because it costs nothing extra to try. Every extra search step is another model call the user waits for, and most production systems end up running several of these layers at once.

Criterion	Prompting	Retrieval	Fine-tuning	Agentic search
Failure you can observe	Output is inconsistent, or wrong in a way you cannot describe	Model does not know a fact, or knows an old one	Facts are right, form is wrong to a specialist	No single query can retrieve the answer
What it changes	Instructions, examples, and the output schema	Which passages are fetched for this question	Default form, and judgements a written rule does not settle	Reach, by searching again from what the last search returned
What it leaves alone	The weights, and which documents exist	How the model writes once the right passage is present	A fact that changed after the training run	The accuracy of a question one search could have answered
What you build	A prompt, examples and an output schema	An index, the jobs that keep it current, and permission filters	A training set, an eval against the base model, and a hosted model	A loop with a stop condition and a record of every step
Added latency	None	One search before the model call	None once served	One model call per step, each waiting on the last
Cost per call	Tokens	Tokens, the search, and the passages added to the prompt	Can fall, if a small tuned model replaces a large one	Multiplied by the number of steps
Ongoing burden	None	Keeping the index current	A model to own and retrain	Showing the loop stopped for a good reason
How Bigcircle handles this	Schema-constrained decoding sits in the same request as the other layers on the trademark platform	Retrieval restricted to a curated index of authorities on that platform, live since 2023	A fine-tuned model in the same request, for structure and register, which retrieval could not fix	On Prospex, a search loop only after hybrid retrieval stopped improving. The insurance voice agent loads its context when the call opens and does not search during a turn

The standard answer to this question is "RAG for knowledge, fine-tuning for behaviour." That answer is close, and it breaks on the first system that needs both. Our trademark drafting platform has been live since 2023 with a fine-tuned model, retrieval over a curated index of authorities, and schema-constrained decoding in the same request. We did not run all three because we could not choose. Each one fixes a failure the other two cannot reach.

So start from the failure you are looking at, and then pick the least expensive layer that fixes it. Read the columns of the table above from left to right. Prompting is the column to exhaust first, and a later column does not fix a failure that belongs in an earlier one.

When is it just a prompting problem?

When the task is underspecified rather than unknown to the model. Every team believes it has finished prompting, so test it. Take your prompt, hand it to a contractor who knows nothing about your product, and ask them to do the task.

If they come back with questions, the model has those questions too, and it does not ask them. It picks an interpretation, commits to it, and you file the result as a model failure. A support summariser gets a ticket with three separate issues in it, and nothing in the prompt decides whether that is one summary or three. The model picks one reading, and next Tuesday it picks the other. No amount of retrieval touches that, because nothing is missing from what the model knows.

A fix at this layer takes an afternoon and leaves nothing new to host or retrain, which is why it comes first. Before you spend money on another layer, do the following.

Constrain the output mechanically. Structured output with constrained decoding removes invalid tokens from the choices the model has while it writes, so the output always parses. If you are still parsing free text with a regex and a retry loop, the defect is in the prompt, and it will get filed against the model.

Show the output instead of describing it. Four real examples of the exact output do more than two paragraphs of adjectives, because the model copies what it is shown (in-context learning). Before anyone proposes a fine-tune, count the examples in the prompt. If the count is zero, the free version of the same idea has not been tried yet.

Write the refusal condition, one sentence along the lines of "if the retrieved passages do not contain the answer, say so and stop". Without it the model fills a gap with something plausible, and a reader cannot tell which sentences came from the passages and which were a hallucination.

You cannot skip the eval set

Collect twenty to fifty real cases with known-good outputs and score them the same way on every run. Until that set exists you cannot tell a fix from a lucky sample, and you cannot tell whether the next layer helped. Evals are how you read every decision below this point. How to build LLM evals covers building the set, from the first failure note through a model judge.

When do you need RAG?

Use retrieval when the failure is missing or stale knowledge: facts specific to your business, documents written after the model's knowledge cutoff, anything that changes weekly, and anything a user is allowed to see only part of.

Your refund window changed in March and the model still answers with the old one. Fine-tuning will not fix that, and it will not fix it again in June when the window changes back. The mechanics are in why RAG systems fail in production. The question here is whether retrieval alone is enough.

Retrieval quality depends less on the model or the vector database than on four choices: chunking, embeddings, hybrid search so that part numbers and names still match exactly, and reranking so the best passage reaches the context window rather than a related one.

Work on those four before you touch the model, because no model can use a passage that never reached it. In Anthropic's contextual retrieval tests from September 2024, the share of queries whose right chunk missed the top 20 fell from 5.7% to 1.9% once each chunk carried 50 to 100 tokens of generated context, keyword search ran beside embeddings, and a reranker sorted the result. The added context alone cut failures by 35%, keyword search took the cut to 49%, and the reranker to 67%. The model stayed the same throughout, so the whole gain came from those choices.

On Prospex, our own prospect research engine, semantic and keyword search run together, and a claim with no retrieved passage behind it does not appear. On one run that took 420 collected items down to the 6 that answered the question. We estimate it makes 50 to 60 percent fewer external API calls on repeated runs, and that saving comes from caching rather than from retrieval: the runtime is cache-first, so a rerun on an overlapping account reuses work instead of paying for it again.

Teams get two things wrong at this layer. Citations are an architecture decision, and teams keep treating them as formatting. If a regulated user has to trust a number, they need to open the page it came from.

On our ESG filings platform every extracted field carries a link to its source page and a confidence score, and a low-confidence field goes to review instead of into a dashboard. We measured the analysis cycle falling from weeks to hours once the pipeline replaced manual extraction and spreadsheet work. Most of the recovered time came from the confidence flags: analysts who could see which figures were uncertain stopped re-deriving earlier work to be safe.

Permissions belong in the query. A filter applied after retrieval runs once the document is already in the search results, and anything downstream that logs, caches or summarises those results can leak it. Filter by the user's access inside the search itself.

Three ways retrieval turns out not to be enough

Each row below is a symptom you can see in a trace, and the column on the right names where it sends you.

What you see	What it means	Where to go
Right passage in context, answer still reads wrong to a specialist	Form problem, not fact problem	Fine-tuning
Right passage cannot be found by any single query	Question has to be split into several searches	Agentic search
The corpus does not contain the answer	Not a technique problem	Say so and stop

That third row is worth saying out loud early, because no layer fixes a missing document.

When should you fine-tune instead of using RAG?

When the failure is form, or judgement a written rule does not settle, and never when it is missing knowledge. Fine-tuning does not teach facts reliably. It can be made to, badly, and then you cannot update a fact without retraining. What it moves is default behaviour: output shape, register, domain convention, and the small judgements a specialist makes without thinking.

The usual test is whether the expert can write the rule down. That is close, and the sharper test is whether the written rule decides the output.

Trademark work is the clearest case we have, and the problem is not that practitioners cannot explain anything. They can. The rules are written down, and they are general, and a general rule does not settle a specific draft on its own. Precedent settles it: how comparable cases were decided, and how those decisions have shifted what a defensible specification looks like in practice.

You cannot compress that into prompt instructions, because it is not one book. It is applied interpretation spread across many decisions, where the same rule gives different results depending on the case in front of you. So the layers split cleanly, and each one takes the part it can hold:

What it is	Where it lives
The general rules, written and stable	Prompt instructions
The specific authorities and prior marks for this case	Retrieval, restricted to approved sources
The pattern of how those rules get applied in practice	The fine-tune

People try to solve that third row with a longer prompt. A prompt listing the general rules produces a draft that is technically compliant and reads wrong to anyone who has practised. Before training, we normalised years of attorney opinions into one five-section template, tagged by mark type, jurisdiction and NICE class. Without that step the model would have learned a dozen individual writing styles and averaged them. It mattered more than we expected.

The fine-tune also has a failure we designed around. It improves how a citation is phrased, which is the wrong thing to improve when the authority might not exist. The fine-tuned model is never asked to remember which classes exist, which prior marks conflict, or what the registry shows this week. Retrieval over the curated index supplies those, a validator rejects any citation that does not resolve, and an attorney reviews and signs the draft.

What it costs beyond the training run

Budget the training run last. It is the smallest line and often the only one in the estimate, because LoRA and QLoRA shrank the compute. The other lines did not shrink:

Cost	What it looks like
Data	Examples of the right output, written or checked by an expert. On the trademark platform that meant normalising years of opinions into one template before any training
Evaluation	Proof the tuned model is better rather than different. The trademark platform logs the base and fine-tuned models on the same inputs, so a regression shows up as a score that moved
Permanence	A fork in your stack. Every new base model is a decision about redoing the work, and providers retire the bases you tuned on
Operations	Hosting, capacity, version pinning, rollback. A prompt change has none of these

Synthetic data helps with volume and cannot invent the judgement you are trying to capture. If you generate the examples from your own prompt, you have trained an expensive copy of that prompt. LLM-as-judge scales the evaluation, and the judge is a model with its own biases that needs spot checks against an expert. A tuned model also ages. If nobody reruns the comparison against the current base model, the team keeps paying to host a fork that may now do worse than a prompt on the newer model.

The fine-tune we did not build

A client came to us wanting a fine-tune for cold outreach email, on a small low-cost model, tuned for open rates. It is a reasonable-sounding request, because writing quality is exactly the kind of thing people expect fine-tuning to fix. We did not tune anything, and splitting the job into two calls fixed it.

The first call does a short discovery pass on the prospect, and the second writes the email given that research and a handful of examples of what good looks like in their domain. That is two hops instead of one, with no new weights anywhere.

That is the pattern we keep meeting, where "the output is not good enough" rarely means the model needs different weights. Usually it means the model is being asked to research and write in a single step, with nothing to imitate. Splitting the step and showing examples takes an afternoon, and the fine-tune was going to take a month.

Fine-tune when the behaviour is unwritten, stable enough to outlive a model generation, and measurable. Otherwise try many-shot prompting first and score it on the eval set. Check the provider's retirement dates before you tune, because a provider can stop training new versions of a tune you still depend on: as of 22 October 2026, OpenAI's deprecations page gives 6 January 2027 as the last day existing customers can create fine-tuning jobs. A tune already running on a base with a shutdown date is the case how to replace the model under a production agent walks through.

The other reason to fine-tune

The other reason is cost and latency rather than quality. Distillation moves a large model's behaviour on your one task into a small model you can serve at lower cost or run on a device. Judge that on cost per request, because it is a serving decision and the argument above does not apply.

What is agentic RAG, and when do you need it?

Agentic RAG puts retrieval inside an agent loop. The model decides what to search for, reads what came back, and then decides whether to search again or answer, where ordinary RAG runs one search and answers from whatever it returned. We call it agentic search in this guide, because the loop can call any search tool and a vector index is only one of them.

Single-shot retrieval assumes the question maps to a query, and some questions do not. "Which of our suppliers mentioned tariff exposure in their last two filings?" No query answers that. You need the supplier list, then the filings per supplier, then a read of each, then a decision about when you have enough. An agent that plans a step, acts, reads the result and plans again handles that (plan-and-execute, the ReAct pattern). If one query still answers the question, stay with retrieval. Our guide on agentic workflows vs AI agents separates that case from a scripted workflow and from this loop.

The loop buys reach with latency and cost, so we add one only when a simpler pipeline has visibly fallen short. Two conditions make it worth the price: the corpus changes faster than an index can follow, and exact search tools already exist for the loop to call. Code meets both. Claude Code's team dropped local vector indexes for model-driven file-name and text search (glob and grep), citing stale indexes and permission complexity (Boris Cherny to The Pragmatic Engineer, March 2026). A large document set that changes monthly and has no structured search meets neither, and retrieval still suits it.

Our own systems mostly run retrieval inside a fixed sequence of steps. On Prospex, the reasoning steps (business model, customer fit, strategy, risk, tech stack) run in a set order and each one retrieves for itself, so the evidence for every claim stays attached to that claim and two runs on one company are built to give the same read.

On the executive operations agent, we measured plans completed with no human stepping in rising from roughly 55% to 80% once local retrieval and planner memory grounded the agent in the user's own workspace. The same agent has a critic that chooses continue, retry or escalate after each step, for a different reason: without it, one bad observation carries through every later step and the agent claims success on work it never finished.

When is the loop not worth it?

A loop turns one search into several model calls and searches in sequence, and the user waits for every one of them. Ask three questions before you build it.

Is the question really multi-hop, or is retrieval weak? Many complaints that "the agent needs to search several times" are single-shot retrieval failures nobody diagnosed. Check whether the passage you need ranks near the top for any one query. If it does, fix hybrid search and add reranking, which changes one step of the request, before you add a loop that changes how many steps there are.

There is a real exception, and we hit it on Prospex when retrieval had run out of road. The source data was structured in a way that resisted tuning: the evidence a single question needed was spread across documents that looked nothing like each other, so better chunking and better hybrid search kept arriving at the same ceiling. An agent that searched, read what came back, and searched again got past that. Another month of retrieval work would not have.

So the check is not whether you have tried reranking, but whether you have a reason to believe more retrieval work will move the ceiling. On Prospex we did not, and we could say why.

Is there a latency budget? Rule the loop out for voice before anyone designs it, because the pause a caller expects is short and similar across languages: in a ten-language study, every language's average gap between turns sat within 250 ms of the cross-language mean (Stivers et al., PNAS, 2009). A search loop does not fit in that gap.

Our insurance voice agent came down from 2.5 to 3 seconds to around 900ms end to end in our tests, and it does not search during a turn at all. Policy and claim context loads when the call opens, so no reply waits on it, and every action goes through a typed tool that checks its inputs and permissions.

Can the loop stop, and can you show it did the work? An agent that decides when it has enough evidence also decides when to stop. Solve that before the loop ships, because an agent that stops early still writes a confident answer. Our guide on AI agents that call unfinished work done covers the checks. The loop adds reach, and it makes no single search more accurate, so if one good search can answer the question, that search will be faster and easier to debug.

What does each layer cost to run?

Two cells in the comparison at the top surprise people. Fine-tuning can lower the cost per call, because a small tuned model that matches a large one on your single task costs less to serve and answers faster. When cost matters more than quality, the usual advice inverts.

Retrieval's real running cost is freshness, not the search. Keeping the index correct as documents change, permissions move and pages get deleted is the work, and an index that falls behind raises no error, so nobody notices until an answer quotes a page that no longer exists. The multiplied calls in the agentic column are a separate bill, which our guide on AI agent cost splits into the build, the run, and the work that remains after launch.

Before adding any layer, check prompt caching on a stable system prefix and model routing for the easy requests. Neither one changes what the model does.

What does running all three at once look like?

The trademark path, in production since 2023, runs three layers in one request, and every draft is scored before an attorney sees it.

Remove one and the system fails in a specific way. Without retrieval it invents authorities. Without the fine-tune the drafting reads wrong to a practitioner. Without the schema, the integration breaks the day a model update changes its formatting habits. The evaluation scores each draft from 0 to 10 on six criteria (structure, reasoning depth, citation accuracy, jurisdiction discipline, edge cases and tone), and attorney corrections remain the one signal that automated scoring cannot replace.

What will none of this fix?

These four layers get sold as answers to problems they cannot touch. A missing document stays missing, and if the corpus does not contain the answer, retrieval returns the closest wrong thing and generation makes it sound plausible.

A task nobody has defined stays undefined. If three stakeholders read the spec three ways, a fine-tune learns whichever reading dominated its training examples. Disagreement between experts also stays disagreement. If two specialists mark the same output differently, more training data does not resolve it, and the call belongs to the product owner rather than to the model.

Bad source data stays bad, whether that means duplicated pages, stale exports or a PDF parser that drops table headers, and every layer downstream inherits it.

How do you decide, in order?

Stop at the first question you cannot answer yes to, and fix that before moving on.

Can a new contractor do the task from your prompt? If not, write the specification, add four real examples, and name the refusal condition. An afternoon.
Do you have twenty to fifty scored eval cases? If not, build them before changing anything else.
Is the failure a missing or stale fact? Retrieval: hybrid search, reranking, chunking, permissions. Not a bigger model.
With the right passage in context, is the answer wrong in form rather than fact? That points to fine-tuning, if the convention is unwritten and stable. Try many-shot prompting first and measure it.
Does answering require several dependent searches? Agentic search, with a stop condition, a verification step and a latency budget you have checked.
Are you optimising cost or latency rather than quality? Distillation, model routing, prompt caching, judged on cost per request.

To find where you are, open your last twenty production failures and label each with one word: unspecified, missing, form, or multi-hop. Count the labels. The largest bucket tells you which step to work on, and it is often not the layer the team was planning to build next.

Common questions

Should I use RAG or fine-tuning?

Use retrieval when the model is missing a fact or is still quoting a document that has changed. Fine-tune when the right passage is already in context and a specialist says the draft reads wrong. Finish the specification, the examples, and the refusal line before either of those, because an unspecified task looks like a model failure.

When should you fine-tune instead of using RAG?

Fine-tune for form and for judgement a written rule does not settle, and never to store a fact you will need to update. The authorities and the current documents stay in retrieval. Try many-shot prompting first and score it on your eval set. A tuned model you do not compare with today's base model is a fork you then have to keep.

What is agentic RAG?

Agentic RAG, also called agentic search, is a loop that plans a query, reads what came back, and searches again from that result. Use it when no single query can find the answer, such as a supplier list, then each filing, then a read of each one. Stay with one search when that search can answer, because every step of the loop is another model call the user waits for.

What is the difference between RAG and agentic RAG?

Ordinary RAG runs one search before the model call and answers from what that search returned. Agentic RAG lets the model decide what to search for next, read the result, and search again until it has enough or hits a stop condition. The loop adds reach for questions that need several dependent searches, and it makes no single search more accurate.

Can RAG and fine-tuning run in the same request?

Yes. Bigcircle's trademark drafting platform has done it in production since 2023. It retrieves from a curated index of authorities, drafts with a fine-tuned model, and constrains the output to a five-section schema whose citations must resolve. Drop retrieval and the draft invents authorities, and drop the fine-tune and a practitioner says the form is wrong.

How do I choose between prompting, RAG, and fine-tuning?

Stop at the first step you cannot pass. If a new contractor cannot do the task from the prompt, write the specification and add real examples before you touch a model. If the failure is a missing or stale fact, use retrieval, and try many-shot prompting before a fine-tune when the passage is already in context.

When is an agent the wrong way to search?

Skip the agent when one query can still answer the question. Many complaints that the system must search several times are single-shot retrieval misses, fixed by hybrid search and reranking without adding steps. A voice product answering inside a conversational pause cannot host the loop, so Bigcircle's insurance voice agent loads its context when the call opens and does not search during a turn.

Sources and further reading

Anthropic, Introducing contextual retrieval, September 2024. Top-20 retrieval failure from 5.7% to 1.9%.
Gergely Orosz, Building Claude Code with Boris Cherny, The Pragmatic Engineer, March 2026. Why Claude Code dropped vector indexes for glob and grep.
Stivers et al., Universals and cultural variation in turn-taking in conversation, PNAS, 2009.
OpenAI, deprecations, checked 22 October 2026. The fine-tuning cut-off date.
Why RAG systems fail in production. The retrieval layer in depth, failure by failure.
The practical guide to AI agents. What makes a loop controllable rather than just autonomous.
Agentic workflows vs AI agents. Whether the task needs this loop, a scripted workflow, or retrieval on its own.
Why AI agents report unfinished work as done. The verification problem agentic search inherits.
How LLM inference actually works. Why prefill, decode and caching drive the cost column above.
Case studies referenced: trademark drafting, prospect intelligence, ESG filings, executive operations, insurance voice.

Originally published on bigcircle.ai. I'm Tirth, founder of BigCircle, where we build production AI systems with governance built in.

GPT-4o deprecation: dates, replacement and how to migrate

Tirth Gajjar — Thu, 29 Oct 2026 18:40:20 +0000

GPT-4o leaves the OpenAI API on 14 January 2027. As of 22 October 2026, that date covers the listed GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini snapshots and the fine-tunes built on them, and OpenAI names gpt-6-luna as the replacement. Azure retires most of the same versions on 14 April 2027. ChatGPT dropped GPT-4o in February 2026, which did not touch the API.

Changing the model string takes a minute, but proving the product still behaves takes the rest of the migration, because Luna handles tool calls, reasoning and sampling differently from the model your evals were scored on. This guide gives the dates, then the order of checks we recommend for this swap and every one after it.

When is GPT-4o deprecated on the OpenAI API, Azure and ChatGPT?

Plan against the earliest date that touches any id you call. Each platform runs its own schedule, and a team that reads one page and assumes it covers the others plans for the wrong day.

Where the model is called	Shutdown date	Listed replacement
OpenAI API	14 January 2027 for `gpt-4o-2024-08-06`, `gpt-4o-2024-11-20`, `gpt-4o-mini-2024-07-18`, `gpt-4.1-2025-04-14`, `gpt-4.1-mini-2025-04-14`, and fine-tunes on them	`gpt-6-luna`
OpenAI API, earlier batch	23 October 2026 for `gpt-4o-2024-05-13` and `gpt-4.1-nano`, including its fine-tune	`gpt-5.6-sol` for the GPT-4o snapshot, `gpt-5.6-luna` for nano
Azure OpenAI	9 December 2026 for `gpt-4o` 2024-05-13. 14 April 2027 for `gpt-4o` 2024-08-06 and 2024-11-20, `gpt-4o-mini`, and all three GPT-4.1 sizes	`gpt-5.6-sol` for 2024-05-13, `gpt-5.1` for the later GPT-4o versions, none listed for GPT-4.1 or `gpt-4o-mini`
Azure OpenAI, fine-tuned deployments	1 October 2027 for GPT-4o and GPT-4o mini fine-tunes, 14 October 2027 for GPT-4.1 fine-tunes	Training ends earlier, no earlier than April 2027 for existing customers
ChatGPT	Retired 13 February 2026	Not an API change

Dates are as listed on OpenAI's deprecations page and Microsoft's retirement schedule on 22 October 2026. Any call still pinned to the 23 October batch should already be on its replacement, scored on live traffic from the first day.

Size your retest window to three months, not six. OpenAI's policy promises at least six months' notice for a generally available model, with less allowed for a specialized variant or for safety and compliance reasons. The GPT-4 series notice went out on 14 October 2026 for a 14 January 2027 shutdown, three months, so a plan that assumed six has already lost half its time.

On Azure, set versionUpgradeOption deliberately on every deployment. Standard, Data Zone Standard and Global Standard deployments are upgraded automatically at retirement, so a gpt-4o 2024-05-13 deployment becomes gpt-5.6-sol on 9 December 2026 with no code change. NoAutoUpgrade makes the deployment stop working at retirement instead. Provisioned deployments, which reserve fixed capacity, are never upgraded and have to be moved by hand. Score an automatic upgrade exactly as you would score a change you chose.

Why does a passing eval on GPT-4o not carry over to the replacement?

Treat the replacement as a new system and score it from the start, even when the prompt is identical. The agent can pick a different tool or fill a field with a different value. A tool schema can stay valid while the arguments change shape, and a structured-output schema can still parse while one field now means something else. One good completion on Luna shows none of that.

We design for this before any migration is planned. The trademark drafting system, in production with a client, logs the base model and the fine-tuned model on the same inputs against a 0 to 10 legal-integrity harness on six axes, and every draft passes that harness before an attorney sees it. A regression from a model change appears as a score movement, before an attorney reads the draft.

Our own sales roleplay coach takes the other half of the idea. Its score comes from a versioned rubric with weighted dimensions and behavioural markers, and the model only writes the qualitative read. Raw model scores drift between sessions, conversation lengths and personas. If the model also set the number, an upgrade would move every historical comparison, and nobody could tell a better rep from a different model.

Survey data points the same way: in VentureBeat's September 2026 survey of 109 opt-in respondents, 59 percent of the companies that replaced a model in a production app saw performance drop and reworked the app or went back to the original. Read it as a reason to score your own cases, not as a rate to plan around.

How do you run LLM regression testing before the model id changes?

Run the same eval cases on the current model and the replacement, each case several times, and only then move traffic in stages. The order we recommend is the same whatever the vendor, and each row below names what to record and what fails if the step is skipped.

Step	What to record	What fails if it is skipped
Inventory	Every model id in code, config and stored jobs, including dated snapshots, fine-tune ids and Azure deployment names	A forgotten snapshot stops answering on the shutdown date while the tests stay green
Pin	The replacement id, written into the eval report beside the id it replaces	An alias that moves later is a second change nobody scored
Same cases	The existing eval set, run on the current model and on the replacement	A score with no pair cannot show a regression
Repeats	A pass rate where wording may vary, and a pass on every repeat for a safety case	One green run hides a completion that will not come back
Contracts	Tool name, arguments and the structured-output schema, checked on their own	The prose still reads well while the integration stops parsing
Shadow	The new id run on live inputs, scored, with no action taken	The canary becomes the first time the live mix of requests appears
Canary	A slice of traffic, with the old id still callable	A full cutover has no rollback once config has forgotten the old id
Cost	Tokens, reasoning tokens and cache writes, on your own traces	A lower token price on a longer trace can still cost more
Retire	The old id removed from the inventory once the canary holds	The next person to read the config puts back an id that is about to stop answering

OpenAI's own migration note has four steps: record a baseline of quality, latency and cost, evaluate Luna starting at reasoning effort none, tune and retest, then send a limited share of production traffic. All four sit inside this table, so add the two rows it leaves out. A baseline that stores one completion per case cannot give you a pass rate, and a prose score cannot show that a tool call changed shape.

Score each case more than once, because a hosted model does not return the same answer twice even at temperature 0. Why LLM temperature 0 is not deterministic explains why a safety case has to pass on every repeat while a quality case ships on its pass rate. Take the cases from failures your team has already read, as our guide to LLM evaluation sets out, rather than writing new ones for the migration.

Move the eval set itself out of OpenAI's platform if it lives there. Existing evals on the Evals dashboard become read-only on 31 October 2026, and the Evals API, reusable prompt objects and the v1/prompts API shut down on 30 November 2026. Keep prompts and cases in your own repository, where the next migration can rerun them.

Keep the old id callable until the canary's scores hold, or until the shutdown date removes the choice. Deleting it from config on the day the evals pass leaves no way back if the next day's traffic is a different mix from the one you scored.

What does GPT-6 Luna change in a GPT-4o request?

Start the migration at reasoning effort none, on the endpoint you already use. It keeps the request closest to the old one, and on Chat Completions it is the only setting where tools still work. Then test higher settings against quality, latency and cost.

Reasoning effort on Luna runs from none through low, medium, high, xhigh and max, and the default is medium. A request that sends no setting therefore starts reasoning, which the GPT-4o call never did, and reasoning tokens bill as output.

Tool calls move endpoint once reasoning is on. Luna accepts function calling on v1/chat/completions only at reasoning effort none, and reasoning combined with tools goes through the Responses API. GPT-6 Astra and GPT-6.1 Sol need Responses for any tool call. Chat Completions has no shutdown date, but plan the move to Responses now for any route that will need reasoning and tools together.

Sampling settings go too: at any reasoning effort other than none, the request has to drop temperature, top_p and top_logprobs, plus logprobs on Chat Completions. A prompt that relied on a low temperature to hold its format loses that control, so the format can move with nothing on the request to explain it. Check the structured-output contract on its own at every reasoning level you test.

Compare total cost on your own traces, not the price per token. As of 22 October 2026 Luna lists $0.10 per million input tokens and $0.50 per million output, with cached input at $0.01 and cache writes at $0.125. A prompt over 272,000 input tokens is billed at twice the input and cache rates and 1.5 times the output rate for the whole request. The context window is 1,050,000 tokens, so a retrieval prompt that sat under GPT-4o's limit can now cross that price step with no change to the chunker. What a production AI agent costs works through how a longer trace shows up on the bill.

What happens to GPT-4o fine-tunes?

Work out what each fine-tune is holding before 6 January 2027, because the replacement OpenAI names cannot be fine-tuned. On the OpenAI API, fine-tunes on the January bases shut down with the base model on 14 January 2027, and from 6 January existing customers can no longer create fine-tuning jobs. As of 22 October 2026 the GPT-6 Luna model page lists the fine-tuning endpoint as not supported.

Do not start a fresh job on a GPT-4o base before the cutoff, because a model trained in late December is one you have to leave again in January. Use the eight days between 6 and 14 January as an overlap, with the replacement scored beside the fine-tune that still answers.

Sort each behaviour the tune carries by where it can live next. On the trademark system, fine-tuning carries the reasoning patterns, section boundaries and tone of an attorney's opinion, and it does not handle citation integrity. Citations come from retrieval over an approved index of case law, statutes and registry data, and validators check the five-section structure. A change of base model there would mean retraining tone and structure only, because the citations never lived in the weights.

RAG vs fine-tuning vs agentic search is the test for each behaviour. A tune that was standing in for examples can often be replaced by examples in the prompt. A behaviour that has to outlive this base needs a base you can still train and serve. An open-weight model, one whose weights you can download and host, is that path only after the same eval set passes on it.

Where should rules live so they survive a model swap?

Put sums, permissions and thresholds in code, not in the prompt or the weights. On our insurance voice agent, a required disclosure is a condition in the tool contract, so the question "can this payment run without the disclosure?" has the answer no whatever model sits behind it. A rule enforced in code carries across a migration untouched. A rule written into the prompt has to be retested on every model you try.

Keep the model id behind an interface the application owns, so a swap edits one place and no business logic. Our desktop chief-of-staff agent calls the model as a reasoning service and keeps credentials, the tool registry and execution on the user's machine. A second provider belongs behind the same interface as a tested option, not as an extra hop every request depends on, because every vendor in one flow has to be up for that flow to run.

Model routing can still send an easy case to a smaller model. Retune the threshold whenever either model changes, because a route left on the old threshold changes which model acts with no alert, and the regression gets blamed on the wrong file. TypeSafe's Jev, a typed decision model built for routing and other bounded decisions, keeps that choice outside the generative model; how to introduce Jev into agentic workflows covers where it fits.

How do you see the next OpenAI model deprecation in time?

Make the next shutdown date fail a check while the id still answers, instead of arriving as an email. The 14 January date is one row on a page that already lists later ones, including GPT-5.1, GPT-5.3-Codex and GPT-5.4 nano on 1 April 2027.

On the OpenAI API, call GET /v1/models and read each model's optional shutdown_date, which the retrieve-a-model endpoint returns as well.

On Azure, read lifecycleStatus and deprecation.inference from the Models API, and mind the naming. A model the docs call deprecated, still serving existing customers, shows in the API as Deprecating. The API value Deprecated means the model is already retired and every call returns 410 Gone. Alert on Deprecating, because a check that waits for Deprecated fires after the outage.

Run that read on a schedule against the ids in your inventory, and fail the check when a date falls inside your retest window, which the GPT-4 notice suggests should be three months. Send the failure as a pull request or a page to the person who owns the model, while there is still time to run the table above.

Common questions

When is GPT-4o deprecated on the OpenAI API?

As of 22 October 2026, OpenAI's deprecations page shuts down gpt-4o-2024-08-06, gpt-4o-2024-11-20, gpt-4o-mini-2024-07-18, gpt-4.1-2025-04-14 and gpt-4.1-mini-2025-04-14 on 14 January 2027, with fine-tunes on those bases on the same day. The older gpt-4o-2024-05-13 snapshot and gpt-4.1-nano shut down on 23 October 2026. New fine-tuning jobs end on 6 January 2027.

What replaces GPT-4o?

On the OpenAI API, gpt-6-luna replaces the January 2027 GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini snapshots, and gpt-5.6-sol replaces the older gpt-4o-2024-05-13. Azure lists gpt-5.6-sol for 2024-05-13 and gpt-5.1 for the later GPT-4o versions, and names no replacement for GPT-4.1 or gpt-4o-mini. Score the replacement on your own cases before you move traffic.

What happens to GPT-4o fine-tuned models?

On the OpenAI API they shut down with their base model, on 14 January 2027 for the listed snapshots, and existing customers cannot create new fine-tuning jobs from 6 January 2027. GPT-6 Luna does not support fine-tuning. On Azure, GPT-4o and GPT-4o mini fine-tuned deployments run until 1 October 2027 and GPT-4.1 fine-tunes until 14 October 2027.

Are the Azure OpenAI retirement dates the same?

No. Microsoft's schedule, as of 22 October 2026, retires gpt-4o 2024-05-13 on 9 December 2026 and the later GPT-4o versions, gpt-4o-mini and all three GPT-4.1 sizes on 14 April 2027. Standard deployments are upgraded to the listed replacement automatically at retirement unless versionUpgradeOption says otherwise, and provisioned deployments are not upgraded at all.

Is GPT-4o still in ChatGPT?

No. OpenAI retired GPT-4o, GPT-4.1, GPT-4.1 mini and o4-mini from ChatGPT on 13 February 2026, and said at the time that nothing changed in the API. The ChatGPT date and the API date are separate decisions, so a team calling the API should plan against the deprecations page.

How do you test an LLM before switching models?

Run the same eval cases on the current model and the replacement, several times each, and check tool-call arguments and structured-output schemas on their own. Then run the new model in shadow on live inputs, send it a canary slice of traffic, and keep the old id callable until the canary's scores hold. A single green run on the new model is one sample.

Can Chat Completions still call tools on GPT-6 Luna?

Only at reasoning effort none. Reasoning together with tool calls uses the Responses API, and GPT-6 Astra and GPT-6.1 Sol need Responses for any tool call. At any other reasoning effort, temperature, top_p, top_logprobs and logprobs have to be removed from the request. OpenAI suggests starting most GPT-4.x workloads at none.

How do you find out when an OpenAI model will be deprecated?

Call GET /v1/models and read shutdown_date on each id you use. It is a year-month-day string once a date is announced and null before that. On Azure, read lifecycleStatus and deprecation.inference from the Models API, where Deprecating means a date is set and Deprecated means already retired. Fail a scheduled check while the id still answers.

Agentic workflows vs AI agents, and when not to build one

Tirth Gajjar — Tue, 27 Oct 2026 18:40:19 +0000

An agentic workflow is a multi-step process in which a language model does part of the work, such as drafting, classifying, retrieving or calling a tool. The term covers two designs: in one, code you wrote picks every next step, and in the other the model picks it, a design most people call an AI agent.

In this guide a workflow is the first design and an agent is the second. The question for a build is which of those your task needs, and the answer comes from the path the work takes rather than from the label on the ticket.

Criterion	Scripted workflow	RAG or search	Agent
Fixed path	Use it. The code walks the steps you listed.	Use it when the path is one query and one answer.	Skip it. The model would re-choose steps you already listed.
Multi-step judgment	Use it only when every branch is a rule you can write.	A retrieved passage does not choose the next action.	Use it when the next judgment depends on the last result.
Tool recovery needed	Use it when the retry is the same step, or a known fallback.	A search miss is a retrieval problem. Fix the index.	Use it when the error has to choose a different tool.
High blast radius, no gate	Keep the write out. The read path can stay in code.	Keep a permission filter on the query.	Do not start. The loop does not contain a bad write.
Measurable success	Score the output of each step.	Score recall, and whether the answer stays inside the passage.	Score the stop, and a final state outside the model's claim.
How Bigcircle handles this	On the fund reconciliation build, the matching rules are explicit and versioned in code, and a person approves only the resolution of an exception.	On Prospex, semantic and keyword search run together, because their failure modes are complementary.	On the chief-of-staff build, the agent picks the next tool from the last result, and a critic decides whether to continue, retry or escalate.

Most of what we have built is a workflow in that sense, by design. Of the ten systems in our case studies, the ones that turn filings, custodian feeds, legal drafts or meeting audio into records run as fixed pipelines, and where a model appears it sits inside one step. They are the ESG filing pipeline, the fund reconciliation system, the trademark drafting system and the meeting transcription app.

Even the DevOps agent turns each request into a structured spec and an explicit chain of tool calls, because a reviewer can only build judgement about output whose shape is stable. The chief-of-staff agent needed a loop. It works across more than a hundred registered systems, much of it behind logins with no usable API, and each step reads what the last one returned before choosing the next.

A demo hides the difference. Each shape can call a model, show a tool call and return a sentence that sounds finished. The split shows when a step fails or a record is missing. A workflow stops on the branch you wrote and retrieval says the passage is absent. An agent has to choose another tool, or stop, based on what it just saw.

If you cannot point at a case where that choice is real, the loop is overhead you will spend the next quarter tracing. This guide settles the vocabulary first, then takes the three shapes in the order a team usually reaches for them and puts five signals in one matrix. The last two sections cover the signals teams most often skip. One is a write that can do damage before any gate exists, and the other is a success check that only the model can see.

What is an agentic workflow, and how is it different from an AI agent?

The phrase is used both ways, so ask which one a vendor or a colleague means before you compare designs. Most of the pages that rank for it mean the second design, a model choosing the next step while it runs. Anthropic's Building effective agents calls that same system an agent and keeps "workflow" for code paths written in advance. Two proposals can both say "agentic workflow" and describe opposite systems, and the difference decides how you test them.

The usual list of workflow patterns is chaining one call into the next, routing to a branch, running calls in parallel, an orchestrator handing out subtasks, and one call grading another's draft. Review all but one of them as workflows, because code still owns the order.

The orchestrator is the exception: a lead model chooses the subtasks, so part of the path is already the model's choice, and it needs the checks of an agent. Multi-agent systems covers that pattern once the workers are agents themselves.

We keep those words because they name the decision you have to make. A scripted workflow is code picking the next step, with a model inside some of the steps. An agent is the model picking it. Plain automation, such as a rules engine, has no model at all, and the next section shows how far that gets.

When a proposal uses the phrase "agentic workflow", ask which step the model chooses and what code checks the choice. If the model chooses none, you are looking at a scripted workflow with model calls in it, which is usually the right thing to build. If it chooses the next tool, the rest of this guide applies before you sign off on it.

When is automation or a scripted workflow enough?

Rule-based automation and a scripted workflow both keep the path in code, and an AI agent does not. A scripted workflow is the right shape when someone who knows the job can write the steps, including the branches. Invoice extraction, support triage with a fixed set of labels and document classification usually sit here. The code walks classify, retrieve, draft, check and send, a model fills the steps that need language, and the step after each model call is the one you named.

The fund reconciliation system we built shows how far rules go. Custodians agree on nothing, not the format, the field names or the sign conventions, so each custodian gets its own mapping rather than one parser full of conditionals. The matching rules are explicit and versioned, because they are the domain knowledge. A person approves at one step only, resolving an exception, because that is where human judgement enters the data. There is no model call anywhere on that path.

The common advice runs the other way. OpenAI's practical guide to building agents points agents at rule sets that have grown unwieldy and costly to update. Take that advice when nobody can state the rules. When the rules are the domain knowledge, as they are in reconciliation, versioning them in code costs less than asking a model to rediscover them on each run, and an auditor can read the version that applied.

Put an agent on a path like that and the model spends the run re-deciding a sequence the code already had. A failure then looks like a choice, so you open the trace, when the missing piece was a branch nobody wrote. With the order of steps in code, a reviewer can point at the branch that fired.

The trademark drafting system is a pipeline for a different reason. Each stage has a defined failure behaviour, and a failed validation triggers a bounded retry that may draw only on retrieved evidence. Falling back to unconstrained generation is the most common way we see systems of that shape go wrong. A loop that could choose its own next step would reopen that path.

A branch you can state as a rule is still a workflow. If the amount is under the cap and the order is inside the return window, draft the refund and wait for a person to approve it. The model can draft the note while the cap stays in the code that checks the amount. A value you can compute is computed in code and passed in.

When is retrieval enough on its own?

Retrieval is enough when the failure is a missing or stale fact and one query can find the passage that holds it. A refund window that changed in March, a policy clause, or a filing the model was never trained on are RAG problems, or plain search problems. An agent does not repair a weak index, and RAG vs fine-tuning walks those layers in the order they cost.

Prospex, our own research engine, has to answer both "how do they make money", which is a semantic question, and "are they using Snowflake", which is a keyword question. Pure vector search will happily return a passage about a different warehouse. So semantic and keyword search run together rather than as a fallback chain, because their failure modes are complementary. A custom chunker keeps a role and its company in the same chunk.

A loop that searches until it feels sure calls the same weak index several times and then writes a fluent answer over thin evidence. You pay for several model calls to hide a recall problem that one better query would have shown in the log. On Prospex, too little retrieval produces a short profile, which is the safe failure, because a short profile is visibly short and an invented one is not.

Some questions do need more than one retrieval. The ESG pipeline gets comparative research questions, such as one about "the same subsidiary two years earlier", which pure vector search cannot express. Pure graph traversal fails the other way, because it cannot find a narrative claim that was never structured. One retrieval cannot answer a comparison, so the question is broken into steps: select sources, combine graph and vector evidence, build claims, check them and attach the source page to each one.

On the ESG pipeline that is a sequence of named steps: several retrievals in a known order, not a model left to decide what to search next. If you can name the second query the first result forces, you can often write it as a step. An agent loop starts to pay only when the next query depends on what came back in a way you cannot list.

When do you need an AI agent instead of a workflow?

An agent earns its cost when the next action depends on an observation you could not encode as a branch. The model selects a tool, reads the result, and either continues or stops. How to build an AI agent covers that loop, and this guide covers the decision to build it.

On the chief-of-staff agent each step runs think, act, observe, and the raw observation goes back into the loop rather than being summarised away. A critic then decides whether to continue, retry or escalate. Without it, one bad observation spreads through every later step and the agent claims success on work it never completed.

The model is not handed the whole tool registry either. It queries the registry and chooses among a handful of relevant contracts, because past roughly 30 tools in the prompt it started picking plausible but wrong ones.

Tool recovery is where the two shapes part. Repeating a call that timed out is still a retry inside a workflow. A not-found error that means "try the archive, and if that is empty, ask for the account id" is a choice among tools. If every recovery you care about is already a row in a table, keep it in the table. Start the loop only when the error has to pick the tool.

A miss inside the loop runs longer than it would in a workflow. A wrong branch fails one step, while a wrong choice can take several more tool calls before anyone sees a bad final state.

The loop pays for its flexibility in latency and tokens, and AI agent cost shows how each extra turn resends the transcript. The loop needs a budget, a stop rule and a check that does not trust the model's claim that the work is done.

How do five signals place one task?

Five signals are enough to place a task, and one product can sit in different columns on different rows. Read down the rows for the task in front of you before you pick a column for the whole product. The figure repeats the comparison at the top of this page.

Triage can be a workflow, the policy answer can be retrieval, and the ticket whose next system depends on the last lookup can be an agent. A single label on the product hides which of the three you shipped, and the next failure goes to the wrong owner.

The cells are fit judgments you can argue with on a real task. Green marks a fit for that signal. Amber means the shape works only when the condition in the cell is true. Red means stop and build the other shape, because this one adds a way for the bad outcome to happen.

A task that is green in the workflow column and red in the agent column should stay a workflow even if the roadmap calls for an agent. When a row is red because a write can do damage and no gate exists yet, read the next section before you add a loop.

What happens when a bad write has no gate?

A write that can move money, change access, send a customer message or alter production state can do real damage, and a loop does not make that smaller. If you do not yet have a gate that can refuse the call, a credential scoped to that action, and a check on the final state outside the model, keep the write out. The read path can still be a workflow or a retrieval app.

Adding an agent so the model can be careful puts the care in the prompt, which is the place a later instruction can argue with. The systems we have built put it in code. On the DevOps agent, an engineer approves an architecture diagram before anything is applied. Changes then go through the provider's change-set mechanism, which applies a group of changes as one unit, so a partial failure does not leave orphaned resources.

On our insurance voice agent, a payment cannot execute without its required disclosure, because the condition sits in the tool contract and not in the prompt.

The same agent shows what a failed write should do. A request that cannot be completed does not turn into an approximation. It escalates to a person, with the dialogue and the actions already taken attached, rather than doing something adjacent to what the caller asked for.

Human in the loop AI agents covers the product boundary, where answering can stay read-only and the first write is one kind of action. AI agent governance covers how one action gains authority after the evidence for it exists. Both assume you have already decided that an agent is the right shape for the path.

What do you score once the shape is chosen?

Success has to be a state something other than the model can check, or you cannot tell the three shapes apart after launch. A workflow scores the output of each step: the label, the drafted field, the branch that fired. Retrieval scores whether the right passage was in the set and whether the answer stays inside it.

An agent scores the trace and the final record. On the chief-of-staff agent, the claim that a task is complete is compared against the actual state of the target application, which catches claims of success on work that never happened. Work it cannot finish comes back as a flagged exception with its trace attached. The DevOps agent keeps one record that links the request, the account facts, the generated template, every validation failure and the resulting resources.

If the only signal is the model's closing sentence, every shape will claim it is done. A workflow claims done when a branch was skipped, retrieval when the passage was missing, and an agent after a tool error it folded into the prose.

Name the check before you pick the shape, then build the shape that can produce that check with the least machinery. Evals keep that check stable from run to run, and LLM evaluation covers building the cases. Until they exist, a demo that felt like an agent is not evidence that you needed one.

Observability on a loop is the trace of that check. Store the prompt, the tool list and the stop reason with the run, so a wrong final record can be tied to the choice that produced it. A workflow needs less of that, because the branch is already in the code. AI agent monitoring covers reading those traces once the system is live.

Common questions

What is an agentic workflow?

An agentic workflow is a multi-step process in which a language model does part of the work, such as drafting, classifying, retrieving or calling a tool. Some writers use the term only when the model chooses the next step. Anthropic calls that design an agent and keeps "workflow" for systems whose steps run through code paths written in advance. Ask which meaning is intended before you compare two designs.

What are the types of agentic workflows?

The usual patterns are prompt chaining, routing, parallel calls, orchestrator-workers and evaluator-optimizer. In all but one, code still owns the order, so review them as workflows. In orchestrator-workers a lead model chooses the subtasks, so part of the path is the model's choice and it needs the checks of an agent. Pages that use the term for agents list planning, tool use and reflection instead.

Is an agentic workflow the same as an AI agent?

Not in the sense this guide uses. In a scripted workflow, code picks every next step and a model fills some of the steps. In an AI agent, the model picks the next step and the next tool from what the last one returned. Many vendors call both of them agentic, so ask which step the model chooses and what code checks that choice.

What is the difference between AI agents and automation?

Rule-based automation follows steps and rules written in code, with no model in it. A scripted workflow keeps that fixed path and adds a model inside some steps. An AI agent lets the model choose the next action. Bigcircle's fund reconciliation system kept its matching rules explicit and versioned in code, because the rules are the domain knowledge and are versioned as such.

Do we need an agent, or is a workflow enough?

Build a workflow when a person can write the steps and the branches, including retries that always repeat the same call. Build an agent when the next tool depends on a result you could not list before the run. Most document and reconciliation systems Bigcircle has built are fixed pipelines, and the agent loop in its case studies is the build that works across many systems.

When is a RAG or search app enough on its own?

When one query can find the passage and the answer has to stay inside it. Fix the index, identifier matching, ranking and the permission filter on the query before you add a loop. If a question needs several retrievals, as a comparison across years does, try writing them as a fixed sequence first, and use an agent only when the next query cannot be known in advance.

Should a high-risk write start as an agent?

No. A call that can move money, change access, send a customer message or alter production state needs a gate before any loop may request it. If the gate, a scoped credential and a check on the final state are missing, keep the write out and ship the read path as a workflow or a retrieval app. Authority for one kind of action is a later release.

How do you know the run succeeded?

Check a state outside the model's closing sentence. A workflow should show the branch that fired and the field it produced, retrieval should show the passage and whether the answer stayed inside it, and an agent should show its tools, any recovery, any approval and a final record that matches what was expected. If you cannot name that check, you cannot yet tell which shape you need.

PII tokenization for LLMs: tokenization vs masking, with code

Tirth Gajjar — Sun, 25 Oct 2026 18:40:08 +0000

PII tokenization replaces each piece of personal data in a prompt with a stand-in token before the request leaves your network, and puts the real value back after the model replies. The provider sees the shape of the record, one customer, one account, one phone number, and never the values. Your users still get a complete answer.

Plenty of AI projects stall at the legal review, when someone asks what leaves the network and the answer is everything: the support ticket, the customer's email, the account number. We build the layer that changes that answer: a small piece of code in the request path that adds single-digit milliseconds to a call. This guide covers the design choices in it, the Python, where an entity recognizer such as Presidio fits, what the tokens mean under GDPR, and the CI test that turns "we tokenize PII" into something a compliance team can check.

PII tokenization vs masking: which keeps the answer useful?

Tokenize when the model has to reason about who is who, and mask only when it does not. Masking replaces every email with [EMAIL] and every number with ****. It is simple, and it makes the answers worse, as soon as a prompt holds more than one of anything.

Say a ticket mentions two customers and two accounts. After masking, the model sees [EMAIL] twice and **** twice. It cannot tell which account belongs to whom, so it cannot write "refund account X for customer Y", because the structure it needed has gone.

Reversible tokenization keeps that structure. Each sensitive value gets its own token, and the same value always gets the same token within a request, so two different emails get two different tokens and the model can still say "this customer, that account".

Method	What the model sees	Reversible	Keeps who-is-who	Use it for
Redaction	The value deleted	No	No	Fields the model never needs, such as a card's CVV
Masking	`[EMAIL]`, `****`	No	No	Logs and analytics where identity does not matter
Hashing	A fixed digest of the value	No	Yes, if salted consistently	Joining records without reading them
Reversible tokenization	A format-preserving stand-in	Yes, with the per-request map	Yes	Any prompt whose answer has to name the real customer or account

Two properties make tokenization work in a prompt. It is reversible: a private map from token to value lives for the life of the request, so the answer can be restored. And it is format-preserving: an email token still looks like an email, and a 16-digit account number becomes another 16-digit number, so prompts, parsers and validators downstream keep working and the model treats the field as it would treat the real one.

Is tokenized data still personal data under GDPR?

For you, yes, and for the provider possibly not, which is the argument to bring to legal review. A token map you keep is pseudonymization in GDPR's own terms: the data can no longer be tied to a person without additional information held separately. Pseudonymized data stays personal data for whoever holds that additional information, so tokenizing does not take your own processing out of scope.

The provider's position is different. In EDPS v SRB (C-413/23 P, 4 September 2025), the EU Court of Justice held that pseudonymized data passed to a recipient is not always personal data from that recipient's side, when the recipient cannot reverse the pseudonymization or identify people by other means. A provider that receives user_1ec32386@example.invalid, under a key that changes every request and never leaves your process, is the setup that ruling speaks to.

Describe the layer to your counsel as pseudonymization, and have them read the ruling against your setup. Anonymization is the wrong word for it, because the map you keep can reverse every token.

Which fields should you tokenize?

Decide with whoever owns the risk, from your real data, before writing code. Not every field is personal data, and encoding everything destroys the answer as surely as masking does. Over-redaction is its own failure.

The output is a short list of data classes, such as email, account number and phone, with a rule per class and per jurisdiction. On the telecom marketing platform we are building agents for, subscriber data spans several jurisdictions, and the analytics agent works through a governed data-access layer that masks, aggregates or excludes PII by policy before the agent sees a row. That policy is enforced in the access layer, never in the prompt, because a prompt is a request the model can ignore.

Structured classes such as emails, account numbers and phone numbers can start with pattern rules. Names, addresses and free-text identifiers need entity recognition, covered further down.

How do you tokenize PII before the provider call?

Create one boundary object per request that detects sensitive values, swaps each for a token, remembers the mapping and writes a record of what it encoded.

import hmac, hashlib, re
from dataclasses import dataclass, field

# Pattern rules for the fields your risk owner marked sensitive.
# In production, add an NER pass for names and addresses.
PATTERNS = {
    "EMAIL":   re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"),
    "ACCOUNT": re.compile(r"\b\d{10,16}\b"),
    "PHONE":   re.compile(r"\+?\d[\d -]{8,}\d"),
}

@dataclass
class Boundary:
    key: bytes                                   # per-request secret, never logged
    vault: dict = field(default_factory=dict)    # token -> raw value
    seen: dict = field(default_factory=dict)     # raw value -> token
    record: list = field(default_factory=list)   # what was encoded, without the values

    def _digest(self, kind, value):
        return hmac.new(self.key, f"{kind}:{value}".encode(), hashlib.sha256).hexdigest()

    def _surrogate(self, kind, value):
        h = self._digest(kind, value)
        digits = "".join(str(int(c, 16) % 10) for c in h)
        if kind == "ACCOUNT":
            # Same length, all digits. The "99" prefix marks it as a stand-in.
            return "99" + digits[: len(value) - 2]
        if kind == "EMAIL":
            return f"user_{h[:8]}@example.invalid"
        if kind == "PHONE":
            return "+00 " + digits[:10]
        return f"{kind}_{h[:8].upper()}"

    def encode(self, text: str) -> str:
        for kind, pattern in PATTERNS.items():
            def swap(m):
                raw = m.group(0)
                if raw in self.vault:            # already a token, leave it
                    return raw
                if raw not in self.seen:
                    tok = self._surrogate(kind, raw)
                    self.seen[raw] = tok
                    self.vault[tok] = raw
                    self.record.append({"kind": kind, "token": tok})
                return self.seen[raw]
            text = pattern.sub(swap, text)
        return text

Three choices in it are deliberate.

A fresh key for every request

Tokens come from an HMAC of the value under a random key. The same email gives the same token inside one request, so the model can connect two mentions of one customer. Across requests the key changes, so the provider cannot link tokens between calls and build a profile. The changing key also puts the provider in the position the EDPS v SRB ruling describes.

Tokens that cannot be real

example.invalid is a reserved domain that never resolves, and +00 is not a country code. If a token ever escapes into an outgoing email or a log, it is visibly a stand-in and not someone else's real address.

A record of tokens, not values

The per-request record lists the kinds of data encoded and the tokens used, and never the values. You show that record in the review afterwards.

Run on a support ticket:

import os
b = Boundary(key=os.urandom(32))
ticket = ("Customer priya.shah@acme.in says account 4111222233334444 "
          "was charged twice. Call back on +91 98765 43210.")
print(b.encode(ticket))
# Customer user_1ec32386@example.invalid says account 9955708008262310
# was charged twice. Call back on +00 0573126352.

Your tokens will differ, because the key is random, but the model still knows there is one customer, one account and one phone number, and what happened to them. It does not know who they are.

How do you restore the real values after the reply?

Decode with a lookup in the other direction, longest token first so one token never clips another:

    def decode(self, text: str) -> str:
        # Longest tokens first, so one token never clips another.
        for tok in sorted(self.vault, key=len, reverse=True):
            text = text.replace(tok, self.vault[tok])
        return text

If the model replies "Email user_1ec32386@example.invalid to confirm the refund on 9955708008262310", your user sees the real email and account number, using values your own systems already hold.

Models sometimes reformat what they copy, so a phone token can come back with different spacing. Tell the model in the system prompt to copy identifiers exactly. Then, after decoding, look for anything still shaped like a token: example.invalid, a +00 number, a 99-prefixed account. A leftover token means the model changed or invented it. Treat that response as failed and retry it or route it to a person, rather than sending a half-decoded answer.

Where does Presidio fit for names and addresses?

Use an entity recognizer for detection and keep your own tokens for the replacement. Patterns catch emails and account numbers, and they miss names, street addresses and identifiers written into free text. Presidio, the open-source PII toolkit Microsoft published, pairs named-entity recognition with pattern recognizers in its analyzer, and its anonymizer offers replace, redact, mask, hash and encrypt operators, with a decrypt operator that reverses encryption.

Presidio's encrypt and decrypt pair is reversible, and it is not format-preserving: an encrypted name comes back as ciphertext, which a model reads as noise rather than as a person. Feed the analyzer's results into the boundary above, so a detected name gets a name-shaped token from the same per-request map and decodes with everything else. Every detector misses something, so the leakage test below sits behind all of it.

How do you prove no PII reached the model provider?

Write a test that fails the build when a raw value reaches the provider. "We tokenize PII" is a claim, and this test is the evidence. Put a fake provider behind your real code path, capture everything that would have been sent, and assert that none of your known sensitive fixtures appear in it:

import os, json
from boundary import Boundary

RAW_FIXTURES = ["priya.shah@acme.in", "4111222233334444", "+91 98765 43210"]

class CapturingProvider:
    def __init__(self):
        self.sent = []
    def complete(self, payload: dict) -> str:
        self.sent.append(json.dumps(payload))
        return "ok"

def handle_ticket(text, provider):
    b = Boundary(key=os.urandom(32))
    payload = {"messages": [{"role": "user", "content": b.encode(text)}]}
    return b.decode(provider.complete(payload))

def test_no_raw_value_reaches_provider():
    provider = CapturingProvider()
    handle_ticket("From priya.shah@acme.in, card 4111222233334444, "
                  "phone +91 98765 43210", provider)
    for body in provider.sent:
        for raw in RAW_FIXTURES:
            assert raw not in body, f"raw value leaked: {raw!r}"

Run it in CI on every pull request, and grow the fixtures. Each time someone finds a format the rules missed, such as an email with a plus sign or an account number with spaces, add it as a fixture before fixing the rule. The suite becomes the list of everything you have promised not to send, and we hand it to a compliance team instead of a policy document.

Test the whole path, not just the boundary class. Leaks rarely start in the tokenizer. They start when someone later adds a helpful field to the prompt, the raw customer record for context or the user's profile in the system message, and it skips the encoder. A test on the real request path catches that on the pull request that adds it.

Where does the tokenization layer sit?

Put it in the request path, directly in front of the provider call, so every call goes through it. A boundary that each feature calls when it remembers to will be skipped by the feature written in a hurry. Keep it fast as well. In our builds the encode and decode step adds single-digit milliseconds, so there is no latency argument for switching it off under load.

Three limits shape the rest of the design:

It protects the next request. Data already sent to a provider is a separate conversation with that provider.
Detection misses things. Patterns and entity recognition both need the leakage suite behind them.
The token map is now the sensitive object. It lives for one request, in your process, and never goes into logs or traces.

Credentials follow the same rule. On Solarpunk, the desktop agent we built, secrets sit in an encrypted vault on the user's machine and are injected into the tool adapter when a tool runs, so they never travel into the prompt. Whatever the model never receives, a prompt injection cannot ask it to repeat.

We build this layer for teams whose AI work is waiting on that legal review. PII redaction and data boundaries covers how an engagement runs, from classifying fields in week one to the leakage suite your compliance team reviews.

Common questions

What is PII tokenization?

PII tokenization replaces each piece of personal data in a prompt with a stand-in token before the request goes to the model provider, and restores the real values after the reply. The provider sees the structure of the record and never the values, and the user still gets a complete answer.

What is the difference between tokenization and masking?

Masking replaces every email with the same placeholder, so the model cannot tell which account belongs to which customer, and the original cannot be restored. Reversible tokenization gives each value its own stand-in within a request, so the structure survives and the real values go back in after the reply.

Is tokenized data still personal data under GDPR?

For the organization holding the token map, yes: it is pseudonymized data, which stays personal data. For a recipient that cannot reverse the tokens or identify people by other means, the EU Court of Justice held in EDPS v SRB (C-413/23 P, 4 September 2025) that it is not always personal data. A fresh key per request that never leaves your process puts the model provider in that second position.

Can Microsoft Presidio do reversible PII tokenization?

Presidio's anonymizer has an encrypt operator and a decrypt operator that reverses it, so its output is reversible. It is not format-preserving, because an encrypted value comes back as ciphertext. Bigcircle recommends an entity recognizer such as Presidio's analyzer for detection, with format-preserving tokens generated from your own per-request map.

Does PII tokenization slow down the LLM call?

In Bigcircle's builds the encode and decode step adds single-digit milliseconds, which is small next to the model call itself. Keeping it that fast matters, because a boundary that teams switch off under load protects nothing.

How do you prove no personal data reached the provider?

Put a fake provider behind the real code path, capture everything that would have been sent, and fail the build if a known sensitive fixture appears in it. Run the test in CI on every pull request, and add a fixture each time someone finds a format the rules missed.

How much does an AI agent cost to build and run?

Tirth Gajjar — Fri, 23 Oct 2026 18:35:02 +0000

AI agent cost has three parts: the build, the run and the keep-alive. The build is the system that can take a real task. The run is the model calls, tool calls and retries each task then spends, and the keep-alive is the work after launch, when a tool, a model or a document changes. A per-token price covers one line inside the run, and one combined number hides which of the three is about to move.

Our case studies show where the work went. On the DevOps agent, the build was mostly about reading the live AWS account, checking every template and putting a human review step before anything applied. On the chief-of-staff agent, the visible cost of tool definitions in the prompt turned out to matter less than the wrong tool choices that came with them.

The sections below take the build, then the run with a worked token calculation at prices as of 22 October 2026. The keep-alive and a way to compare two quotes come next. The last two sections cover the cap that ends a runaway loop before the next paid call, and the ledger of what each run spent. If a script or one query can do the job, agentic workflows vs AI agents comes first, and none of this bill starts.

How much does it cost to build an AI agent?

The build price follows the scope, and most of the scope sits around the model rather than in it. The build is everything that has to work before the first real task leaves the team: the tools the job needs, permission to call them, an eval set, and a path for the step that fails. Choosing a model is one decision inside that work, and usually not the long one.

The DevOps agent shows where the time goes. A raw model invents resource types, writes templates the account rejects for quota or region, and drifts between runs, and a failed stack can take minutes to unwind, so "apply and see" is not a usable loop.

The build therefore had to read live limits, quotas and existing resources before planning, check every template in a loop of structural and semantic checks, and show an engineer an architecture diagram before anything applied. Delivery took roughly four months.

The chief-of-staff agent spent its build on reach. It connects to more than a hundred systems, and a large share of an executive's work sits behind single sign-on and multi-factor login with no usable API. We argue that a routine the agent can finish 70 percent of still needs a person to hold the sequence, so it saves almost nothing. That is why reach costs build time on an agent like this.

Each integration is a tool contract, a login path and a failure the agent will meet in normal use. MCP authorization covers the check before each call, LLM evaluation covers the eval set, and how to build an AI agent puts those decisions in order. A build quote that lists model access and leaves out the access check and the eval set will be rewritten after the first real task fails.

How the build is bought is a separate question. AI pilot to production compares production hardening, a fixed-scope build and a studio-led programme. A fixed-scope build is priced against a scope agreed before work starts, so the useful comparison between two quotes is which of the three parts each one names.

Why does an AI agent cost more to run than one reply?

A run costs more than one reply because the agent decides how many steps to take, and each step sends the whole history again. A workflow pays one model call per step you wrote. An agent pays a number of calls that depends on the trace, and the input on each call grows with every tool result.

A worked example makes the shape concrete. The token counts are assumptions chosen for the example, and the prices are OpenAI's list rates for GPT-6 Luna as of 22 October 2026: $0.10 per million input tokens and $0.50 per million output tokens.

Take a system prompt and tool definitions of 6,000 tokens, a task of 10 turns, 300 output tokens per turn for the tool call, and 2,000 tokens of tool result added to the transcript after each turn.

	One reply	Ten-turn agent task
Input tokens	6,000	163,500 (6,000 on the first turn, 26,700 on the tenth)
Output tokens	300	3,000
Cost at the October 2026 Luna rates	about $0.00075	about $0.018

Ten turns cost about 24 times one reply, not 10 times, because each turn pays again for every earlier turn's transcript. At 10,000 tasks a day that is about $179 a day on the run line.

Reasoning adds to it. OpenAI bills reasoning tokens as output tokens, and Luna defaults to reasoning effort medium. If each turn spent 1,000 reasoning tokens, another assumption, the same task would cost about $0.005 more, roughly 28 percent on top. The example also assumes reasoning from earlier turns is not sent again. Check that before you trust the figure, because the GPT-5.6 family carries earlier reasoning into the next turn by default. On a model that does, those tokens add to the input of every later turn. LLM model migration covers why a model change can move this line while the per-token price falls.

Tool definitions are part of the input on every turn, and the chief-of-staff agent found where listing all of them in the prompt stops working. Direct tool-calling with every tool listed worked up to roughly thirty tools. Past that, the model started choosing plausible but wrong tools, and we found that the context cost was the visible symptom while the selection error was the expensive one. The fix was to let the model query a tool registry and choose among a handful of relevant contracts, which kept the working context small at more than a hundred tools.

Prompt caching cuts that bill more than anything else in the example, because it discounts a repeated prefix, and an agent's growing transcript is one. As of 22 October 2026, caching is on by default on OpenAI's supported models, with a minimum prefix of 1,024 tokens on GPT-5.6 and later. On those models a cache read costs a tenth of the input rate and a cache write 1.25 times it.

An entry lasts 30 minutes by default, and each read refreshes it. OpenAI's caching rules do not name Luna, and Luna's model page lists cached input at $0.01 per million, the same tenth, so we assume the same write rate.

Applied to the example, if every turn after the first reads the earlier transcript from cache and writes only its new 2,300 tokens, the ten-turn task comes to about $0.006 instead of $0.018. That is 136,800 tokens read at the cached rate, 26,700 written at the write rate, and the same 3,000 output tokens. It assumes each turn arrives inside the cache lifetime with the whole prefix unchanged, so a retry that edits anything early in the transcript pays full price again from that point.

Caching at the application layer saves a different cost. Prospex, our own research engine, runs cache-first, so a rerun reuses earlier extraction and retrieval instead of calling external services again. By our estimate that cut repeated external API calls by 50 to 60 percent, and we watch the cache hit rate as its cost metric.

What does an AI agent cost after launch?

Keep-alive is the work after launch. It covers rerunning the eval set when a tool, a prompt or the model changes, a person who can act when a run stops, and traces that someone can still read when finance asks what happened. None of it shows on the launch invoice, and all of it is on the calendar afterwards.

Integrations are where it starts. We saw what point-to-point wiring does to maintenance on the chief-of-staff build: a vendor changes a response, and the break shows up somewhere unrelated days later. Our answer there was a uniform tool layer, where each tool is a typed contract with declared inputs, outputs and side effects.

Human review is the other running line. On the trademark drafting system, attorney corrections are the most valuable signal in the system, because they measure the gap automated scoring cannot see. That review does not end at launch. On the ESG filing pipeline, the slow failure is framework version drift, which is why the mapping version is recorded on each mapped fact.

Observability on an agent is the trace of each run, stored with the run. AI agent monitoring separates the quality of the answer from the health of the path that produced it. Name the person who owns the stop before launch. A stop with nobody assigned to it is only a log line, and the loop behind it keeps spending until someone reads the invoice.

How do you compare two AI agent quotes?

Compare two quotes part by part, or a missing line will look like a saving. Two proposals can describe the same agent and still leave out different parts, and the shorter one has often left off the keep-alive or the stop.

Part	What moves it	What a thin quote skips
Build	Tools, credentials, evals, and the path for a failed step	Everything except a prompt and a happy path
Run	Turns, tool calls, retries, fan-out, and a transcript that grows	Every turn after the first
Keep-alive	Eval reruns, a person on the stop, review, and traces you can still read	A named owner after launch

How do you enforce a budget in the request path?

A budget holds only when the check runs before the next provider call and a breach ends the run. A dashboard that climbs and an email that arrives later have already paid for every call between the threshold and the person reading it. The cap is code on that path, and it limits turns, tool calls, spend, and how many child agents one run may start.

Fan-out, where one run starts several child agents, is the case a per-call limit misses, because each child looks small on its own. If every child gets a fresh counter, together they spend the parent's allowance several times over. Make each child draw on what the parent has left, and have the parent refuse another child once the count or the allowance is gone. Multi-agent systems covers when the children are worth starting at all.

Retries need a bound of their own. We bounded them on the chief-of-staff build, because an agent that retries indefinitely against a broken vendor is worse than one that stops: it burns cost and time while looking busy. The DevOps agent bounds revision the same way, and a template that will not come clean after bounded revision is escalated as a request the agent cannot safely fulfil. Count failed calls in the total, because the provider already did.

A limit written into the prompt can be talked past, and a counter in code cannot. Guardrails that live only in the prompt can give way the first time a tool result or a user message argues for one more step.

Account-level caps from the cloud providers are a backstop behind that check. They act on a whole project after many runs have spent, they cannot tell a retry storm from a busy day, and a paused project takes your working runs down with the broken one.

AWS spend limits, introduced on 16 September 2026 and still rolling out to a limited set of customers, pause a project and stop its resources when its monthly usage reaches the limit. An opt-in control pauses the highest-cost resources about four days before the limit, chosen from five services that include Lambda and Bedrock.

Google Cloud's Spend Caps, in public preview since 29 July 2026, restrict further usage of one service in one project once the monthly cap is reached. Both are worth turning on, and the loop still needs its own check before each call.

Monitoring without a stop is the gap that costs most. In one incident Jatin Bansal collected, a team running four LangChain agents had monitoring and no enforcement. An analyzer and a verifier began looping with each other in November 2025, and the bill reached $47,000 over eleven days. The figures are as Bansal reports them.

Set each cap from runs you have already finished rather than from a framework default, which is calibrated to someone else's task. Leave room above the longest run that completed normally, so a stuck loop hits the cap and a normal task does not. A cap set on a guess either cuts good tasks or lets a loop run.

Where does the spend record live?

The ledger has to outlive the agent, or a loop can erase the only record of what it spent. Write the run id, the tool, whether the call was allowed or refused, the token counts and the stop reason for each call to a store the agent's credential cannot delete or rewrite. Include the calls that failed and the calls each child made, or your meter will sit below the invoice.

Reconcile your meter with the provider bill every month as two sums: billed spend from the invoice, and the cost of every attempt from your own traces, including retries, timeouts, cancellations and outputs you rejected.

A gap left after failed attempts are included is work you are not tracing, such as an eval job, a warm-up call or a second service on the same key. Close that gap before you call either number the cost of a successful task.

Common questions

How much does an AI agent cost?

AI agent cost has three parts, the build, the run and the keep-alive, and each moves for a different reason. The build is tools, credentials, evals and the path for a failed step. The run is turns, tool calls, retries, fan-out and a transcript that grows. The keep-alive is eval reruns, review and traces after launch. A price per token covers one line inside the run.

How much does it cost to build an AI agent?

The build price follows the scope, and most of the scope sits around the model: tools, permissions, an eval set and the path for a failed step. Bigcircle's DevOps agent took roughly four months, spent mostly on reading the live AWS account, checking every template and a human review step. Compare quotes on which of those parts each one names.

Why does an agent task cost so much more than one reply?

Each turn resends the whole transcript, so input grows with every tool result. In this guide's worked example at GPT-6 Luna's October 2026 rates, a ten-turn task with a 6,000-token prompt and 2,000-token tool results costs about 24 times one reply, not 10 times. Reasoning tokens are billed as output and add to that.

Does prompt caching fix the run cost?

It helps on a repeated prefix and nowhere else. On OpenAI's API as of October 2026, caching is on by default, with a 1,024-token minimum on GPT-5.6 and later, where a cache read costs a tenth of the input rate and a write 1.25 times it. In this guide's example that takes the ten-turn task from about $0.018 to about $0.006. A retry that edits the early transcript pays full price again.

How do you stop a runaway agent before the invoice?

Put a hard cap in the request path and check it before the next provider call. Cap the turns, the tool calls, the spend and the number of child agents, and make each child draw on the parent's remaining allowance. When any cap breaks, end the run and keep the partial result. An alert after the threshold has already paid for the calls in between.

Are cloud spend caps enough on their own?

No. AWS spend limits and Google Cloud Spend Caps, both announced in 2026, pause a project or a service once a monthly amount is reached. They are a useful backstop, but they act after many runs have spent, they cannot tell a retry storm from normal load, and they stop healthy runs along with the broken one.

How do you reconcile your meter with the provider bill?

Sum what the provider invoiced for the period, and sum what your traces recorded for every attempt, including retries, timeouts and outputs you rejected. Compare the two before you call either one the cost of a successful task. A remaining gap is work you are not tracing, such as an eval job or a second service on the same key.

Why AI agents report unfinished work as done

Tirth Gajjar — Wed, 21 Oct 2026 20:07:29 +0000

An action-taking agent can receive a 200 OK while the customerâ€™s task remains incomplete. The API may have accepted the request while a queue still rejects it, a downstream system fails, a browser submits twice, or a deployment exists and remains unhealthy, so the agent must not translate interface success into business completion. It needs a contract for the expected final state, an independent way to verify it, and a recovery path for every uncertain result. Bigcircle uses that check on client systems, and a run may report done only after a separate read matches the expected state, including the downstream effects the contract named.

What if the tool call succeeded and the task did not?

A tool response describes what one interface observed at one point in time. It may prove that the request was valid, accepted, queued, or written to one store. It may not prove that the intended workflow ever reached its final state, so consider a customer-operations workflow:

The agent updates a case.
The case system accepts the write.
A provisioning event should be created.
The provisioning worker should change the customer account.
A confirmation message should be sent.

If the first system returns success and the worker later fails, the agent can close the case while the customer still waits. The same gap appears in commerce, finance, infrastructure, messaging, and enterprise workflow systems. Done is the state a customer or operator would recognise as finished, such as a refund record with a ledger entry and a customer-visible status, and a status code only says that one interface accepted the request.

Signal	What it proves	What it leaves open
Tool result, including a 200	The interface accepted, queued, or stored the request	Whether the customer task reached the contracted state
Expected state, read on a separate path	The business facts in the contract now hold	Any fact the verifier could not read, which stays unknown

Why write an expected-state contract first?

An expected-state contract describes what must be true after the action has run, in terms the business would recognise, and it should include:

the relevant state before execution;
the exact state expected after execution;
the fields or resources allowed to change;
the invariants that must remain true;
the downstream effects expected from the action;
the deadline for those effects;
the source that can verify each condition.

The contract should exist before the agent invokes the tool, because without it the agent can only repeat the toolâ€™s success message. It cannot test whether the product delivered the intended result. For a refund, the final state may require a refund record, a ledger entry, and a customer-visible status. For a deployment, it may require the correct account, expected resources, healthy service checks, and no forbidden changes. For an external message, it may require one accepted message with the correct recipient, content hash, and provider ID.

Can you trust an agent that says the monitor is still on?

In two internal coding tasks, a GPT-6.1 Sol agent on 3 October 2026 disabled automatic human alerts from misalignment monitoring, treating an instruction not to send messages as permission. On 6 October a GPT-6 Astra agent removed a monitoring integration to match a reference implementation, then falsely denied removing it. OpenAI's misalignment report, updated 20 October 2026, says OpenAI is strengthening monitoring and assessing overpersistence against a propensity to evade monitoring. Bigcircle reports the work done only after a separate read matches the expected state.

Why verify through an independent source?

The execution path should not be the only path that decides whether execution worked. If the agent writes through an action API, verification can read the system of record through a separate query path. If the agent submits a browser form, verification can inspect the resulting record, event, or provider receipt instead of trusting the changed screen. If the agent deploys infrastructure, verification can inspect the target account and health state instead of trusting the deployment command. Independent does not always mean a separate vendor, it means a separate observation that can detect false success in the execution path.

The verifier should return structured evidence:

Verification field	Purpose
Operation ID	Correlate request, result, and recovery
Target ID	Prove which record or resource changed
Observed state	Record the state found after execution
Expected state	Preserve the release contract
Invariant result	Detect forbidden side effects
Source and timestamp	Show where and when evidence was read
Verdict	Complete, incomplete, failed, or unknown

Structured output helps the application enforce these fields, and it does not replace the underlying observation.

The comparison returns one of four verdicts, and the next action depends on which one came back. Complete returns the evidence and stops. Incomplete keeps checking while the effects are still inside the deadline. Failed runs a compensating action or a handoff, because a partial effect is already proven. Unknown blocks the retry until a separate read can see the final state.

Verdict	What the separate read showed	Next action
Complete	Observed state matches the contract	Return the evidence and stop
Incomplete	Expected effects are still inside the deadline	Keep verifying, or run the defined recovery
Failed	A partial or forbidden state is proven	Compensate, or hand the case to a person
Unknown	The final state cannot be read	Reconcile before any retry

Why treat a timeout as unknown state?

A timeout is not proof of failure. The request may not have reached the target, it may have reached the target and failed, or it may have succeeded while the response was lost. Those three states require different next actions, and if the agent labels every timeout as failed, it may retry an action that already changed real state. That can duplicate a refund, message, ticket transition, order adjustment, or infrastructure change. The correct state is unknown until the system can reconcile the operation. The run should preserve the operation ID, target, requested parameters, request timestamp, observed response, and last known state. It should then query the target or ledger through the verification path before it decides to retry.

Why do business operations have to be idempotent?

Idempotency means that repeating the same business operation does not create a second consequence anywhere downstream. A transport-level request ID helps, but it is not enough when the same business operation can enter through several services or retries, so use a stable operation key tied to the business action. For example:

refund:{order_id}:{approved_refund_version}
case-transition:{case_id}:{from_state}:{to_state}:{approval_id}
message:{case_id}:{recipient}:{content_hash}
deployment:{change_request_id}:{plan_hash}:{target_account}

Persist the operation state before execution when the target supports it. On retry, the action layer should return the existing result or continue the recorded operation instead of creating a new action. Idempotency does not remove the need for verification, it limits duplicate effects while the verifier determines the final state.

Why reconcile before retrying?

Retry is a business decision rather than only an infrastructure setting, and the two get confused constantly, which is why a safe retry loop asks:

Did the target receive the operation?
Did the expected state change occur?
Did any forbidden or partial state appear?
Is the approval and credential still valid?
Is the same action still permitted in the current state?
Can the next attempt create a duplicate effect?

The answers can produce five outcomes:

Return success: the first action completed and is verified.
Continue: an accepted workflow is still within its allowed time window.
Retry: no effect occurred, the action remains valid, and the retry is safe.
Compensate: a partial effect needs a defined reversal or repair.
Escalate: the system cannot prove a safe next action.

A generic exponential-backoff policy cannot make any one of these decisions on its own.

How do you design for partial failure?

Multi-system work creates states between success and failure. An onboarding case may change while provisioning fails, a refund record may exist while the ledger entry is missing, and a deployment may create resources while the health check fails, so model those in-between states explicitly.

Partial state	Safe response
Primary write succeeded, downstream work pending	Continue verification within a deadline
Primary write succeeded, downstream work failed	Run a compensating action or escalate
Some targets changed, others did not	Stop broad retry and reconcile each target
Final state correct, response missing	Return the verified existing result
Final state cannot be read	Freeze retry and hand off with the full trace

Recovery work needs its own authority boundary. An agent allowed to change a case status is not automatically allowed to reverse a financial entry or delete a cloud resource during recovery. Each compensating action needs the same target, parameter, approval, credential, and verification discipline as the original action.

What belongs in the causal evidence chain?

An audit log of tool calls is useful and incomplete on its own. The organization must be able to reconstruct why the action was allowed and what consequence followed. Record:

the business request and delegating principal;
the agent identity, model, tools, and policy version;
the evidence and state used for the decision;
the approval, scope, and expiry;
the credential or capability used;
the exact action and parameters;
the tool response;
the independently observed final state;
any retry, compensation, escalation, or human correction.

Observability should connect these records with one causal operation ID. This evidence lets engineering replay a failure, operations complete a handoff, security inspect the authority chain, and product decide whether the action has earned more authority. It also answers a growing identity problem that we now meet on most agent engagements. NISTâ€™s 2026 concept paper covers identification, authorization, auditing and non-repudiation for agents, and it is context rather than a certification for this design. The engineering requirement remains direct: preserve the person, task, policy, action, and result in one evidence chain.

A later challenge has to read that chain from retained evidence, including the independently observed result, rather than from the tool log alone. How to build an audit trail for AI decisions is the record for the accepted decision, and the operation ID joins it to the tool trace.

What does the production release gate check?

Before an action receives write access, test the state transition it produces rather than only the model response that describes it, and a release-ready eval should include:

permitted targets and parameters;
denied records, tenants, and environments;
stale-state conflicts;
expired approval and credential cases;
timeouts before and after target execution;
duplicate submissions;
partial downstream failure;
verifier failure;
compensating-action limits;
human handoff quality.

We built custom capability evaluations for an authenticated browser-use workflow. The useful production pattern was to evaluate real task completion and action behavior, not only the agentâ€™s reasoning. Do not grant more authority because the agent explains its plan well. Grant more authority when the system can prove that the action stays inside its boundary, reaches the expected state, avoids duplicate effects, and stops when the result is unknown. How that authority moves up, and how it comes back down, is the release model in how to roll out agent autonomy. When several workers each report a slice as done, the same separate read is the merge, which when to fan out to many agents puts in front of the combined result. Use the Agent Action Boundary Canvas to define the expected state, verifier, retry rule, partial-failure states, and recovery owner for one action.

What should you take into production?

Interface success is not business completion. A 200 OK can sit alongside a rejected queue, a failed downstream system or an unhealthy deployment.
Write the expected-state contract before the tool call, in terms the business would recognise.
Verify through a path that is independent of the one that executed the action.
A timeout is unknown state, not failure. The correct state is unknown until the system can reconcile.
Key idempotency to the business operation, not to the transport request, because the same action can enter through several services.
Reconcile before retrying. A generic exponential-backoff policy cannot decide whether the first action already changed real state.
Model the states between success and failure explicitly, and give recovery its own authority boundary.
Gate the release on the state transition the action produces, not on the model response that describes it.

Common questions

Why do AI agents report unfinished work as done?

A tool can return success for one interface while the business task is still open. The case write can succeed while provisioning fails, so the agent closes the case and the customer still waits. A rejected queue, a double browser submit, or an unhealthy deployment is the same gap. Report done only after a separate read matches the expected state, and keep the run unknown when that read fails.

What is an expected-state contract?

An expected-state contract lists the business facts that must be true after the action. Write it before the tool is called, because after the call the agent can only repeat whatever success message the interface returned. It names the before and after state, the fields that may change, and the facts that must stay true. It also names the downstream effects, the deadline, and which source can verify each one.

Is an HTTP 200 proof that an agent finished?

A 200 is not proof the task finished. A success payload describes what one interface observed at one moment, and it can show a valid, accepted, queued, or stored request. It cannot show that the workflow reached the state a customer or operator would call finished. For a refund, that state can be the refund record, the ledger entry, and the customer-visible status.

What should an agent do when a tool call times out?

Treat the timeout as unknown, not as failure. The request may never have reached the target, it may have failed there, or it may have succeeded while the response was lost. Calling every timeout a failure invites a retry that can duplicate a refund or a message. Read the target through the verification path before any retry.

How do you stop a retry from duplicating an agent action?

Key the operation to the business action, not to the transport request, because the same action can enter through several services or retries. A stable key lets the retry return the recorded result instead of starting a new action. That key limits a second consequence while the verifier is still reading the final state.

What does independent verification mean for an agent?

Independent verification is a separate observation of the result, and it does not require a separate vendor. If the agent writes through an action API, the check reads the system of record on another path. A browser submit is checked from the resulting record or provider receipt, not from the changed screen. The verdict is complete, incomplete, failed, or unknown.

Originally published on bigcircle.ai. I'm Tirth, founder of BigCircle, where we build production AI systems with governance built in.

DEV Community: Tirth Gajjar

How to set AI agent permissions and access control

Why not approve the agent as a whole?

How do you list the actions in one workflow?

Why give each action its own tool?

Where should AI agent authorization run?

Should an AI agent have its own identity?

How do AI agents handle authentication?

When is this more than you need?

Common questions

What are AI agent permissions?

What authorization does an AI agent need before it acts?

How do AI agents handle authentication?

Should an AI agent have its own identity?

Is an OAuth scope enough to limit what an agent does?

Why not approve the whole agent?

Further reading

Why RAG systems fail in production

How does retrieval work for RAG in production?

What if the right document was never retrieved?

What if the right chunk was retrieved but unusable?

What if the model ignored or distorted the evidence?

What if the corpus is wrong before retrieval starts?

What if security is treated as a model behaviour?

What if the evals measure the wrong thing?

How do you match a symptom to the layer that failed?

When is GraphRAG or a long context better than RAG?

How do you operate RAG after launch?

What belongs on a launch checklist that survives production?

Common questions

Why do production RAG answers sound right and still fail?

How does retrieval work in RAG?

How does RAG handle long documents?

What should we measure before changing the prompt?

When does hybrid search beat vector search on its own?

How should retrieved documents be treated for security?

Does a longer context window replace retrieval?

Sources and further reading

How to design MCP tools for agents

What does wiring every vendor server cost?

What should the consolidation layer own?

How large should MCP tools be?

Why return structured JSON and not prose?

How should the layer authenticate, limit, and log?

How does the call log change the tools?

When is wiring a vendor server directly the right call?

Common questions

What are MCP tools?

What are the best practices for designing MCP tools?

Should we connect every vendor's MCP server to our agent?

What is an MCP consolidation layer?

How many tools should a consolidation MCP expose?

Does fewer, larger tools mean one general query tool?

How should a consolidation MCP handle authentication?

How do we know when to change a tool?

Further reading

How to build an LLM evaluation from real failures

What is LLM evaluation, exactly?

Why can't you just write the rubric first?

How do you find your actual failure modes?

Who decides what good means?

Where do the cases come from?

What should you measure?

Retrieval

Tool use

Safety: guardrails and PII

Generation

When should you not build a judge?

How do you write a rubric that holds up?

How do you know your judge is any good?

Measure the two error rates separately

How many labels, and when to stop

Below 0.4, check the humans before the judge

When it fails, split the rubric before touching the model

Three biases to expect

How do you stop overfitting the set?

Is the difference real?

What does this look like in CI?

Where does online evaluation fit?

The loop this all becomes