DEV Community: shakti mishra

Workload-Fit Routing: A Four-Decision Test for Assigning Models to Agentic Tasks

shakti mishra — Sat, 19 Sep 2026 14:51:58 +0000

Stop Picking a Model. Start Routing Workloads.

Procurement wants one model. Your architecture wants two.

That tension is where most enterprise AI programs quietly lose money. Someone runs a bake-off, picks a winner on aggregate benchmark scores, signs an enterprise agreement, and then routes every workload — spreadsheet reconciliation, CI/CD debugging, contract analysis, browser automation — through the same endpoint. The bill arrives in Q3 and nobody can explain it.

Here's the thing the benchmark leaderboard can't tell you: "best model" is not a property of a model. It's a property of a model plus a workload plus a tool surface plus an acceptance criterion.

Consider two requests hitting the same architect in the same week. Finance wants an agent that opens a sales spreadsheet, reconciles the numbers, builds a chart, updates a deck, and saves the document. Engineering wants an agent that investigates a recurring inventory-service bug, traces dependencies, proposes a fix, and verifies that the fix doesn't break a neighboring service.

Those look similar on a roadmap. They fail in completely different ways. The spreadsheet workflow fails when the agent writes to the wrong file. The debugging workflow fails when a plausible-looking patch hides the actual defect. In neither case is "produced a fast answer" the definition of done.

You're not hiring the smartest colleague. You're assigning responsibilities, tools, and acceptance criteria to two different jobs.

The reflex vs. the fix

Most teams run this loop:

REFLEX:  pick a model → give it every task → optimize the bill later
FIX:     define the job → test the model + tools → verify the outcome

The reflex isn't stupid. It's operationally cheap — one vendor, one key, one SDK, one set of rate limits. It only becomes expensive when the failure modes diverge, which is exactly what happens the moment your agents start acting instead of answering.

The fix requires you to score each workload on four decisions before any model gets assigned.

The four-decision routing test

1. Execution — can it complete the required actions across your tools?

Not "can it write code that would work." Can it drive your actual tool surface: your CRM API, your shell, your browser, your filesystem? This is an agentic capability question, and it's mostly orthogonal to reasoning quality.

2. Building — can it understand the problem and produce a correct, maintainable artifact?

This is the analytical axis. Does the output survive code review? Does the contract comparison hold up when a lawyer reads it? An artifact that's wrong in a subtle way is worse than no artifact, because it consumes review cycles and then fails in production anyway.

3. Time — does it meet the deadline for an accepted result?

Note the word accepted. Time-to-first-token is a demo metric. Time-to-accepted-outcome includes retries, tool errors, and the human who has to look at it. A model that answers in four seconds and gets rejected twice is slower than one that takes forty seconds and lands.

4. Cost — what does that accepted result cost, fully loaded?

cost_per_accepted_outcome =
    (input_tokens_uncached × rate_in)
  + (input_tokens_cached   × rate_cached)
  + (cache_write_tokens    × rate_cache_write)
  + (output_tokens         × rate_out)
  + (retry_count × full_attempt_cost)
  + (human_review_minutes  × loaded_hourly_rate / 60)
  ÷ acceptance_rate

That last division is the one everyone forgets. At a 70% acceptance rate, your real cost per outcome is 1.43× your per-attempt cost. Token price is maybe a third of the equation.

Reading benchmarks as capability signals, not rankings

The useful move isn't "which model is #1." It's "what does this specific test actually measure, and which of my workloads does that map to?"

Working from the Artificial Analysis September 15 snapshot at maximum reasoning effort (with Fable's default fallback enabled — meaning another model may service some requests, so treat these as directional, not matched):

Astra reads as an operator. It leads on AutomationBench-AA, which measures completing multi-step business workflows across SaaS apps through APIs while respecting business rules. It also posts stronger Terminal-Bench 4.0 results — command-line engineering and ops work. OpenAI reports it ahead of GPT-5.6 Sol and Claude Opus 5 on OSWorld 2.0's offline partial-credit comparison, though that published setup doesn't establish a matched head-to-head against Fable 5.1.

Map that to: CRM record updates, support escalation processing, employee onboarding flows, expense-vs-budget checks, failed build triage, dependency resolution, CI/CD pipeline debugging, supplier invoice retrieval, portal form filling, frontend QA.

Fable reads as an analytical builder. It leads on SciCode — scientific programming specifically, not software development broadly — and on AA-LCR v1.1, long-document reasoning.

Map that to: engineering simulations, numerical work, research-oriented code where domain reasoning and code generation have to interlock, supplier contract comparison, RFP analysis, financial report review, conflicting-policy detection, regulatory requirement mapping.

Then the economics split in an interesting way: Astra shows lower cost per task in the cited evaluation, while Fable is cheaper on cached input. Those favor different shapes of work.

Cached-input pricing is a workload-shape question, not a discount. If the same large context — a policy corpus, a codebase, a contract set, a documentation tree — gets reused across many calls, cache-friendly pricing dominates the bill. If every call carries a fresh context, it's irrelevant. Check your reuse ratio before you let a pricing page pick your model.

The pattern that beats picking one: plan with the builder, execute with the operator

Once you stop treating model choice as a procurement decision, an obvious architecture falls out. Split the workflow at the plan/act boundary.

                    ┌──────────────────────────────┐
   task + context → │  PLANNER  (analytical model) │
                    │  • decompose the goal        │
                    │  • read the long context     │
                    │  • emit typed action plan    │
                    └───────────────┬──────────────┘
                                    │  structured plan
                                    │  (JSON, validated)
                    ┌───────────────▼──────────────┐
                    │  VALIDATOR (deterministic)   │
                    │  • schema check              │
                    │  • policy / scope check      │
                    │  • blast-radius estimate     │
                    └───────────────┬──────────────┘
                          reject ◄──┤──► approve
                                    │
                    ┌───────────────▼──────────────┐
                    │  EXECUTOR (operator model)   │
                    │  • drive tools step by step  │
                    │  • no re-planning authority  │
                    │  • emit per-step receipts    │
                    └───────────────┬──────────────┘
                                    │
                    ┌───────────────▼──────────────┐
                    │  VERIFIER                    │
                    │  • did the artifact change?  │
                    │  • does it pass acceptance?  │
                    └──────────────────────────────┘
                         pass → done | fail → replan

The critical design constraint: the executor does not get replanning authority. If it hits an unexpected state, it fails the step and returns control to the planner. That one rule is what makes the split worth the added latency — it keeps an operator-tuned model from improvising its way into your production filesystem.

Here's the handoff contract, which matters more than either model choice:

{
  "plan_id": "recon-2026-09-19-001",
  "goal": "Reconcile Q3 regional sales against the GL extract",
  "acceptance": {
    "type": "assertion",
    "check": "variance_report.total_unexplained < 0.005 * gl_total"
  },
  "steps": [
    {
      "id": 1,
      "tool": "sheets.read",
      "args": { "file_id": "1AbC...", "range": "Q3!A1:M400" },
      "writes": false
    },
    {
      "id": 2,
      "tool": "sheets.write",
      "args": { "file_id": "1AbC...", "range": "Variance!A1" },
      "writes": true,
      "blast_radius": "single_sheet",
      "requires_approval": true
    }
  ],
  "on_unexpected_state": "halt_and_return"
}

Every step declares whether it writes. Every write declares a blast radius. The validator can now enforce policy before a token of execution is spent — and your autonomy limits become config, not vibes.

Autonomy is a dial, not a switch

The four decisions give you capability. This gives you containment:

Tier	Model class	Write access	Human gate
T0	Small / cheap	None (read-only)	None
T1	Operator	Scratch + sandbox	Post-hoc audit
T2	Operator	Scoped production writes	Pre-approve plan
T3	Builder + Operator	Broad production writes	Approve plan + verify artifact

Most workloads that get routed to a frontier model belong at T0 or T1. Classification, extraction, routing, summarization of a single document, formatting — these don't need frontier reasoning, and paying for it is the quiet majority of overspend in most agent platforms I've seen described.

You do not need a frontier model at every step. You need the right amount of intelligence at each step.

What to actually do on Monday

List your top five agentic workloads. For each, write the acceptance criterion as an assertion, not a sentence.
Score each on the four decisions. Execution and Building especially — most teams have never separated them.
Compute cost per accepted outcome, including retries and review minutes. Divide by acceptance rate.
Find the workloads with high context reuse. Those are your cache-pricing candidates.
Pick one workflow and split it at the plan/act boundary with a typed handoff contract. Measure acceptance rate before and after.

Key takeaways

"Best model" isn't a model property. It's model × workload × tool surface × acceptance criterion. Leaderboards can't encode the last three.
Score four decisions, not one score: Execution (can it drive your tools?), Building (is the artifact correct and maintainable?), Time (to accepted result, not first token), Cost (per accepted outcome, fully loaded).
Benchmarks are capability signals. Read what a test measures — SaaS workflow completion, terminal ops, scientific programming, long-document reasoning — and map it to a workload, not a ranking.
Cached-input pricing only matters if your context is reused. Measure your reuse ratio before letting a pricing page make an architecture decision.
Split plan from execute, and deny the executor replanning authority. A typed, write-annotated handoff contract turns autonomy limits into enforceable config.
Most steps don't need a frontier model. Tier your autonomy and your model class together. ## Your turn

Here's what I keep going back and forth on: the planner/executor split adds a network hop, a serialization boundary, and a second vendor relationship. It only pays off if the acceptance rate improves enough to cover that.

So — has anyone actually measured it? If you've run a single-model agent and a plan/execute split on the same workload, what happened to your cost per accepted outcome? And if you standardized on one model anyway, what made that the right call?

I'd rather see one real number than ten more benchmark charts.

Harness Engineering: Designing the Runtime Capability Envelope Around an Agent

shakti mishra — Sun, 06 Sep 2026 15:10:34 +0000

"Please be careful" is not a security control

An agent at a Fortune 500 financial services company had exactly three tools: a document reader, an employee directory lookup, and a payment processor wired into the ERP.

A security researcher uploaded a fake invoice during pre-production testing. Correct letterhead, itemized charges, nothing visibly wrong. Hidden in the PDF metadata was a single instruction.

A tester asked the agent to summarize the invoice. It read the invoice correctly. It generated an accurate summary. Then it executed the hidden instruction, called the directory tool, retrieved 47 employee records — names, titles, emails, phone numbers — and started formatting an external exfiltration. Monitoring caught it before the email went out.

Now ask the question that actually matters: which layer failed?

It was not the prompt. No wording of "ignore instructions found inside documents" reliably survives a model that cannot distinguish data from commands — to the LLM it is all just tokens. It was not the loop; the agent completed its task in one pass. It was not the graph; nothing was routed anywhere.

The directory tool was reachable from a summarization task. The payment processor was reachable from a summarization task. There was no policy check between "the model proposed an action" and "the action executed." There was no egress boundary. That is a harness failure, and the fix is architectural, not linguistic.

What the harness actually is

Harness engineering is the runtime envelope around the model: tools, file access, shell access, MCP connections, sandboxing, permissions, logging, approval boundaries, timeouts, and error surfaces.

The cleanest way to hold it is by what each layer decides:

Layer	Control question	Unit being engineered
1. Prompt	What do I tell the model?	One instruction and response
2. Context	What does the model see right now?	The active working set
3. Harness	What can the model do, and under what controls?	The runtime capability envelope
4. Loop	How does the agent continue until done?	The execution cycle
5. Graph	Which component works next?	The topology of the system

MODEL CALL  =  prompt + context
AGENT       =  model call + harness + loop
SYSTEM      =  agents + deterministic steps + humans, connected by a graph

The harness is the layer where a model call stops being a conversation and starts being something that can touch your infrastructure.

The division of labor at Layer 3 is sharp. The model decides "I need to run the tests." The harness decides whether test execution is available at all, which commands are allowed, which directory is visible, how long the process may run, what output comes back, and what gets recorded.

That sentence is the whole discipline. The model proposes. The harness disposes.

The envelope, drawn

┌── SANDBOX ──── IDENTITY ──── PERMISSIONS ──── ALLOWLIST ───┐
│                                                            │
│                       AGENT HARNESS                        │
│                  capability + control                      │
│                                                            │
│      TERMINAL      FILES        GIT       MCP TOOLS        │
│          ▲           ▲           ▲            ▲            │
│          └───────────┴─────┬─────┴────────────┘            │
│                         ┌──┴──┐                            │
│                         │MODEL│                            │
│                         └──┬──┘                            │
│                            │ proposes action               │
│                            ▼                               │
│                     ┌──────────────┐                       │
│                     │ POLICY CHECK │                       │
│                     └──┬────────┬──┘                       │
│                  ALLOW │        │ DENY                     │
│                        ▼        ▼                          │
│                  tool action   deny + explain              │
│                        │        │                          │
│                        └───┬────┘                          │
│                            ▼                               │
│                       observation ─────► back to MODEL     │
│                                                            │
└── TIMEOUTS ── APPROVAL GATE ── SECRETS ── LOGS + TRACES ───┘

Three details in that diagram carry most of the weight.

The control chips sit on the boundary, not inside it. Sandbox, identity, permissions, allow list, timeouts, approval gate, secrets isolation, and logs are properties of the envelope. They are not tools the agent calls and they are not instructions the agent can be talked out of. If a control is something the model can reason its way past, it is a prompt, not a harness.

Every proposed action passes through a policy check before it becomes a tool action. Not after. The interesting architectural choice is what sits on that edge — an allow list, a scope check against the current task, a blast-radius classifier, an approval gate for anything above a threshold.

Deny returns an observation, it does not throw. This is the part teams get wrong most often. A denial that surfaces as an opaque exception gives the agent nothing to work with, so it retries the same forbidden action until the budget dies. A denial that returns "blocked: directory lookup is not in scope for a summarization task" lets the agent adapt or escalate. Deny and explain, not deny and crash.

Capability without control expands blast radius

That is the whole thesis in six words, and it reframes tool design as a security decision rather than a features decision.

Return to the invoice incident and map it against the envelope:

SHIPPED CONFIGURATION — blast radius = every registered tool

  "summarize this invoice"
          │
          ▼
        Agent ──► doc reader        ──► summary                  ✅
          ├─────► directory lookup  ──► 47 employee records      ❌
          │                                     │
          │                                     ▼
          │                            external send attempt     ❌
          └─────► payment processor  ──► reachable, untriggered  ❌


HARNESSED CONFIGURATION — blast radius = one tool

  "summarize this invoice"
          │
          ▼
     scope check ──► doc reader ──► summary ──► egress boundary  ✅
          │
          ├─────► directory lookup     DENIED — out of task scope
          └─────► payment processor    APPROVAL GATE — irreversible

The failing configuration is not unusual. It is what you get when tools are registered once, at the agent level, and every task inherits the full manifest. The task was summarization; the capability envelope was "everything this agent has ever needed."

The harnessed version changes four things, none of which touch the prompt:

Per-task scoping. The tool manifest is derived from the task, not from the agent. A summarization task does not get a directory lookup.
Blast-radius tiering. The payment processor is not merely allow list or denied — it sits behind an approval gate because a wrong call is irreversible.
An egress boundary. Data leaving the envelope is its own control point, independent of which tool produced it.
Deny paths that return observations. The agent learns it was blocked and why. Note what this buys you against indirect prompt injection specifically. You cannot reliably teach a model to distinguish instructions embedded in retrieved content from instructions issued by the user — that is the defining weakness of the architecture. What you can do is guarantee that a successful injection reaches a capability envelope narrow enough that the injection has nothing useful to do.

To be precise about the claim: input filtering, output classifiers, and instruction-hierarchy training all reduce the rate of successful injection, and you should run them. What they cannot do is give you a bound. The harness is the only layer that produces one — it caps what a successful injection can reach, independent of how the injection got through. Detection lowers the probability. Only the harness lowers the damage.

MCP authorizes the connection. It does not authorize the task.

This is the distinction teams collapse, and it is worth being precise about — including in the direction that flatters MCP.

MCP is not authorization-silent. The spec is built on OAuth 2.1: the MCP server is a resource server, a separate authorization server issues tokens, PKCE with S256 is mandatory, and RFC 9728 protected resource metadata plus RFC 8707 resource indicators bind a token to a specific server. Scopes are first-class (mcp:read, mcp:write), the 2025-11-25 revision added step-up authorization so a server can demand more scope mid-session, and token passthrough is explicitly forbidden — a server must reject tokens not issued for it.

So "MCP doesn't do auth" is a claim that has been out of date for a while. Say it in a design review and someone will correct you.

What the spec deliberately leaves open is narrower and more interesting:

MCP SPECIFIES                    │  STILL YOURS TO BUILD
─────────────────────────────────┼──────────────────────────────────
OAuth 2.1 client/server roles    │  which tools THIS task may use
PKCE, resource indicators        │  per-task manifest derivation
protected resource metadata      │  blast-radius tiering
coarse scopes (mcp:read/write)   │  approval gates on irreversible ops
no token passthrough             │  MCP server → downstream API auth
                                 │  agent-acting-on-own-behalf auth
                                 │
                          trust boundary

Two of those gaps are named limitations in the spec's own discourse. How an MCP server authenticates onward to the database or API behind it is described as outside the specification's scope. Server-to-server and agent-acting-on-its-own-behalf authorization is intentionally left silent in the core spec. And scope granularity is a known pain point — Rich Authorization Requests have been discussed for exactly this reason and are not in the protocol.

Which lands on the point that matters for Layer 3. An MCP server exposing a delete_records tool, behind a correctly implemented OAuth 2.1 flow, with a valid token carrying mcp:write, has answered "is this client allowed to talk to me?" It has not answered "should a summarization task invoke destructive deletion right now?" A scope is not a task scope. That second question is the harness's, and nothing in the protocol will answer it for you.

Teams that treat "we adopted MCP" as "we solved tool governance" have correctly secured the connection and left the blast radius untouched.

Where the harness ends and the loop begins

The boundary between Layer 3 and Layer 4 is where most architecture arguments stall, so here is the split as a pair of questions:

Harness: Can the agent execute the test — in which sandbox, with what permission, under what timeout?
Loop: Should a failed test trigger another attempt, what must change before retrying, how many attempts are allowed, and what counts as done? You can have an excellent harness and a terrible loop. An agent can be perfectly sandboxed, fully logged, permission-scoped to exactly the right surface — and still retry the identical failing fix until it exhausts its token budget. Nothing in the harness stops that, because nothing in the harness is supposed to.

The inverse is worse. A disciplined loop with a wide-open harness means every retry is a fresh opportunity to do damage with precision and persistence.

This is also why "the layers are not a build order" matters. Sketch the graph, define the control boundaries, then tune prompts. The harness is a concentric ring around the model, not a stage in a pipeline — and in practice the weakest control sets the reliability ceiling for the entire system.

Why I keep the definition narrow

You will encounter a much broader use of "harness" in the wild — one that means everything in an agent that is not the model. Under that reading, the harness swallows the instruction file at your repo root, the retrieval pipeline, the verification loop, the memory store, the orchestration, all of it.

That definition is not wrong. It is just useless for the thing Layer 3 exists to do.

The honest caveat in the five-layer framework is that the boundaries leak. Memory can plausibly be filed under context, harness, or runtime state. Verification can sit inside a tool boundary, a retry loop, or a separate graph node. These are five concerns, not five cleanly separated software components, and anyone who tells you the seams are crisp has not shipped one.

So the question is not which definition is philosophically correct. It is which one produces better decisions in a design review. Two tests:

Can you audit it? "Is our harness good?" is unanswerable if the harness is everything. "Which tools are reachable from this task, who authorized them, and what happens on deny" is a question with a finite answer that a security reviewer can actually check. A category that spans the entire system is a category you cannot pass or fail.

Does it survive a postmortem? When an agent burns its budget retrying an identical broken fix, the narrow definition tells you immediately that the harness held and the loop failed — go fix the retry policy. The broad definition tells you "the harness failed," which is true, uninformative, and points at no owner.

Diagnostic clarity is the whole value of the layered model. It gives a team a better answer than "the AI was weird." Collapse verification, retrieval, and permissions into one word and you have traded that clarity for a tidier vocabulary.

Use the wide definition when you are explaining to an executive why the model is not the product. Use the narrow one when you are deciding what your agent is allowed to touch on Monday.

Measuring harness quality

A layer you cannot measure is a layer you cannot defend in a design review. Three metrics travel well:

Metric	What it tells you	Failure signal
Tool success rate	Whether the capability surface actually works	High denial-adjacent failures mean your allowlist is wrong, not your model
Unsafe actions denied	Whether the policy check is doing anything	A permanent zero means either a perfect agent or a dead control
Permission boundary integrity	Whether scope holds under adversarial input	Any crossing is a P0, not a tuning issue

That middle row deserves emphasis. If your deny counter never moves, you do not have evidence of safety — you have an untested control. Treat a flat-zero denial rate the way you would treat a monitoring system that has never fired an alert: assume it is broken until you prove otherwise with a deliberate probe.

Which points at the natural companion practice. Automated red teaming exists to move that counter on purpose — adversarial probing that scores attack-response pairs into an attack success rate, run continuously rather than once before launch. Harness metrics tell you the envelope is holding. Red teaming tells you whether anyone has actually pushed on it.

Both belong on the same dashboard, alongside latency and cost. As Microsoft's framing puts it: a chatbot saying something offensive is a PR issue; an agent executing a prohibited action is a security breach.

What to build first

If you are retrofitting a harness onto an agent already in production, the ordering that produces the fastest risk reduction:

flowchart LR
    A[Inventory the real<br/>capability envelope] --> B[Scope manifests<br/>per task]
    B --> C[Policy check on<br/>proposed action]
    C --> D[Deny + explain<br/>observations]
    D --> E[Approval gates on<br/>irreversible actions]
    E --> F[Instrument<br/>the boundary]

Inventory the actual capability envelope. Not the documented one. Enumerate every tool reachable from every task type, including transitively through MCP servers. Most teams find at least one surprise.
Derive tool manifests per task, not per agent. This single change eliminates the largest class of blast-radius incidents, and it costs you a routing decision rather than a rewrite.
Insert the policy check on the proposed-action edge. Start with an allowlist. Add scope checking second. Add blast-radius tiering third.
Make denials informative observations. Cheapest change on this list, and it converts a retry storm into an escalation.
Put irreversible actions behind approval gates. Deletion, payment, external send, production writes. If undoing it requires a human, invoking it should too.
Instrument the boundary. Logs and traces on every proposed action, allowed or denied. You cannot investigate an incident inside an envelope you did not record. Prompts still matter. In an agent system the prompt is the steering wheel — but the harness is the brakes, the seatbelt, and the guardrail on the shoulder. Nobody has ever argued that a good driver makes the guardrail optional.

Key takeaways

The model proposes, the harness disposes. The model decides it needs to run a command; the harness decides whether that capability exists, in which sandbox, under what permission and timeout.
Controls belong on the boundary, not in the prompt. If the agent can reason its way past a constraint, that constraint is an instruction, not a control.
Capability without control expands blast radius. Tool registration is a security decision. Scope tool manifests per task, not per agent.
Deny and explain, not deny and crash. A denial that returns an interpretable observation lets the agent adapt; an opaque exception produces a retry storm.
MCP authorizes the connection, not the task. OAuth 2.1, PKCE, and scopes answer "may this client talk to this server." Whether this task should invoke this tool right now is the harness's question, and the protocol leaves it open.
A denial counter stuck at zero is an untested control, not a safe system. Pair harness metrics with deliberate adversarial probing.
Keep the definition narrow enough to audit. "Harness" is often used to mean everything that isn't the model. That framing is fine for explaining why the model isn't the product, and useless for deciding what your agent may touch on Monday.

The question worth arguing about

Here is the uncomfortable audit. Pick your most autonomous production agent and enumerate every tool reachable from its lowest-stakes task — a summarization, a lookup, a status check.

If a hostile instruction landed inside the content that task processes, what is the worst thing it could reach?

If you cannot answer that in under a minute, the envelope is not designed; it accumulated.

So: is your agent's tool manifest scoped per task, or per agent — and if it's per agent, what's the actual reason? I suspect "the framework defaults that way" is the honest answer far more often than anyone wants to admit.

Standard RAG vs. Agentic RAG: Moving Retrieval From Pipeline Stage to Runtime Decision

shakti mishra — Sun, 30 Aug 2026 15:23:38 +0000

The assumption every RAG demo makes

Standard RAG assumes the user's question maps onto one vector search. One query in, one embedding, one top-k lookup, one answer.

That assumption holds up in demos, because demos ask demo questions. "What's our parental leave policy?" is one document. Retrieve it, stuff it into the prompt, done.

Then you ship, and a real user types: "Did the carrier rate change we approved in Q2 actually reduce our cost per shipment in the Northeast, and does that hold if I exclude the Boston depot?"

That question needs a policy document, a rate table, a transactional aggregate, and a filtered re-computation. Your retriever will embed the whole sentence, find the three chunks nearest to it in vector space, and hand the model text that is topically adjacent and factually useless. The model, being a good sport, will answer anyway.

The problem isn't the embedding model or the chunk size. You hardcoded how many times to retrieve, and where to retrieve from, at design time, for a question you hadn't read yet.

Agentic RAG moves that decision to runtime. Planners, memory, MCP servers, sub-agents: all of it is implementation detail hanging off that one change.

Architecture 1: standard RAG is a straight line

STANDARD RAG — fixed pipeline, one pass
┌──────┐   1. prompt+query    ┌─────────────┐
│ User │ ───────────────────► │ Chat UI     │
└──────┘                      └──────┬──────┘
   ▲                                 │ 2. query
   │ 6. response                     ▼
   │                          ┌─────────────┐
   │                          │  Retriever  │
   │                          └──────┬──────┘
   │                                 │ 3. fetch (top-k, one shot)
   │                                 ▼
   │                     ┌───────────────────────────┐
   │                     │  Knowledge Sources        │
   │                     │  docs · PDFs · code · DB  │
   │                     │  APIs · web index         │
   │                     └───────────┬───────────────┘
   │                                 │ 4. chunks
   │                          ┌──────▼──────┐
   └──────────────────────────│     LLM     │
                              └─────────────┘
                       5. prompt + query + enhanced context

The defining property is that the model is never consulted about retrieval. It receives context and produces text, and retrieval already finished by the time it runs.

That's a design choice with real advantages. One embedding call plus one vector query is cheap and predictable, latency sits in a tight distribution, and you can cache aggressively. Failures are legible: a bad answer means bad chunks, and you can go read the chunks. Your eval harness is a fixed input and a fixed output, so it works.

Standard RAG is the right architecture when your corpus is homogeneous and your users mostly want lookup rather than synthesis. Don't let anyone talk you out of it for those workloads.

Architecture 2: agentic RAG makes retrieval a decision

AGENTIC RAG — planner decides retrieval at runtime
┌──────┐  1. prompt+query   ┌─────────────┐
│ User │ ─────────────────► │ Chat UI     │
└──────┘                    └──────┬──────┘
   ▲                               │ 2. query
   │ 6. response                   ▼
   │                    ┌──────────────────────┐      ┌──────────────┐
   │                    │  Aggregator /        │◄────►│ Planning     │
   │                    │  Orchestrator Agent  │  3.  │ ReAct · CoT  │
   │                    └───────┬──────────────┘      └──────────────┘
   │                            │         ▲
   │                            │         └──────┐  ┌──────────────┐
   │                            │ 4. fan-out     └─►│ Memory       │
   │                            ▼                   │ short · long │
   │            ┌───────────────┼───────────────┐   └──────────────┘
   │       ┌────▼────┐     ┌────▼────┐    ┌─────▼───┐
   │       │ Agent 1 │     │ Agent 2 │    │ Agent 3 │
   │       └────┬────┘     └────┬────┘    └────┬────┘
   │            │  MCP servers / tool layer    │
   │       ┌────▼────┐     ┌────▼────┐    ┌────▼────┐
   │       │ SQL DB  │     │ Search  │    │ Data    │
   │       │         │     │ index   │    │ Explorer│
   │       └─────────┘     └─────────┘    └─────────┘
   │                            │ 5. prompt + query + enhanced context
   │                     ┌──────▼──────┐
   └─────────────────────│     LLM     │
                         └─────────────┘
                  ◄── loop back to (3) if context insufficient

Four things are different.

1. A planner sits between the query and the retriever. The aggregator agent decomposes the question before anything gets embedded. ReAct and chain-of-thought are the mechanism that turns one user sentence into a retrieval plan. The Northeast shipping question decomposes into four steps: fetch the Q2 rate change record, aggregate shipment costs by region and date, re-aggregate with the Boston depot excluded, compare.

2. Retrieval fans out across heterogeneous backends. Standard RAG normalizes everything into one vector index at ingestion time. Agentic RAG queries systems in their native language at query time: SQL against the warehouse, semantic search against the document index, a time-series query against event data. You don't embed a rate table, you query it.

3. MCP is the integration seam. When your retrievers are MCP servers rather than bespoke functions, the tool surface is declarative and swappable. You add a data source by registering a server instead of shipping a new agent build. It's a plugin architecture applied to retrieval.

4. Memory carries retrieval state across turns. Short-term memory holds what this session already fetched, so hop three doesn't re-retrieve what hop one found. Long-term memory holds durable facts about the user and their prior queries. Without it, a multi-hop system re-derives the same context every turn and you pay for it twice.

The structural difference is the loop back to the planner. Standard RAG has one retrieval pass. Agentic RAG has a loop with a termination condition, and that loop is where both the value and the risk live.

What runtime retrieval costs you

Dimension	Standard RAG	Agentic RAG
Retrieval decision	Design time	Runtime
LLM calls per query	One	One per hop, plus planning
Latency	One round trip, tight variance	Accumulates per hop, long tail
Cost per query	Flat	Variable, roughly linear in hops
Determinism	High	Low, the same query can take a different plan
Debugging	Inspect the chunks	Reconstruct the trace
Eval strategy	Answer quality on fixed input	Answer quality and trajectory quality
Multi-hop questions	Fails silently	Handles
Structured data	Poor, embeddings flatten schema	Native, via SQL tools
Failure mode	Confident answer from wrong chunks	Loops, fans out, burns budget

The determinism row is the one that costs most in practice. Standard RAG failures are boring and reproducible. Agentic RAG failures are interesting, which is a bad property for a production system: the same question asked twice can take different paths and produce different answers, and neither is wrong so much as differently sourced. Reproducibility was doing quiet work in your incident response process, and you give it up when you move the decision to runtime.

The sufficiency gate

Before adding another hop, check whether the context you already have is enough to answer, and put a hard ceiling on the case where it isn't. That check does more for a production agentic RAG loop than a better planner does.

MAX_HOPS = 4
TOKEN_BUDGET = 20_000

def agentic_retrieve(query: str, tools: dict) -> RetrievalResult:
    plan = planner.decompose(query)          # ReAct-style sub-questions
    ctx, trace, spent = [], [], 0

    for hop in range(MAX_HOPS):
        gap = planner.next_gap(query, ctx)   # what's still missing?
        if gap is None:
            break                            # sufficiency gate: stop early

        tool = router.select(gap, tools)     # sql | search | timeseries
        chunks = tool.invoke(gap)

        spent += count_tokens(chunks)
        if spent > TOKEN_BUDGET:
            trace.append(("halt", "budget_exceeded", hop))
            break

        ctx.extend(chunks)
        trace.append((tool.name, gap, len(chunks)))

    return RetrievalResult(
        context=dedupe(ctx),
        trace=trace,                         # emit this. always.
        exhausted=len(trace) >= MAX_HOPS,    # flag for review queue
    )

next_gap returning None is the sufficiency gate. It lets a single-hop question cost one hop instead of four. Without it, your agentic system pays multi-hop prices on lookup questions, which are most of your traffic, and that shows up on the bill.

MAX_HOPS and TOKEN_BUDGET are not optional. An unbounded retrieval loop against a live SQL tool is a denial-of-service vector pointed at your own warehouse, and a badly worded user question is enough to trigger it.

trace is what makes the system debuggable. Emit it on every request, store it next to the answer, and make it queryable. When agentic RAG fails, the answer tells you nothing about why.

Where teams get this decision wrong

The most common mistake is adopting agentic RAG for a single-hop corpus. If your knowledge base is a few thousand support articles and users ask article-shaped questions, a planner adds latency and cost to reach the same document standard RAG would have found in one pass. Route on question shape rather than on architectural fashion.

Skipping the router is close behind. A planner handed eleven tools and no routing heuristic will explore all of them. Constrain the tool set per query class before the planner ever sees it.

Then there's treating the two architectures as either/or. A classifier in front of both paths usually works better: the cheap single-hop route for lookup questions, the agentic route for questions that decompose. If most of your traffic is lookup, most of your traffic shouldn't be paying for a planner.

Evaluation is the expensive mistake. In standard RAG, answer quality is a decent proxy for system health. In agentic RAG it isn't, because a correct answer reached through six wasteful hops is a cost and latency problem waiting for your traffic to grow. Track hops per query, tool selection precision, redundant retrieval rate, and how often the sufficiency gate fires.

Finally, teams forget which argument actually closes the case, and it isn't multi-hop reasoning. Embeddings are a lossy representation of a relational schema. If your answers require joins, aggregations, or filters, no chunking strategy saves you, and you need a tool that speaks SQL.

Key takeaways

The difference between the two architectures is control flow, and everything else follows from it. Standard RAG decides how to retrieve at design time; agentic RAG defers that to runtime.
Standard RAG isn't a legacy pattern. On a homogeneous corpus with single-hop questions it's cheaper, faster, more reproducible, and easier to evaluate.
A sufficiency gate is what keeps the cost model viable, because it stops the loop early on the ordinary questions that make up most of your traffic.
Always bound the loop. MAX_HOPS and a token budget give you a worst case you can name.
Structured data is the clearest signal to switch. Embeddings flatten schema, so joins and aggregations need a tool that queries the database directly.
Evaluate the trajectory as well as the answer. A right answer reached the wrong way is a latent production problem.

Separating the Idea, State, and Artifact Planes in Agent Pipelines

shakti mishra — Sat, 22 Aug 2026 23:17:55 +0000

The agent worked. The architecture didn't.

This failure mode won't show up in your eval suite.

An AI architect builds a personal automation pipeline: an agent scouts trending content, pipes ideas into a chat thread, he reacts, things get logged. The agent did its job. It found the trends, wrote them down, and responded when he asked it to.

He killed it anyway, and not over hallucination or token cost. He killed it because everything lived in "a flat scroll of messages. No structure, no views, no way to see what is in scripting versus what is scheduled." He was spending his entire one-hour creation window scrolling backwards through a Telegram thread, looking for something he'd written the day before.

The model wasn't the problem. The substrate was, and almost everyone shipping agents today has the same problem, because the default agent UI is a chat window and the default system of record is the transcript.

If your agent's output lands in a conversation, you've built an append-only log with no schema and no query interface. Retrieval is O(n), and a human eye does the scanning. That works fine in a demo and falls apart somewhere around week three.

Why the transcript fails as a system of record

Break a chat log down to its data properties and the problem is obvious:

CHAT TRANSCRIPT AS DATABASE
---------------------------
Schema            none (freeform text)
Primary key       timestamp (not semantic)
Indexes           none
Query interface   Ctrl+F, human eyes
Mutation          impossible (append-only, no UPDATE)
Aggregation       none
Views/filters     none
State transitions untracked

The row that matters is Mutation: impossible. Real work has state that changes: an idea becomes a draft, the draft gets scheduled, the scheduled item ships. In a chat log none of those transitions exist as data. They exist as later messages contradicting earlier ones, and the reader has to reconstruct the current truth by replaying the whole thread.

That's event sourcing without a projection. You get the event stream and no materialized view.

It also compounds. Every session starts with context reconstruction: what did I decide? where did I leave off? which of these forty messages is still live? The human pays that cost every time, and it grows with the length of the thread while the thread's value stays flat.

You'll know you have the problem when you catch yourself using the agent to search the agent's own output.

The fix: three planes, not one thread

The working version of the pipeline separates concerns into three data planes, each with a different write pattern and a different owner.

        ┌──────────────────────────────────────────┐
        │  PLANE 1: IDEA / INTAKE                  │
        │  Google Sheet · append-only              │
        │  Writer: agent (scheduled, unattended)   │
        │  Volume: high · Precision: low           │
        │  Never edited in place                   │
        └───────────────────┬──────────────────────┘
                            │  human selects (the gate)
                            ▼
        ┌──────────────────────────────────────────┐
        │  PLANE 2: STATE / PIPELINE               │
        │  Notion DB · typed records               │
        │  Writer: agent on human instruction      │
        │  Volume: low · Precision: high           │
        │  status ∈ {Idea, Scripting, Filming,     │
        │            Scheduled, Published}         │
        └───────────────────┬──────────────────────┘
                            │  references by ID
                            ▼
        ┌──────────────────────────────────────────┐
        │  PLANE 3: ARTIFACT                       │
        │  Google Docs · Drive · rendered media    │
        │  Writer: human (scripts) + agent (renders)│
        │  Addressed by link from Plane 2          │
        └──────────────────────────────────────────┘

Each plane has a job the other two are bad at.

Plane 1 optimizes for recall rather than precision. The scheduled scan writes 20+ ideas a day across TikTok, Reels, X and YouTube, each row carrying the trend, a hook, an angle, a caption, hashtags, format notes, and a suggested post time. Most of those rows are garbage, which is fine. It's an intake buffer, and the rule is never curate in place.

That rule earns its keep. Agent writes are non-deterministic, so if your agent mutates rows in the same table it writes to, you lose the ability to tell "the agent changed its mind" from "I changed my mind" from "the agent silently dropped something." Append-only intake gives you an audit trail for free, and it makes human selection the only path from noise to signal.

Plane 2 optimizes for queryable state: one typed record per unit of work, with a status enum. A single glance answers what's in scripting, what's waiting to film, and what ships next, which is the question the chat thread could never answer.

Plane 3 holds the payload. Documents and media are referenced by ID from Plane 2, never inlined into it. That's ordinary normalization, and agent systems break it all the time by pasting entire drafts into chat messages.

The state machine itself is simple enough to copy:

  Idea ──► Scripting ──► Filming ──► Scheduled ──► Published
   │           │             │
   └───────────┴─────────────┴──► Dropped

Five states and one escape hatch. That's the whole orchestration model, and both the human and the agent can read it, which matters: an agent that sees status = "Scripting" can act on it directly. An agent reading a chat thread has to infer status from prose, and inference is where reliability starts slipping.

Splitting the agent into workflow, runner, and schedule

The second architectural idea in the piece is a clean three-way split of what most people mash into a single "agent":

Component	Answers	Analogue
Workflow	How the job is done	DAG definition / playbook
Runner	Who owns and executes it	Worker identity + permissions + memory
Schedule	When it fires	Cron / event trigger

If you've ever built data infrastructure, this is Airflow's DAG-vs-worker-vs-scheduler split arriving in the agent world, and the fact that it keeps getting rediscovered independently is a decent signal that it's correct.

The practical consequence is that each axis varies independently. Same workflow, different runner (dev vs prod credentials). Same runner, different schedule (daily scan vs on-demand). Same schedule, swapped workflow (v1 to v2 of your edit pipeline) without touching the trigger or the identity.

The monolithic alternative is a chat agent where the how, the who, and the when are tangled together in a prompt you retype every session. You can't version that, and you certainly can't hand it to anyone else.

ENTANGLED (chat agent)
  "hey can you scan trends and put them in my sheet like
   you did last time, you know the format" ──► ???

DECOMPOSED
  Workflow:  trend_scan.v3   (deterministic playbook)
  Runner:    content_bot     (Drive+Notion scopes, memory: voice/topics)
  Schedule:  0 6 * * *       (daily 06:00)

The decomposed version is a config artifact. You can diff it, review it, and roll it back.

Skills are portable; runtimes are not

This is the detail with the longest shelf life. The video editing logic (cut to vertical 1080×1920, word-synced captions, motion graphics, brand logos, phone-legible thumbnail) was a Claude Skill wrapping Remotion. When the pipeline moved to a different orchestrator, that logic wasn't rebuilt from scratch. It was ported into a saved Workflow and invoked by a Runner.

The value was never in the runtime.

It was in the encoded decision rules: caption style, safe zones, font sizes, brand colors, render steps. Those took work to get right and would be expensive to rediscover. The execution environment around them is commodity infrastructure that churns every eighteen months or so.

       ┌─────────────────────────────────────┐
       │  DURABLE  (you author this once)    │
       │  • decision rules & checklists       │
       │  • constraints, safe zones, brand    │
       │  • step ordering & failure handling  │
       └──────────────┬──────────────────────┘
                      │ portable
       ┌──────────────▼──────────────────────┐
       │  COMMODITY (swap every ~18 months)  │
       │  • agent runtime / orchestrator      │
       │  • model provider · tool bindings    │
       └─────────────────────────────────────┘

So write skills as instruction artifacts rather than runtime code. A skill expressed as a markdown playbook with explicit steps, constraints, and decision rules is a text file you can carry anywhere. Bind it hard to one vendor's SDK and you've scheduled a rewrite.

If you already have Claude Skills, you already have portable assets. They're the durable layer, not vendor lock-in, and every orchestrator (including the one you're using today) is replaceable.

Where to put the human: automate the overhead, not the critical path

The pipeline has five stages. Two of them are deliberately not automated:

1. Trend scan   AGENT   (scheduled, unattended)
2. Curation     HUMAN   → agent files the selection
3. Scripting    HUMAN   ← agent assists on request
4. Filming      HUMAN   (fully manual, by design)
5. Editing      AGENT   (workflow-invoked)

Automation sits at the ends, intake and post-production, while the human occupies the middle where judgment lives. The stated reason for keeping scripting human is sharp: "the point of the system is not to remove me from the work that is mine."

The engineering framing is Amdahl's law. Total time is critical path plus serial overhead. In a creative or knowledge pipeline the critical path is judgment, and it's irreducible because you are the value being produced. The serial overhead is everything around it: noticing, logging, filing, linking, formatting, rendering.

Speed up the overhead and the whole system gets faster. Try to speed up the critical path by automating judgment and you don't get faster output, you get more output that nobody wanted.

There's a second constraint the piece names explicitly, and it deserves a term: the friction budget. As he puts it: "anything that is not seamless does not get done. If an idea requires me to open six tabs, copy something from one place to another, and remember where I put the draft, it dies."

Treat friction as a hard budget. For a part-time operator with one hour, the budget is near zero: six tabs exceeds it, and the workflow gets abandoned no matter how good the agent is. It's the same reason an accurate internal tool that takes six clicks to reach loses to a rougher one sitting behind a single button. Adoption follows friction more than capability. Most agent projects that "failed" cleared the capability bar and blew the friction budget.

What to build on Monday

If you're standing up an agent pipeline, personal or production, the port is mechanical:

Pick a real system of record: anything with a schema, a status field, and views. A database, a project tracker, even a spreadsheet. Not the chat transcript.
Split intake from state. Agents write to a high-volume, append-only intake table. Nothing moves to the state table without passing a gate, either human approval or a deterministic filter. Never let an unattended agent mutate your source of truth in place.
Model the states explicitly, as a short enum both the human and the agent can read. If your agent has to infer status from prose, add the field.
Decompose how, who, and when into workflow, runner, and schedule, versioned separately. A retyped prompt is not an architecture.
Write skills as portable instruction artifacts. Playbooks in text, loosely bound to whatever runtime is current. Assume the runtime churns.

6. Audit your friction budget. Count the tabs and copy-paste steps between "I have an idea" and "I'm doing the work." If it's more than two, the workflow will be abandoned, and the model won't be the reason.

Key Takeaways

A chat transcript is an append-only log with no schema, no index, and no UPDATE, so it can't serve as a system of record for stateful work, however good the agent writing to it is.
Separate the idea plane, the state plane, and the artifact plane. Intake optimizes for recall and stays append-only, state optimizes for queryability and stays typed, and artifacts are referenced by ID.
Split the agent into workflow (how), runner (who), and schedule (when) so each axis versions and rolls back independently. Airflow settled on the same split a decade ago.
Skills are the durable layer and runtimes are commodity. Encode decision rules as portable instruction artifacts so an orchestrator swap is a port rather than a rewrite.
Automate the serial overhead, not the critical path. Judgment is where the human is the product. Noticing, filing, and rendering are where the hours leak.

- Adoption follows friction more than capability. A workflow that exceeds the user's friction budget dies regardless of model quality.

The question worth arguing about

Every major agent product is converging on the same interface: a chat box. But the moment an agent's work has state that outlives a session, the chat box stops being a UI and starts being a bad database.

So which is it? Is chat a transitional interface we'll look back on the way we look back on command-line-only databases, with agents eventually shipping real structured front ends? Or is the transcript fine, and the fix is just better memory and retrieval bolted onto the thread?

If you've killed an agent that technically worked, what made you stop using it? I'd bet more of those stories are about state and friction than about the model.

Graph Engineering Explained: The Missing Fifth Layer of AI Agent Architecture

shakti mishra — Sun, 16 Aug 2026 20:37:46 +0000

Every "my agent isn't working" postmortem starts the same way: someone rewrites the prompt. Adds a constraint. Adds an example. Ships it again. Three iterations later the agent still can't hold up in production, and the team is quietly out of ideas — because the prompt was never the layer that broke.

There are five control layers standing between a raw model call and a system you can actually trust with a business outcome: prompt, context, harness, loop, and graph. Most teams staff and instrument only the first one or two. The failures that show up in production — wrong tool called, same mistake retried forever, output routed to the wrong reviewer — live almost entirely in the layers nobody named.

Graph engineering is the newest and least understood of the five: it's the layer that decides which component runs next, when agents work in parallel versus in sequence, and where a human has to sign off before anything expensive or irreversible happens. This piece breaks down all five layers, works through a single production failure end to end, and shows where evals fit as the measurement system running through every one of them.

The mental model: five rings around the model

MODEL CALL  =  prompt + context
AGENT       =  model call + harness + loop
SYSTEM      =  agents + deterministic steps + humans, connected by a graph
EVALS       =  evidence that every layer actually works

Prompt and context sit closest to the model. Harness and loop turn a model call into something that can act and recover. Graph turns a collection of agents, functions, and human checkpoints into a coordinated system. None of these layers replace each other — they're concentric controls, not pipeline stages, and a production agent uses all five simultaneously. The weakest layer sets the ceiling on how reliable the whole thing is, no matter how good the other four are.

Layer	Controls	Fails as
Prompt	Role, goal, constraints, output contract	Ambiguous instructions
Context	What reaches the window: docs, history, tool results	Missing or noisy evidence
Harness	Tools, file/shell access, sandboxing, permissions	Overprivileged or unsafe actions
Loop	Retry policy, validators, stop conditions, escalation	Infinite retries on the same mistake
Graph	Routing, parallelism, recovery paths, human gates	Work reaching the wrong next step

A production failure, diagnosed layer by layer

Consider a coding agent built to fix low-risk defects in an internal payments service. The prompt is reasonable: inspect the issue, avoid unrelated changes, run the tests, return a PR summary. On a clean sample repo, it works. On the real repository, it falls apart in four distinct ways:

It misses an architecture decision buried in the docs — a context failure.
It runs a shell command with a broader scope than intended — a harness failure.
It retries the same failing test without changing its hypothesis — a loop failure.
It sends the pull request down the wrong review path — a graph failure. The natural instinct is to ask "how do we improve the prompt?" That's the wrong question. Only one of these four failures traces back to the instruction layer — and it isn't the one that caused the damage. Each failure needs a fix in the layer that actually owns it, not a paragraph bolted onto the system prompt.

Layer 1 — Prompt: steers one model call

Analyze the reported defect and propose the smallest safe fix.
Do not change unrelated behavior.
Return the root cause, files changed, test evidence, and residual risk.
Stop and ask for approval if the fix changes an external contract.

The unit being optimized here is a single model interaction. A stronger prompt reduces ambiguity, but it cannot supply a missing design document, restrict a dangerous tool, or decide who reviews the output. In an agent system, the prompt is the steering wheel — not the car.

Layer 2 — Context: what the model can actually see

Ask a model to summarize risk in an 80-page contract. Dumping the whole document into the window and retrieving the liability, indemnification, termination, and data-use clauses (plus the org's risk policy) produce two very different answers from the same prompt. The instruction didn't change — the evidence available to answer it did. For the coding agent, the missing architecture decision is a retrieval problem. Rewording the prompt might paper over one test case; fixing context assembly fixes the whole class of failure.

Layer 3 — Harness: the runtime envelope

The harness is everything around the model call: tools, file access, shell access, MCP connections, sandboxing, permissions, timeouts, logging, approval boundaries. The model can decide "I need to run the tests" — the harness decides whether that's even possible, which commands are allowlisted, which directory is visible, and what gets recorded. MCP standardizes how an agent connects to tools; it does not decide that an agent deserves production write access. Identity, least privilege, and approval policy still belong to the host and its surrounding control plane. This is usually the first layer a security team asks about, and it's exactly where the broad shell command should have been caught.

Layer 4 — Loop: the retry contract

Loop engineering owns the cycle — act, observe, evaluate, adjust, repeat — plus retry policy, validators, completion criteria, budgets, and escalation rules. It's a distinct concern from the harness:

Harness asks: Can the agent execute the test, in which sandbox, with what timeout?
Loop asks: Does a failed test trigger another attempt, what has to change before retrying, how many attempts are allowed, and what counts as done? You can have a perfectly sandboxed, fully logged harness and still watch an agent burn its entire budget retrying the identical failed fix. The coding agent's repeated test failure needed a new-hypothesis requirement and a retry cap — not broader filesystem access.

Layer 5 — Graph: coordinating the system

Graph Engineering is the operational paradigm for building complex AI agents and multi-agent systems by representing their workflows as explicit stateful graphs rather than relying on unstructured, single-agent loops or linear prompt chains. Instead of letting an LLM autonomously decide every execution step in an unpredictable loop ("prompt and pray"), graph engineering imposes architectural boundaries. It treats the overall task as a state machine where nodes execute discrete logic (LLM calls, tool execution, validation), edges direct routing decisions, and a schema-defined state persists throughout the lifecycle.

Graph engineering controls the topology of the whole workflow. Nodes can be agents, deterministic functions, evaluators, or human gates; edges define sequencing, routing, parallel branches, recovery paths, and where the loops from layer 4 actually live. Loop asks "how does this one agent keep working?" Graph asks "which component runs next, and how does the system coordinate?"

flowchart LR
    A[Triage] --> B[Planner]
    B --> C[Coding Agent]
    C --> D[Deterministic Tests]
    D -->|pass| E[Security Reviewer]
    D -->|fail| C
    E --> F{Human Approval}
    F -->|approved| G[Merge]
    F -->|rejected| B

That's the fix for the coding agent's fourth failure: an explicit route from code change to tests, to security review, to human approval before merge — instead of an implicit hope that the right person eventually sees it.

LangGraph frames itself as a low-level orchestration runtime for exactly this: mixing deterministic steps with model-driven steps while preserving state, durable execution, and human interrupts. The useful idea isn't "draw boxes and arrows" — it's splitting responsibilities that a single overloaded chat session was quietly doing all at once (plan, research, write, and approve its own work), and keeping a human where mistakes get expensive.

Graph complexity isn't free, and the data backs that up: Anthropic reported its multi-agent research system beat a single-agent setup by 90.2% on an internal breadth-first research evaluation — but the multi-agent runs consumed roughly 15x the tokens of a normal chat interaction. That number is specific to Anthropic's research workload, not a universal multiplier, but it captures the trade-off precisely: graphs earn their complexity only when the task's value and parallelism justify the bill. Reach for a graph because the workflow genuinely branches, not because orchestration frameworks are the interesting part of the stack right now.

Core Pillars of Graph Engineering

 [ Shared Typed State Object (e.g., Pydantic / TypedDict) ]
                         |
  +----------------------+----------------------+
  |                                             |
  v                                             v
[ Node: LLM / Tool / Task ] ---------> [ Conditional Edge ]
  |                                             |
  +----------------------+----------------------+
                         |
                         v
                [ Node: Validator / Human Checkpoint ]

1. State Management

The explicit data structure passed through every execution step in the graph.

Typed Schemas: Defines exact variables, tool outputs, message histories, and system metadata.
Reducers: Functions that determine how state field updates from parallel or sequential steps are merged (e.g., appending items to a list vs. overwriting a variable).

2. Nodes (Units of Execution)

Self-contained, bounded steps inside the system. A node takes the current state, performs logic, and returns a state patch.

Agentic Nodes: Specialized LLM calls tailored to a single role (e.g., Researcher, Refiner, Evaluator).
Deterministic Nodes: Standard code executions (API calls, data parsers, formatting utilities).
Validation Nodes: Output parsers and schema checkers that evaluate prior steps.

3. Edges (Control Flow & Routing)

Rules that connect nodes and govern system transitions.

Fixed Edges: Direct, deterministic routing from Node A to Node B.
Conditional Edges: Dynamic routing based on LLM outputs or state evaluation (e.g., if confidence < 0.8, route to Human Review; else route to Execution).
Cyclic Paths: Loops designed for iterative refinement (e.g., Draft -> Evaluate -> Revise -> Evaluate).

The concern that cuts across all five: evals

There's a sixth thread running through every layer, and it isn't a sixth ring — it's the measurement system for the other five.

Does the prompt reliably produce instruction-following output?
Did context retrieval include the decisive evidence, or silently omit it?
Did the harness allow the necessary tool and deny the dangerous one?
Did the loop stop for the right reason, or exhaust its budget on a dead end?
Did the graph route the risky case to a human, or let it fall through? OpenAI's own evaluation guidance follows the same discipline regardless of layer: define the desired behavior, run representative test inputs against explicit criteria, analyze results, iterate. Evals aren't a separate architectural concern sitting outside the five layers — they're the evidence that each one is doing its job.

Three honest caveats

The vocabulary isn't equally mature. Prompt engineering and context engineering are established industry terms. "Agent harness" is a real, recognized category. Loop engineering and graph engineering are newer labels for things practitioners have also called agent loops, workflows, and orchestration — useful names, not settled ones.

The boundaries leak. Memory can plausibly belong to context, harness, or loop-level runtime state. Verification can live inside a tool boundary, a retry loop, or its own graph node. These are five concerns, not five cleanly separable software components — don't force a rigid file-by-file mapping onto them.

This isn't a build order. In practice, the graph (the workflow shape) tends to get sketched first, loops and control boundaries get defined next, and prompts get tuned last. Think concentric controls around the model, not a waterfall you execute top to bottom.

Key takeaways

Five control layers sit between a raw model call and a trustworthy system: prompt (steers), context (informs), harness (constrains), loop (persists), graph (coordinates) — and the weakest one sets the reliability ceiling for the whole system.
Diagnose failures by layer, not by rewording the prompt. A misrouted PR, an infinite retry, and an overprivileged shell command are three different bugs in three different layers, and "improve the prompt" fixes none of them.
Graph engineering is topology, not decoration: nodes are agents, deterministic steps, or human gates; edges define sequencing, parallelism, recovery, and where a human has to sign off.
Multi-agent orchestration is a cost trade-off, not a free upgrade — Anthropic's own research system needed ~15x the tokens to get a 90.2% quality gain, and that ratio should inform whether your workflow actually needs a graph.
Evals aren't a separate layer — they're the proof that each of the other five is doing what you think it's doing, and without them "the AI was weird" is the best diagnosis your team will ever get.

Closing CTA

Before you rewrite another prompt: which of the other four layers — context, harness, loop, or graph — is actually the weakest link in your system, and how would you know?

MCP vs. Agent Skills: A Decision Framework for Context Engineering

shakti mishra — Wed, 22 Jul 2026 01:42:31 +0000

MCP vs. Agent Skills: What's the Difference and Which Do You Need?

As AI agents evolve beyond basic chat interfaces into fully autonomous systems, developers keep hitting the same architectural decision: how do we actually extend what an agent can do?

Two concepts dominate that conversation right now — Model Context Protocol (MCP) and AI Agent Skills (SKILL.md) — and they're routinely discussed as if they're competing options. They aren't. They solve two different problems that only look similar from a distance, and picking the wrong one for the job is one of the more common ways teams burn a sprint building an agent that's either disconnected from reality or has no idea what to do once it's connected.

Here's the actual distinction, how each one works under the hood, and a framework for knowing which one — or both — your next agent build actually needs.

The core concept: a badge versus a playbook

Imagine hiring a new developer for your team:

MCP is the interface that lets them connect to your databases, Slack channels, GitHub repos, and cloud infrastructure. The actual access is whatever credential is behind that connection; MCP just standardizes how the agent calls it.
AI Agent Skills are their Standard Operating Procedures. It's the playbook that tells them how your team formats code, writes pull requests, or triages production incidents. An interface without a playbook gets you a new hire who can technically reach every system but has no idea how your team actually wants things done — so they improvise, inconsistently, every time. A playbook without any system access gets you someone who knows exactly what to do but can't touch anything. Production-grade agents need both, and knowing which gap you're actually looking at is what keeps you from reaching for the wrong fix.

                    "How do I extend this agent?"
                                │
              ┌──────────────────┴──────────────────┐
              │                                      │
     Needs access to a                     Needs to follow a
     live external system                  repeatable procedure
              │                                      │
              ▼                                      ▼
      ┌───────────────┐                    ┌───────────────────┐
      │      MCP      │                    │   Agent Skills    │
      │ (interface)   │                    │   (the playbook)  │
      └───────────────┘                    └───────────────────┘

What is MCP (Model Context Protocol)?

MCP is an open standard designed to standardize how LLMs connect to external tools, databases, and APIs.

Before MCP, connecting an agent to a system like Postgres or GitHub meant custom API integrations or a bespoke function-calling schema per provider. Every new tool meant a new adapter; every new model meant rewriting that adapter to match whatever shape that provider's function-calling API expected. MCP replaces that with a universal bridge: build the connection once, as an MCP server, and any MCP-compatible client can use it without provider-specific glue code.

How it works

MCP runs on a client-server architecture over JSON-RPC:

┌────────────┐        MCP (JSON-RPC)        ┌──────────────────┐
│  AI Agent  │ ◄──────────────────────────► │  MCP Server      │
│  (client)  │                              │ (e.g. GitHub,    │
│            │                              │  Postgres, Slack)│
└────────────┘                              └──────────────────┘

An MCP server exposes three primitives to any connected client:

Resources — structured, read-only data the model can pull into context (a file, a database row, a ticket)
Tools — functions the model can actually invoke, with side effects (create a PR, run a query, send a message)
Prompts — reusable, parameterized prompt templates the server offers to the client A minimal client config for connecting an agent to a GitHub MCP server:

{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_TOKEN": "your_token_here"
      }
    }
  }
}

Once connected, the agent can list open issues, read a file from a repo, or open a pull request — not because it memorized GitHub's REST API, but because the MCP server translated GitHub's API surface into a standard interface the agent already knows how to speak. Worth noting: the actual permission boundary here is GITHUB_TOKEN, not the MCP layer itself — MCP standardized the calling convention, and whatever that token is scoped to do is what the agent can do.

MCP is the right layer when the problem is access to live, changing state: current inventory counts, today's support tickets, a database that gets written to every minute. That data can't be baked into a prompt or a static file, because it's stale the moment you write it down.

What are AI Agent Skills?

If MCP is about connecting to systems, Skills are about encoding expertise. A Skill is a folder — typically a SKILL.md file plus optional scripts, templates, or reference documents — that teaches an agent how to perform a specific, repeatable task the way your team wants it done.

---
name: pr-review-checklist
description: Use when reviewing a pull request before approval. Covers this team's standards for tests, security, and rollback safety.
---

# PR Review Checklist

Before approving, verify:
1. Tests cover the new/changed logic, not just the happy path
2. No secrets or credentials in the diff
3. Migration steps are documented if the schema changed
4. There's a rollback plan if this touches a production data path

How it works: progressive disclosure

The architectural idea behind Skills is progressive disclosure. An agent doesn't load every Skill's full content into context at all times — that would burn tokens on procedures it isn't currently using.

Always in context:              Loaded only on demand:
┌─────────────────────┐         ┌──────────────────────────┐
│ pr-review: description │  ───► │ Full SKILL.md body      │
│ triage: description    │       │ + bundled scripts/      │
│ formatting: description│       │   templates, only when  │
└─────────────────────┘          │ the task actually matches│
                                 └──────────────────────────┘

Only the name and description stay resident by default — a line or two each. When a task matches a Skill's description, the full body loads. If the Skill references bundled scripts, those load only when actually needed. That's what makes Skills cheap to accumulate: dozens of them can sit around — commit conventions, an incident-triage runbook, a data-visualization style guide — without a standing context-window tax for the ones not currently in use.

Skills are the right layer when the problem is repeatable procedure or expertise: a style guide, a checklist, a domain-specific workflow. A Skill that's pure prose is genuinely static — it only changes when someone edits it. A Skill that bundles a script or calls tools on the agent's behalf is executable code riding along with a markdown file, and it's worth reviewing with the same rigor as any other code change: ownership, version, and what permissions it actually exercises.

MCP vs. Skills: the key differences

	MCP	Agent Skills
Solves	Access to external systems and live data	Encoding repeatable procedures and expertise
What it actually controls	The calling convention — permission comes from the credential behind it	The agent's behavior; bundled scripts mean real code execution
Requires infrastructure	Yes — a running server, auth, network calls	No — just files in a repo or agent config
Data/artifact freshness	The data fetched through it is live and dynamic	The Skill file itself is static prose, or executable if it bundles scripts
Context cost	Tool/resource schemas loaded as needed	Name + description only, until matched
Typical use case	Query a database, call an API, send a message	Follow a style guide, run a checklist, format output consistently
Portability	Tied to the system it connects to	A folder of files — copy it anywhere (though bundled scripts still carry their own assumptions)

They aren't competitors — they work together

The most common mistake is treating this as either/or. In production, the strongest agents use both, for different halves of the same job.

Take a support agent handling refund requests:

MCP servers give it live access to Stripe (the actual charge), Zendesk (the actual ticket), and the order database (actual fulfillment status).
A Skill gives it your team's actual refund policy: which situations qualify for a full refund vs. store credit, what tone to use, when to escalate to a human instead of resolving automatically. Strip out the Skill and the agent has perfect system access but no policy — it improvises, inconsistently, every time. Strip out MCP and it knows the policy perfectly but can't verify whether the charge even went through. And when both are wired together, it's worth making sure the Skill's conclusion doesn't directly trigger the MCP write with no review step in between — the policy decision and the action it authorizes are worth keeping separately inspectable.

┌───────────────────────────────────────────────────────┐
│ Support Agent                                         │
│                                                       │
│ Skill: "refund-policy"        MCP: Stripe, Zendesk,   │
│ (how to decide, what to say)   order DB (what's true) │
│         │                              │              │
│         └──────────────┬───────────────┘              │
│                         ▼                             │
│          Grounded, policy-correct refund decision     │
└───────────────────────────────────────────────────────┘

A decision framework: which do you actually need?

Does this task need information or actions from a live, external system? A database that changes, an API that has to be called, a message that has to be sent — that's MCP territory.

Does this task need to be done the same specific way every time, based on knowledge that doesn't come from an external system? A formatting convention, a checklist, a domain-specific procedure — that's a Skill.

Does it need both? Most real production agents do. Answering system access and procedural knowledge as two separate questions is what keeps the architecture clean instead of cramming everything into one oversized prompt.

Key Takeaways

MCP standardizes the calling convention, not permissions — the actual access boundary is the credential behind the MCP server, not the protocol itself.
Agent Skills standardize knowledge — they package repeatable procedures into files an agent loads only when relevant, via progressive disclosure. Pure-prose Skills are static; Skills that bundle scripts are executable code and worth reviewing as such.
MCP requires running infrastructure (a server, auth, a network connection); Skills are just files — no infrastructure required, and portable, though a bundled script's assumptions travel with it.
The two solve different halves of the same problem: the strongest production agents combine both, Skills for the "how," MCP for "what's actually true right now."
The fastest way to pick correctly: ask whether the task needs live external state (MCP), repeatable procedure (Skills), or both — most real agents need both.

Closing CTA

If you're building an agent right now, take an honest inventory of what it's actually missing: is it fumbling because it can't reach a system it needs, or because it can reach everything but has no consistent playbook for what to do once it's there? Those are two different fixes, and reaching for the wrong one is the fastest way to burn a sprint on the wrong problem. What's your agent stack missing — access, procedure, or both?

Loop Engineering: The Six-Layer Architecture Behind Self-Improving Agents

shakti mishra — Sun, 12 Jul 2026 20:39:06 +0000

Loop Engineering: The Six Architectural Layers That Separate Self-Improving Agents From Expensive Function Calls

Most AI agents in production today are glorified function calls. They take an input, run inference, produce an output, and forget everything the moment the response streams back. Ship one on Monday, and it's exactly as smart on Friday six weeks later as it was on day one — same edge cases, same wrong answers, same manual overrides eating the ROI you promised leadership.

There's a term circulating in the agent-building community for the architectural fix to this: loop engineering. It's not a new model, a framework you pip install, or a prompt technique. It's a way of thinking about the difference between a system that answers and a system that compounds.

If you're running agents in production and they've stalled, this is almost certainly why.

The enterprise pain point, concretely

Picture a claims-triage agent at a mid-sized insurance company. It reads incoming claims, classifies them, pulls the relevant policy, and drafts a recommendation for a human adjuster.

Launch day: 60% of routine claims handled without escalation. Leadership is thrilled.

Six weeks later: still 60%. The same ambiguous policy clauses get misread the same way, every Monday. Adjusters start double-checking everything, and the time savings evaporate.

The diagnosis isn't the model. It's the architecture. This is an open-loop system:

Input → Process → Output → (stop)

Nothing flows back. The agent has no way to notice it was wrong, no way to learn from the adjuster's correction, no memory of last week's mistake. It's a very expensive function call.

A closed-loop system looks like this instead:

Input → Process → Output → Feedback → Improve → (back to Input)

That second arrow — the one that bends back to the start — is the entire game. But it isn't magic. People talk about self-improving agents as if you sprinkle a feedback step on top and intelligence emerges. It doesn't work that way. The loop is held together by six real architectural decisions, each with real trade-offs.

The six layers

flowchart LR
    A[Automations<br/>trigger layer] --> B[Worktrees<br/>parallel execution]
    B --> C[Skills<br/>procedural memory]
    C --> D[Connectors<br/>real-world links]
    D --> E[Sub-agents<br/>validation layer]
    E --> F[Memory<br/>state layer]
    F -.feedback.-> A

Layer 1 — Automations: the trigger layer

A self-improving system needs to start without you. Automations initiate workflows based on time, events, or system conditions — an email arrives, a database row updates, a claim lands in the queue.

This is the shift from "I asked it" to "it noticed and started on its own." An agent you have to invoke is a tool. An agent that responds to the world is a system.

The catch: triggers are where runaway loops are born. An event-driven agent that triggers another event can cascade. Before wiring automations, define the kill switch and the rate limit first. A loop that starts itself must also be a loop you can stop.

Layer 2 — Worktrees: the parallel execution layer

Once work starts itself, you hit a throughput wall. One agent processing one task at a time doesn't scale to enterprise volume.

The name borrows from git's worktree concept — the same one coding agents already use — where multiple branches exist side by side without stepping on each other. Applied to agents, it means multiple instances execute independently across isolated branches, with three properties that matter:

Task isolation — branches don't interfere with each other, so one agent's work can't corrupt another's.
Concurrent processing — speed improves without collision risk.
Independent state — each branch carries its own context. The common mistake: scaling vertically first (throwing a bigger model at a serial pipeline) when the real constraint is concurrency. Worktrees solve that at the architecture layer, not the model layer.

Layer 3 — Skills: the procedural memory layer

This is where the loop starts to feel intelligent. Skills are reusable units of logic — the "how to do this" an agent shouldn't have to rediscover on every run. A skill is:

Step-by-step — an execution pattern that improves reasoning across workflows
Modular — plug-and-play, so agents adapt faster
Reusable — written once, applied everywhere it fits Picture a skill pool: code understanding, data analysis, web search, summarization, database queries. The agent core combines the right skills per task — a triage claim needs policy lookup plus summarization; an appeal needs document comparison plus precedent search. Same pool, different combinations.

The payoff: you stop stuffing every instruction into the prompt and instead give the agent a library of capabilities it composes on demand. Prompts get shorter, behavior gets more reliable, and the system gets cheaper to maintain.

Layer 4 — Connectors: the real-world links layer

An agent reasoning in isolation is a chatbot. An agent that touches real systems is infrastructure. Connectors are the links out:

APIs — REST, GraphQL, custom endpoints for external data and actions
Data and databases — SQL, NoSQL, cloud storage for persistence
Tools and protocols — MCP servers, CLI commands, web search, enterprise systems "What does the policy say?" is a retrieval question. "Flag this claim for fraud review and notify the adjuster" is a write action — and the second one demands you take security seriously. Every connector is an attack surface and a blast radius. The discipline that matters: least privilege and identity. Narrowest permissions that finish the job, every call authenticated, every action auditable.

The Model Context Protocol (MCP) has become the common standard here precisely because it bakes identity and policy into the connection instead of bolting it on afterward.

Layer 5 — Sub-agents: the validation layer

This is the layer separating teams that ship reliable agents from teams that ship confident hallucinations. The pattern: separate the builder from the judge.

Generator → produces output
     ↓
Validator → checks against rules, policy, known failure modes
     ↓
Approval gate → only validated output moves forward
     ↓
Feedback loop → flagged issues route back for another pass

One agent generates the claim recommendation. A separate agent — different prompt, different job — checks it against policy constraints, regulatory rules, and known failure cases. Pass, and it ships. Fail, and it goes back with a specific reason.

Why two agents instead of one checking its own work? Because a model grading its own output shares the same blind spots that produced the error in the first place. Multiple eyes — even both models — catch more. This is the operational version of "your evals are your moat": the validator sub-agent is where your eval logic lives at runtime, not just in a test suite. Continuous validation is what turns a 60% agent into a 90% agent without touching the underlying model.

Layer 6 — Memory: the state layer

The final layer is what makes the loop a loop instead of a circle that resets to zero. Memory preserves state across passes:

Context storage — past interactions persist across conversations and runs
Retrieval — relevant history is fetched to inform the current decision
Learning loop — system behavior updates based on outcomes Without memory, every pass starts fresh and the agent makes the same mistake forever.

Here's the precise mechanism worth sitting with: the loop does not make the model smarter. The model weights never change. What changes is the context the system carries into each pass. Memory is the substrate that lets feedback from layer five accumulate instead of evaporate. Each pass leaves something behind for the next one to use — that's the whole trick.

Why no single layer is the magic

Put all six on the loop and you see how they hand off:

Automations start it. Worktrees parallelize it. Skills and connectors do the work. Sub-agents validate it. Memory carries the lesson forward to the next pass.

A trigger with no memory is just automation.
Memory with no validation is a system that confidently remembers wrong answers.
Validation with no connectors is an agent grading work it never actually did. The loop only compounds when all six are present and wired in sequence. Skip one and you don't get a slightly worse loop — you get a different failure mode entirely.

What this means when you're staring at a stalled agent

When reviewing an agent architecture, don't start by asking which model was chosen. Ask one question first: where does the loop close?

If the answer is "it doesn't," that's the entire diagnosis — and no amount of prompt tuning fixes a missing feedback layer. For most teams, the fix isn't a better model. It's three of the six layers they skipped. Add a connector to read the adjuster's accept-or-override signal, a validator sub-agent that enforces policy constraints before drafting, and a memory layer that stores every override as a new case. Within a month, the agent moves off its plateau — because for the first time, it can actually learn from the humans correcting it.

The risk nobody's pricing in yet

As loops improve and grow more autonomous, there's a real cost that gets ignored: humans stop understanding the underlying mechanics. As the human-in-the-loop gets eliminated, teams accumulate comprehension debt — a widening gap between what the system does and what anyone actually understands about how it does it. Left unchecked, that becomes cognitive surrender: trusting these systems so completely that nobody's tracking what's happening at all.

Staying the engineer — keeping a working mental model of the system even as it improves without you — is the part of loop engineering that doesn't show up in the architecture diagram but matters just as much.

Key takeaways

Open-loop agents plateau by design. Input → Process → Output → stop means the system is exactly as smart on day 100 as day one, no matter the model.
Six layers close the loop: automations (trigger), worktrees (parallelism), skills (procedural memory), connectors (real-world actions), sub-agents (validation), and memory (state) — each is a distinct architectural decision, not a checkbox.
Validation beats a bigger model. A separate sub-agent checking output against policy and known failure modes is what moves a 60% agent to 90%, because a model can't catch its own blind spots.
Memory doesn't change the weights — it changes the context. The intelligence gain is entirely in what gets carried forward into the next pass, not in the model getting smarter.
Every connector is a blast radius. Least privilege, authentication, and auditability aren't optional once an agent moves from reading data to writing actions.

Closing thought

If your agent has stalled at some mediocre-but-tolerable accuracy for weeks, the fix probably isn't a bigger model or a cleverer prompt — it's a missing layer in the loop. Which of the six are you actually missing: the trigger, the parallelism, the skill, the connector, the validator, or the memory? And if you've closed all six, what's stopping your team from understanding the system it built?

The Missing Context Plane: Why Enterprise Data Architecture Needs a Third Layer for AI Agents

shakti mishra — Sun, 28 Jun 2026 20:24:17 +0000

Your Data Stack Is Working Exactly As Designed — Here's Why Your AI Agents Keep Failing

The data warehouse is not broken.
The transformation layer is doing its job.
The dashboards are accurate.
And your AI agents are still making decisions that would make any experienced analyst wince.

Here's the uncomfortable truth: everything your data stack was built to do, it does well. The problem is that it was built for a different reader — a human one. And when you swap that reader out for an autonomous agent, the gaps that humans quietly filled through judgment, institutional memory, and context become catastrophic failure points.

What the Stack Was Actually Built For

For the past decade, the data architecture workflow looked like this:

Raw Sources → Ingestion → Warehouse → dbt Transform → Metrics Layer → Dashboard → Human Analyst

That last arrow is load-bearing in ways we never had to make explicit.

The human analyst brought what no pipeline could deliver: business context. They knew why a metric moved — because of a pricing experiment, a large customer churn, or a one-time bulk order that skewed the averages. They knew which dashboard was canonical and which one the finance team secretly maintained with different definitions. They knew which policy could bend for a strategic account and which exception had been pre-approved by legal.

None of that lived in the warehouse. It lived in people, Slack threads, one-on-one conversations, and the kind of organizational muscle memory that gets transferred through onboarding and osmosis.

AI agents don't get onboarded.

The seam, Agents Can't Navigate

Modern enterprise data stacks are actually a collection of specialized tools stitched together:

Ingestion: Fivetran, Airbyte
Storage: Snowflake, BigQuery, Databricks
Transformation: dbt
Metrics layer: MetricFlow, Cube
Data catalog: Alation, Collibra, Atlan
Governance: varied
Observability: Monte Carlo, Great Expectations
BI: Tableau, Looker, Power BI Each tool is good at its job. Together, they create seams. Humans learned to navigate those seams invisibly — knowing which field in which system was "the real one," or that the definition of "active customer" changed in Q3 2023 and the old reports haven't been backfilled.

When an AI agent queries a metric, it gets the number. It doesn't get:

The lineage of how that number was computed
The definition revision history
The exception carved out for Enterprise tier customers
The operational judgment about when to trust it versus when to dig deeper The agent sees data. It doesn't see the map that makes the data safe to act on.

Agent Query → Data Returned → [GAP: missing context, policy, memory] → Decision Made

That gap is where the bad recommendations come from.

Governance Without Context Is Not Governance

This is the part that catches most teams off guard.

Traditional governance stacks treat the problem as a layering issue: put permissions here, lineage there, business glossary somewhere else, and tie them together through integrations. The assumption is that combining these artifacts produces governed data.

It doesn't.

Governance isn't a rule. It's the act of deciding how a rule applies in context.

A pricing policy applies until it doesn't — for a strategic customer, for a partner relationship, during a specific campaign window. An approval chain is correct unless the person who can override it has already done so verbally and the exception hasn't been captured anywhere structured.

Those exceptions are not edge cases. They represent a significant portion of where the organization's real judgment, authority, and institutional knowledge actually lives. If that judgment exists only in a Slack thread or an email chain, an AI agent following the documented rule can be technically compliant and operationally wrong at the same time.

The Enterprise Memory Gap

Here's a dimension the traditional stack almost entirely ignores: reasoning memory.

The modern data stack is excellent at storing artifacts:

Tables ✓
Dashboards ✓
Logs ✓
Metrics ✓ It is weak at preserving the reasoning that produced them:
Why was this definition changed?
Who approved this exception and why?
What happened the last time we made this type of decision?
Which tradeoff was accepted and under what constraints? For human teams, this doesn't matter much — people ask their colleagues, pull up old meeting notes, or rely on tribal knowledge. The cost is friction and occasional mistakes. For AI agents operating at scale, this is a structural failure. An agent without access to reasoning history will re-derive decisions that were previously made and rejected, repeat past mistakes, and miss constraints that were hard-won through prior experience.

Human Decision Process:
  [Data] + [Policy] + [Memory of Past Decisions] + [Contextual Judgment] → Action

AI Agent (current):
  [Data] + [Policy] → Action (missing memory + contextual judgment)

Retrieval-augmented approaches can pull fragments from unstructured sources, but it's brittle. A semantic search over Slack messages and Confluence docs is not the same as a structured memory layer that an agent can query with confidence.

The Architecture Shift: Adding a Context Plane

The framework that emerges from this analysis is a three-plane architecture:

┌─────────────────────────────────────────────────┐
│                  CONTEXT PLANE                  │
│  Semantic models · Policies · Memory · Exceptions│
│  Lineage · Definitions · Governance rules        │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│                  CONTROL PLANE                  │
│  Permissions · Auth · Orchestration · Audit logs │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│                   DATA PLANE                    │
│  Ingestion · Storage · Transformation · Metrics  │
└─────────────────────────────────────────────────┘

The data plane is what you have today. The control plane handles auth and orchestration. The context plane is what's missing.

The context plane isn't a search index bolted onto your existing stack. It needs to be:

Produced as work happens — context captured at the point of decision, not reconstructed retroactively
Governed alongside data — with the same rigor as the data itself
Queryable by agents — structured enough that an agent can retrieve it with confidence, not just probabilistically
Updated continuously — because the business context changes as fast as the business does Building this from scratch is hard. The practical path for most enterprises will be evolutionary: identify the workflows where agents are expected to act, map the context and governance those workflows require, and layer in an AI-ready context layer that can unify structured data, unstructured knowledge, policies, memory, and orchestration incrementally.

What "Context-Driven" Actually Means in Practice

The shift from "data-driven" to "context-driven" isn't marketing language — it's an architectural requirement that shows up concretely in how you design agent systems.

A data-driven agent gets a number and acts on it.

A context-driven agent gets a number, plus:

The semantic definition of that metric
The lineage showing how it was computed
The policies governing what actions are permitted
The exceptions that apply in the current situation
The memory of how similar situations were handled before
The confidence signal about data freshness and reliability The interface changes from SELECT metric FROM table to something much closer to a business context API:

context = get_business_context(
    metric="revenue",
    customer_tier="enterprise",
    exception_scope="current_campaign_window",
    include_reasoning_history=True
)

# Returns: value + definition + lineage + applicable policies + past decisions

That's a different architectural bet than most teams are making today.

Key Takeaways

The data stack was optimized for human interpretation — it's working as designed, but the reader changed. AI agents don't inherit the tacit map humans use to navigate data safely.
The seams between tools are where agents fail — permissions in one place, definitions in another, exceptions in a Slack thread. Humans navigate these invisibly; agents hit them hard.
Governance is contextual, not rule-based — a rule that can't express its own exceptions is not enough for an agent operating at scale.
Reasoning memory is the missing layer — storing artifacts (tables, dashboards) is not the same as preserving the reasoning that produced decisions.
The next data architecture adds a context plane — alongside the data plane and control plane, a first-class layer for semantic models, policies, memory, and governed reasoning.

The Question Worth Debating

If context needs to be "produced as work happens" — captured at the point of decision — who owns that? Data engineering? The domain teams making the decisions? Platform teams building agent infrastructure?

Most organizations aren't close to having a clear answer. And until they do, the agents will keep making technically correct but operationally wrong decisions.

Where is your team with this? Are you building a context layer, patching it with RAG, or still hoping the warehouse is enough?

Microsoft AI Agent Architecture in 2026: A Decision Framework for Copilot Studio, Foundry, Agent Framework, and Logic Apps

shakti mishra — Sun, 14 Jun 2026 19:15:12 +0000

Stop Picking AI Frameworks Before You Answer This One Question

If you're building AI agents on Azure, you've been here: a kickoff meeting where someone asks "what framework should we use?" and the next 90 minutes becomes a debate between Semantic Kernel, AutoGen, Copilot Studio, Foundry, and Logic Apps — before anyone has agreed on what the agent actually needs to do.

That is the wrong sequence. And it's why so many Azure AI projects start with the most sophisticated-sounding option and spend the next three months fighting the wrong abstraction.

The Microsoft ecosystem gives you more agent architecture flexibility than any other cloud provider. Conversational or headless. Low-code or pro-code. M365-native or Azure-native. That flexibility is a genuine advantage — but only if you read the decision tree in the right order.

This post walks through that tree. By the end you'll have a repeatable framework for choosing the right architecture the first time, plus a list of SDK retirement dates you need to know before you commit.

The Platform Landscape First

Before the decision tree, it helps to understand the three tiers Microsoft gives you:

┌─────────────────────────────────────────────────────────────────┐
│               Microsoft Agent Platform Tiers                     │
│                                                                   │
│  ┌───────────────────┐  ┌──────────────────┐  ┌──────────────┐  │
│  │  Copilot Studio   │  │  Copilot Studio  │  │  Microsoft   │  │
│  │      Lite         │  │   (Pro Makers)   │  │   Foundry    │  │
│  │                   │  │                  │  │              │  │
│  │ M365 scenarios    │  │ More actions,    │  │ Pro devs,    │  │
│  │ Low complexity    │  │ channels,        │  │ custom code, │  │
│  │ No-code           │  │ orchestration    │  │ full control │  │
│  └───────────────────┘  └──────────────────┘  └──────────────┘  │
│        Simple ◄─────────────────────────────────► Advanced       │
└─────────────────────────────────────────────────────────────────┘

These aren't competing products. They're a spectrum. The question is which tier fits your team's constraints, not which one is "best."

The One Question That Drives Everything

Before frameworks, before data layers, before deployment — the single most important decision is:

What is the interaction pattern?

There are exactly three answers:

                    ┌─────────────────────┐
                    │  What is the        │
                    │ interaction pattern? │
                    └──────────┬──────────┘
                               │
             ┌─────────────────┼─────────────────┐
             ▼                 ▼                 ▼
    ┌─────────────┐   ┌─────────────────┐  ┌──────────────┐
    │ Chat / UI   │   │   Autonomous /  │  │  Headless /  │
    │  (User      │   │  Event-Driven   │  │   API        │
    │  talks to   │   │  (Runs in the   │  │  (Systems    │
    │  the agent) │   │   background)   │  │  call it)    │
    └─────────────┘   └─────────────────┘  └──────────────┘

This fork determines nearly every downstream decision. Teams that skip it and jump straight to framework selection spend months fighting the wrong abstraction. Teams that start here make better choices in minutes.

Path 1: Chat / UI-Based Agents

If a human will talk directly to the agent, follow this branch.

Step 1: Where does the user live?

Is the experience entirely within Microsoft 365?
│
├── YES → M365 Copilot
│         Best fit when users already work in the Microsoft
│         ecosystem and you want strong user-in-the-loop
│         experience without a custom application layer.
│
└── NO  → Does it need Teams or multi-channel support?
          │
          ├── YES (need speed) → Copilot Studio
          │                      Best for business workflows,
          │                      fast time-to-production,
          │                      avoiding a full engineering sprint.
          │
          └── YES (need control) → Pro-code path ↓

Step 2: The pro-code fork

When you need a conversational UI with full engineering control, one more decision determines the right SDK:

Is this M365-centric?
│
├── YES → M365 Agents SDK
│         Deepest integration with Teams, Outlook, SharePoint.
│         Handles auth, channel routing, and M365 data access
│         without custom plumbing.
│
└── NO  → Azure-centric path
          │
          Do you need a custom, interactive frontend
          (streaming, shared state, embedded web/app UI)?
          │
          ├── YES → Agent Framework + AG-UI
          │         AG-UI is an emerging protocol that gives
          │         agents a standard way to talk to modern
          │         frontends. Critical when your agent is
          │         embedded in a React app, not just a chat
          │         window.
          │
          └── NO  → Microsoft Foundry
                    Cleanest Azure-first path when you
                    don't need the custom UI protocol layer.

The summary for UI-based agents comes down to four questions:

Is this conversational? (Yes — you're on this path)
Where does the user interact? (M365, Teams, or custom?)
Low-code or pro-code?

4. M365-first or Azure-first?

Path 2: Autonomous / Event-Driven Agents

These agents run in the background. They react to triggers. They process information and take action with minimal or no direct user interaction. The architecture here follows the workflow, not the chat UI.

Autonomous Agent Decision Tree:
│
├── Want low-code? → Copilot Studio with Event Triggers
│                    Good starting point for business-user-built
│                    automation with Power Automate triggers.
│
├── Need custom UI protocol? → Agent Framework + AG-UI
│                              Same AG-UI story as the chat path —
│                              richer app experiences with
│                              streaming and state.
│
├── Azure-centric + enterprise controls? → Microsoft Foundry
│                                          Deep governance,
│                                          model routing, tracing,
│                                          and evaluation built in.
│
└── Heavy enterprise system integration?
    (SAP, ServiceNow, Salesforce, etc.)
    → Logic Apps AI Agent Workflows
      Logic Apps gives agents access to 1,400+ enterprise connectors.
      When integration is the core problem — not model selection —
      the connector story matters more than the model story.
      These workflows can now be exposed as MCP tools for
      standardized agent access.

Key insight for this path: Autonomous agents aren't chatbots with the chat stripped out. They are workflow systems. The architecture choice should follow the workflow shape, the trigger sources, and the integration surface — not the sophistication of the model you want to run.

This matters especially for enterprise workflows where the agent's job is to call SAP, read from ServiceNow, and write back to Salesforce. The agent framework is almost irrelevant compared to the integration layer.

Path 3: Headless / API Services

No chat window. No copilot interface. The agent exposes a backend service that other systems call. The main variable here is hosting requirements:

Headless Agent Hosting:
│
├── Managed platform service → Azure Container Apps
│                              Best default — handles scaling,
│                              secrets, and networking without
│                              managing Kubernetes directly.
│
├── Local / Edge deployment → Azure IoT Edge / Container Apps
│                              For agents that need to run
│                              close to data or offline.
│
└── Full self-hosting control → AKS
                               When you need full orchestration
                               control, custom networking, or
                               multi-agent mesh configurations.

This branch is less about user experience and more about runtime shape. When the agent is infrastructure — called by other services, not by users — the deployment target and SLA requirements drive the decision.

The Data Layer: Don't Treat It as an Implementation Detail

Here is where most Azure AI projects underinvest. The quality of your agent's responses will depend far more on the retrieval layer than on the model you choose.

Does the agent need custom data?
│
├── Microsoft 365 data (emails, docs, SharePoint)
│   → Graph Connectors + M365 data layer
│
├── Unstructured documents at scale
│   → Azure AI Search (Agentic RAG)
│     Semantic ranking, hybrid search (dense + sparse),
│     integrated chunking and vectorization.
│     Critical when your knowledge base is PDFs, contracts,
│     or policy documents.
│
├── Vector search on structured/semi-structured data
│   → Choose based on existing infrastructure:
│     • Cosmos DB (NoSQL + vector)
│     • PostgreSQL with pgvector
│     • Azure SQL with vector support
│     • Fabric SQL (if analytics context needed)
│
└── Analytics context (dashboards, metrics, BI data)
    → Microsoft Fabric
      When the agent needs to reason over business intelligence
      or time-series data alongside documents.

A practical rule: if the agent answers questions by retrieving information, the retrieval architecture is the most important design decision you'll make. The model is a commodity; how you feed it isn't.

Deployment: Where Does the Agent Need to Live?

After interaction model, build approach, and data layer, the final decision is where users or systems reach the agent:

Target Audience	Deployment Path
M365 users in Copilot	M365 Copilot channel
Teams users	Teams channel
Web or mobile users	Web/mobile channel
Multiple surfaces	Multi-channel SDK route
Other Azure services / APIs	Azure Container Apps, App Service, or AKS

And once you've decided: don't skip monitoring. Observability, governance, and cost visibility need to be part of the design from day one, not retrofitted after the first production incident.

The Retirement Dates You Must Know Before You Commit

This section belongs in every Azure AI architecture conversation right now. Three SDKs have announced or confirmed retirement dates within current planning horizons:

SDK / Service	Retirement Date	Successor
Bot Framework	December 31, 2025	M365 Agents SDK
azure-ai-inference SDK	May 30, 2026	openai SDK
Assistants API	August 26, 2026	Foundry Agent Service / Responses API

If any of these are in your current stack or your evaluation shortlist, start the migration now.

The cost of discovering a retirement date mid-project — after you've trained your team on the SDK, written documentation, and built integrations — is always higher than starting on the right platform. This is especially true for teams with multi-quarter planning cycles.

The Bot Framework retirement is already past as of this writing. If you're still on it, the M365 Agents SDK migration is not optional.

The Short Version (Put This on Your Whiteboard)

STEP 1: What is the interaction pattern?
        Chat UI → go to Step 2
        Autonomous → go to Step 3
        Headless → go to Step 4

STEP 2: Chat UI path
        M365 only? → M365 Copilot
        Low-code + multi-channel? → Copilot Studio
        Pro-code + M365-first? → M365 Agents SDK
        Pro-code + custom UI? → Agent Framework + AG-UI
        Pro-code + Azure-first? → Foundry

STEP 3: Autonomous path
        Low-code → Copilot Studio + Event Triggers
        Enterprise integrations → Logic Apps AI Agent Workflows
        Custom UI protocol → Agent Framework + AG-UI
        Azure-first + governance → Foundry

STEP 4: Headless path
        Managed → Container Apps
        Edge → IoT Edge / Container Apps
        Full control → AKS

Then: Choose your data layer based on data type.
Then: Choose your deployment channel based on who/what calls the agent.
Then: Wire in monitoring and governance before you ship.

Check retirement dates. Start on the right platform.

Key Takeaways

The single most important decision in Azure AI agent architecture is the interaction pattern (Chat/UI, Autonomous, Headless). Every other choice flows from it — framework debates before this question is answered are premature.
Low-code (Copilot Studio) is a strategic choice, not a compromise. If your users already live in Teams and M365, it often gets you to production faster than a custom pro-code solution with equivalent results.
The data layer determines agent quality more than the model tier. Choosing Azure AI Search, pgvector, Cosmos DB, or Fabric depends on the shape of your data — decide this early.
Three SDKs have confirmed retirement dates within 2026 planning horizons: Bot Framework (retired Dec 2025), azure-ai-inference SDK (May 2026), and Assistants API (Aug 2026). Audit your stack now.
Logic Apps AI Agent Workflows with 1,400+ enterprise connectors is the right path when the core problem is integration, not model selection. The connector story matters more than the model story in those scenarios.

The Question Worth Sitting With

Every team I've seen debate Azure AI frameworks starts with "which one is most powerful?" The right question is "which one fits the shape of what we're actually building?"

If you've been through an Azure AI architecture decision recently — what forced the framework choice? Was it the interaction model, the data layer, team skill, time pressure, or something else entirely?

Drop it in the comments. Hearing what drove real decisions is more useful than any decision tree.

12 Engineering Habits That Cut LLM Token Spend at Production Scale

shakti mishra — Sun, 31 May 2026 22:23:02 +0000

Your LLM Bill Isn't One Big Leak. It's Twelve Small Ones.

A team shipped a great AI feature in their product. The cost had quietly tripled in six weeks — same model, same product, no obvious explanation.

That is not a model problem. That is a habits problem.

Token prices are falling. Enterprise AI bills are climbing. That apparent contradiction resolves instantly when you look at the real culprit: volume grows faster than price drops. Google now processes over a quadrillion tokens a month. Deloitte's 2026 CFO guidance names AI the fastest-growing line item in tech budgets. You can have cheaper tokens and a higher bill simultaneously — and most teams do.

The teams that survive this are not the ones with the cheapest model or the best negotiated rate. They are the ones who have engineered clean habits into how they build. Every call. Every feature. Every deploy.

Here are the twelve habits that matter.

Why Token Prices Falling Doesn't Save You

Before the tactics, let's understand the structural trap.

Token price per 1M: $15 → $8 → $4 (falling fast)
Monthly token volume: 10M → 80M → 600M (growing faster)

Month 1 bill: 10M × $15/M = $150
Month 4 bill: 600M × $4/M = $2,400

This is not hypothetical. It is the trajectory of every team that ships an AI feature without cost discipline. The price per token drops. The total bill explodes. And nobody notices until the CFO asks why the AI line item tripled.

The fix is not a cheaper model. The fix is twelve cleaner habits.

The 12 Engineering Habits

1. Choose the Right Model for the Task

Most teams reach for the biggest model by default. Big mistake.

The biggest model is rarely the right one — it is just the safest-feeling one. Every request deserves to be routed to the smallest, cheapest model that still clears the quality bar.

Here is the routing heuristic I use:

Task Type	Model Tier	Examples
Routing, classifying, extracting	Small	GPT-5.4-mini, Claude Haiku 4.5
Drafting, summarizing	Mid	GPT-5.4, Claude Sonnet 4.6
Hard reasoning, final answers	Large	GPT-5.5, Claude Opus 4.8

Most production workloads are small-model work wearing a large-model price tag. Map your request types, measure output quality at each tier, then route accordingly. A simple classifier — yes, possibly a small model — can do this routing cheaply.

def route_model(task_type: str) -> str:
    routing_map = {
        "classify": "claude-haiku-4-5",
        "extract":  "claude-haiku-4-5",
        "summarize": "claude-sonnet-4-6",
        "draft":    "claude-sonnet-4-6",
        "reason":   "claude-opus-4-8",
        "generate_final": "claude-opus-4-8",
    }
    return routing_map.get(task_type, "claude-sonnet-4-6")

2. Reduce Input Tokens

You pay for every token you send — not just the ones you get back.

A 2,000-token system prompt on every call is a permanent tax. You pay it on every single request, forever. Teams pile on examples, repeat instructions across calls, and paste in giant context blocks "just in case." The model does not need your life story.

Audit your system prompts. Run them through a tokenizer. Most teams find 30–50% is cuttable on first pass — redundant examples, stale instructions, context the model never uses.

Write tight. Send less. Then measure whether quality actually dropped. Usually it doesn't.

3. Limit Output Tokens

Long answers feel thorough. They are also expensive.

Set hard output token caps. Ask for bullet points, structured responses, or explicit length constraints in your prompts:

# Instead of open-ended generation
messages = [{"role": "user", "content": "Explain how X works."}]

# Be explicit about length
messages = [{"role": "user", "content": "Explain how X works. Answer in 3 concise sentences, no more."}]

A 50-token answer beats a 500-token one — for the bill and, often, for the user. Verbose model outputs frequently bury the signal in filler. Constraining output length improves both cost and quality.

4. Use Caching Aggressively

Your users ask the same questions over and over. Stop re-generating the same answers.

Two caching layers matter here:

Exact-match caching: Hash the prompt, cache the response. Trivial to implement, immediate ROI for any FAQ-style traffic.

Semantic caching: Use embedding similarity to catch near-duplicate prompts even when the wording varies. A question about "how do I reset my password" and "I forgot my password, what do I do?" should hit the same cached response.

Prompt prefix caching: Most providers now offer cached prefixes for the static portion of long system prompts. If you have a 1,500-token system prompt on every call, prompt caching alone can cut your input cost in half. This is free money. Take it.

Architecture with semantic cache:

User Query
    │
    ▼
[Embedding Model] ──→ [Vector Cache Lookup]
                              │
                    ┌─────────┴──────────┐
                    │ HIT (similarity>0.92)│  MISS
                    ▼                     ▼
             [Cached Response]    [LLM API Call] ──→ [Store in Cache]
                    │                     │
                    └──────────┬──────────┘
                               ▼
                         [User Response]

5. RAG Done Right

Bad RAG is a silent budget killer. Teams dump huge context chunks and hope the model sorts it out. The model does sort it out — and charges you for every token while doing so.

The fix is surgical retrieval:

Chunk small: Smaller, focused chunks mean tighter retrieval. A 256-token chunk beats a 2,000-token chunk if it's more semantically precise.
Retrieve only what the current question needs: Don't retrieve your entire knowledge base. Top-K retrieval with a relevance threshold, not a fixed K.
Trim before you send: After retrieval, run a lightweight reranker or even a cheap model pass to strip irrelevant retrieved chunks before they hit the main context window. Good RAG is not "more context." It is less, but exactly right.

6. Batch Requests

One-by-one API calls carry overhead. Batching eliminates it.

Most providers offer a batch API tier at steep discounts (often 50% off) for non-time-sensitive jobs. If you are processing documents, embedding a corpus, or running evaluations, there is no reason to pay real-time pricing.

# Don't do this for batch workloads
for document in documents:
    result = client.messages.create(...)  # sequential, real-time pricing

# Do this instead
batch_response = client.beta.messages.batches.create(
    requests=[
        {"custom_id": doc.id, "params": {"model": "...", "messages": [...]}}
        for doc in documents
    ]
)
# Runs async, often 50% cheaper

Overnight processing jobs do not need real-time pricing. Route them to the batch tier.

7. Use Function Calling and Structured Output

Free-text answers are expensive to parse. The workflow becomes: generate verbose text → write parsing code → handle edge cases → re-ask when it breaks. Every layer adds tokens.

Function calling and structured output collapse this:

# Expensive: free-text answer you then parse
response = "The user's sentiment is positive, their intent is to purchase..."

# Cheap: structured output with zero parsing overhead
response_schema = {
    "sentiment": "positive | negative | neutral",
    "intent": "purchase | browse | support | other",
    "confidence": 0.0  # float
}

Define a schema. Ask for clean JSON. Skip the natural-language wrapper. Fewer tokens, fewer retries, zero glue code.

8. Build a Prompt Library

Every team rewrites the same prompt five slightly different ways. Each version drifts. Each one gets longer. Nobody owns it.

Build a library of proven, parameterized templates instead:

PROMPT_LIBRARY = {
    "classify_support_ticket": """
Classify this support ticket into exactly one category: {categories}.
Respond with the category name only, no explanation.

Ticket: {ticket_text}
""",
    "summarize_document": """
Summarize the following document in {max_sentences} sentences.
Focus on: {focus_areas}.

Document: {document_text}
"""
}

Write the prompt once, tune it once, measure it once, reuse it everywhere. This is a cost win, a quality win, and a maintainability win.

9. Use a Tiered, Two-Step Approach

Not every request deserves your most expensive model. Filter first.

                    All Requests
                         │
                         ▼
                [Cheap Model — Triage]
                  classify / draft / filter
                         │
               ┌─────────┴──────────┐
               │                    │
         [Simple Case]        [Complex Case]
        ← 80% of traffic →    ← 20% of traffic →
         Return cheap          Escalate to
          model output        expensive model

Most traffic never needs the expensive model. A cheap triage pass — classifying request complexity, filtering noise, handling simple cases directly — routes only the genuinely hard problems up the cost stack. The savings are often dramatic: 60–80% of production traffic handled at small-model cost.

10. Monitor Token Spend by Feature

You cannot fix what you cannot see.

Most teams have no idea which feature burns the most tokens. They guess, they guess wrong, and then they optimize the wrong thing.

Instrument at the feature level:

import time

def tracked_completion(feature: str, messages: list, **kwargs):
    start = time.time()
    response = client.messages.create(messages=messages, **kwargs)

    # Emit to your metrics system
    metrics.increment("llm.input_tokens", 
                       response.usage.input_tokens,
                       tags={"feature": feature})
    metrics.increment("llm.output_tokens",
                       response.usage.output_tokens,
                       tags={"feature": feature})
    metrics.timing("llm.latency_ms",
                   (time.time() - start) * 1000,
                   tags={"feature": feature})
    return response

Track spend by feature, by user tier, by request type. Find the top three cost offenders. Fix those first. A dashboard you check weekly beats a heroic cleanup after the bill explodes.

11. Treat Provider Pricing as a Procurement Decision

Provider pricing is not static. It changes by model, by region, by commitment tier, by workload type. Most teams pick a provider once and never revisit it.

Treat this like infrastructure procurement:

Compare providers on your actual workload mix before committing to production volume
Evaluate reserved capacity and committed-use discounts (often 30–60% off on-demand)
Consider whether a different region offers lower latency AND lower cost
Run periodic cost-per-quality benchmarks across providers as models evolve This is a procurement decision that engineering should own, not just ops. The savings potential is often larger than all twelve other habits combined.

12. Optimize Embeddings Usage

Embeddings feel cheap per call. At scale, they are not.

The most common waste patterns:

Re-embedding the same content repeatedly (no caching)
Embedding near-duplicates separately (no deduplication)
Making individual embedding calls instead of batching

Fix all three:

import hashlib

# Cache embeddings by content hash
def get_embedding(text: str, cache: dict) -> list:
    content_hash = hashlib.sha256(text.encode()).hexdigest()

    if content_hash in cache:
        return cache[content_hash]  # Free — already computed

    embedding = embedding_client.embed(text)
    cache[content_hash] = embedding
    return embedding

# Batch embedding generation
def batch_embed(texts: list[str]) -> list[list[float]]:
    return embedding_client.embed_batch(texts)  # Single API call, batch pricing

Small per-call cost, massive total cost at production volume. Worth the cleanup.

The Pattern Under All Twelve

Read the list again. Notice something?

None of these are clever. There is no secret model flag, no undocumented API parameter, no vendor trick. Every single strategy reduces to the same move: send fewer tokens, and only the ones that matter.

LLM cost optimization is not a one-time project. It is a discipline you bake into how your team builds — the same way you bake in logging, testing, and error handling.

How These Map to Azure

If you are building on Azure, the platform has native homes for most of these strategies:

Strategy	Azure Service
Model routing (small/mid/large)	Azure AI Foundry model catalog — deploy mixed tiers, route between them
Caching + input token reduction	Azure API Management (AI gateway) — semantic caching, token limits, token metrics
RAG optimization	Azure AI Search — hybrid retrieval, tight chunking
Monitoring + token spend tracking	Azure Monitor + API Management token metrics
Embeddings caching	Azure Cosmos DB or Azure AI Search vector store
Batch processing	Azure AI Foundry batch deployment tier

You do not have to wire all twelve by hand. Wire the platform.

Key Takeaways

Token prices are falling; your bill is rising. Volume grows faster than price drops — this is structural, not accidental.
Most LLM costs are habits, not features. Oversized models, verbose prompts, uncached repeated calls, and sequential batching are the main culprits.
Start with three this week: right-size your model routing, cap output tokens, and enable prompt prefix caching. Those three alone typically cut the bill by a third.
Monitor at the feature level. You cannot optimize what you cannot measure. Token spend dashboards by feature reveal the 20% of requests driving 80% of cost.
Cost optimization is a design choice made on every API call — not a quarterly cleanup project.

The Question Worth Arguing About

Here is something I genuinely wrestle with: as models get cheaper and context windows get larger, some of these habits become less critical. Maybe in two years, sending a 10,000-token context window on every call will cost pennies and nobody will care.

Or maybe volume will grow to fill any price decrease, and the discipline matters more than ever.

Where do you land on this? Are you actively optimizing LLM costs in production, or betting that price curves solve the problem for you? Tell me in the comments — I am genuinely curious whether teams are treating this as engineering infrastructure or hoping the market handles it.

The 5-Layer Architecture Every Production Multi-Agent System Needs (And Why Most Skip Layers 4 and 5)

shakti mishra — Mon, 25 May 2026 23:15:27 +0000

Your Multi-Agent AI System Is Just a Dinner Party With No One in Charge

Picture this. Dinner guests arriving in an hour. Four people, each capable, each assigned a job.

One handles the grill. One sets the table. One makes the salad. One runs the music.

Thirty minutes in: the grill isn't heating because nobody opened the propane valve. The salad person is waiting on ingredients that were never passed over. The appetizers are cold because the reheating was supposed to happen ten minutes ago. The DJ paired to the wrong speaker and is now blasting techno into the baby's room.

No one was incompetent. Everyone knew their job. The whole thing fell apart because there was no system for coordination.

That is a near-perfect description of most multi-agent AI systems running in production today.

Each agent is capable — a coder, a researcher, a planner, a writer. But without shared memory, deliberate orchestration, and proper state management, capable agents produce incoherent results. The failure isn't in the intelligence of the individual agents. It's in the architecture that's supposed to make them a team.

This post breaks down the five-layer architecture that separates production multi-agent systems from expensive demos — and names the specific failure modes you will hit if you skip any of them.

The Three Ways Multi-Agent Systems Fail Before You Ship

Before the architecture, the failure taxonomy. These three problems appear, in some combination, in nearly every multi-agent system that didn't make it to production:

The Chaos Problem. No orchestration means agents act in parallel without coordination. One agent fetches data while another modifies it. One writes a response while another has already decided the query requires escalation. The outputs contradict each other, or worse — they corrupt shared state.

The Amnesia Problem. Agents can't access context from previous steps in the workflow. Each call starts fresh. An agent that just retrieved customer history has no way to pass that context to the agent writing the response — unless you explicitly build the memory layer. Most teams don't, until it's too late.

The Black Box Problem. Something goes wrong. You have no trace of which agent made which decision, what state the system was in, or what inputs triggered the failure. You can't reproduce it. You can't fix it. You can only watch it happen again.

If any of these sound familiar from your own experiments, keep reading — the architecture below is designed to close all three gaps.

The Five-Layer Architecture

Here's the framework: five layers that must all be functional before a multi-agent system can deliver consistent value in production. Think of them as load-bearing walls. You can skip one in a prototype. You cannot skip one in production.

┌─────────────────────────────────────────────────────┐
│          Layer 1: Orchestration                     │
│   Orchestrator · Classifier · Agent Registry       │
├─────────────────────────────────────────────────────┤
│          Layer 2: Knowledge                         │
│      Source Bases (RAG) · Vector DBs               │
├─────────────────────────────────────────────────────┤
│          Layer 3: Agents                            │
│   Specialized Agents · MCP Client · Local/Remote   │
├─────────────────────────────────────────────────────┤
│          Layer 4: Storage                           │
│  Conversation History · Agent State · Registry DB  │
├─────────────────────────────────────────────────────┤
│          Layer 5: Integration & Observability       │
│    MCP Server · External Tools · Trace · Evals     │
└─────────────────────────────────────────────────────┘

Layer 1 — The Orchestration Layer: Your AI Conductor

This is the component that kills the dinner party chaos problem. Without it, you have a group chat where everyone shouts simultaneously. With it, you have a conductor who decides who plays, when, and with what information.

The orchestrator is responsible for:

Routing tasks to the correct agent(s)
Managing execution order and sequencing
Preventing duplicate or conflicting work
Synthesizing outputs from multiple agents into a coherent result Embedded within the orchestration layer is a Classifier — a component using NLU or LLM-based intent detection to understand what kind of request just arrived. "This needs the research agent." "This needs both the research and writing agents, in sequence." "This is ambiguous and requires a clarification step before routing."

The Agent Registry is the orchestrator's phonebook. It knows what agents exist, what capabilities each one exposes, and whether each agent is currently available. At small scale (2–3 agents), this is trivial. At production scale with dozens of specialized agents, a governed registry is the only way to keep routing reliable without hard-coding every path.

Microsoft's Agent Framework (MAF) — a fusion of Semantic Kernel and AutoGen — implements this pattern. But the concepts apply regardless of framework. LangGraph's node-based routing, CrewAI's role-based delegation, and custom orchestrators all need to solve the same problem: deterministic routing with dynamic capability discovery.

Layer 2 — The Knowledge Layer: Institutional Memory

Agents need two kinds of knowledge access: domain-specific content and semantic search over unstructured data.

Source Bases are where you store the specialized content that transforms general-purpose AI responses into expert answers. Policy documents. Product FAQs. Regulatory guidelines. Internal runbooks. The implementation varies — knowledge graphs, document repositories, fine-tuned models — but the goal is consistent: give agents the specific information they need to be right about your domain, not just right in general.

Vector databases enable semantic search over that content. When a support agent searches "issues with login after password reset," vector search understands the semantic relationship between authentication state and credential management. Keyword matching doesn't.

The critical retrieval decision that most teams get wrong: RAG vs. MCP is not a style preference. It is a functional distinction.

Use RAG when:
  - Content is static or semi-static (policy docs, FAQs, guides)
  - Search relevance is the primary quality lever
  - You need to synthesize across multiple documents

Use MCP when:
  - The agent needs real-time system state
  - The operation writes or modifies data
  - You need live API access (inventory, CRM, ERP)

"How many units of SKU-123 are in stock right now?" is not a search question. It's an API call to your ERP. Routing it through RAG produces a stale answer. Routing it through an MCP tool call produces the live value.

The mistake teams make: reaching for RAG everywhere because it's simpler to set up, then spending three months debugging why the agent keeps giving wrong inventory data. The answer isn't better embeddings. The answer is the wrong retrieval pattern.

Layer 3 — The Agent Layer: Specialized Workers

Each agent in the system is a specialist. A finance agent. A coding agent. A research agent. A customer-facing support agent. Each is fine-tuned or prompted for its domain, with access to the relevant subset of the knowledge layer.

Agents communicate with external tools via MCP Client — a standardized interface that handles authentication, manages connections, and formats requests consistently regardless of the target tool. This abstraction is what lets you swap out the underlying tool (say, switching from one search provider to another) without rewriting every agent that uses it.

Local vs. Remote Agents: The Security Distinction That Matters

This is the architectural decision most teams don't think about until something goes wrong.

Local agents run in the same execution environment as the orchestrator. They communicate in-memory. They inherit the orchestrator's trust context. Fast, low-latency, straightforward to reason about.

Remote agents operate across a network boundary. They might live in a different security zone, be owned by a different team, or be an external service. This creates five security requirements that don't apply to local agents:

1. Authentication:  Verify the remote agent's identity before accepting its outputs
2. Authorization:   Enforce what data and tools the remote agent can access
3. Trust boundary:  Never assume a remote agent has the same permissions as the orchestrator
4. Data in transit: Encrypt everything crossing the network boundary
5. Audit:           Log every cross-boundary call with identity and payload

Think of local agents as colleagues sharing an office — implicit trust, fast coordination. Remote agents are external contractors calling in. They need a badge, credentials, and an access review before you hand them anything sensitive.

Agent-to-Agent (A2A) protocol handles the standardized communication pattern for remote agents. Microsoft Entra Agent Identity provides the identity infrastructure on Azure. But the discipline is organizational, not just technical — you need policy decisions about which agents can call which other agents before you write a single line of orchestration code.

Layer 4 — The Storage Layer: Where Most Systems Fail Quietly

This is the layer that kills the amnesia problem. It is also, consistently, the layer teams underbuild first.

A production multi-agent system requires three distinct types of persistent storage:

Conversation History — Every interaction, decision, and intermediate output across the workflow. This is what lets an agent in step 7 know what the agent in step 2 found. Without it, each agent starts from zero. With it, context accumulates across the full workflow.

Agent State — The operational status and working configuration of each agent instance. If an agent crashes mid-task, agent state is what lets it recover — or lets a replacement agent pick up exactly where it left off. Without this, a transient failure means restarting the entire workflow.

Registry Storage — Persistent metadata about what agents exist, what capabilities they expose, what their current health status is, and what their recent performance looks like. This is what backs the Agent Registry in Layer 1.

The typical failure pattern: teams build agent state in memory. Works fine in development. Works fine in testing. Falls apart the first time an agent crashes in production, because the state was ephemeral and the workflow can't resume.

Build persistent storage from day one. Retrofitting it into a production system that was designed around ephemeral state is significantly harder than building it correctly at the start.

Layer 5 — Integration and Observability: How You Stop Flying Blind

This is the layer that kills the black box problem. It is also, almost universally, treated as an afterthought — and then desperately retrofitted after the first production incident that nobody could debug.

MCP Server — the standardized interface that external tools expose to your agents. Databases, APIs, web search, calculators, code execution environments. The MCP Server pattern means agents interact with external tools through a consistent interface, with authentication and audit controls baked in, rather than through a proliferation of custom integrations that each have their own auth model and failure mode.

Observability — real-time visibility into every agent action in the system:

Which agents are currently active?
What tasks are in progress?
Where are the latency bottlenecks?
What is the per-agent token consumption and cost?
Where do failures cluster? Without this, you're flying blind. You will know something is wrong when a customer complains. You will not know which agent caused it, what state the system was in, or how to reproduce it.

Evals (Evaluation Layer) — the feedback loop that makes your system better over time. How accurately are agents completing their assigned tasks? Where are they making errors? What types of inputs cause failures? This data feeds back into the orchestration layer and the knowledge layer, enabling continuous improvement.

Without evals: you know your system is broken when users tell you
With evals:    you know your system is degrading before users notice

The evaluation layer is how you close the loop between production behavior and system improvement. Without it, you're not iterating on a system — you're waiting for complaints.

Why This Architecture Is Engineering Pragmatism, Not Research

What makes this five-layer model compelling isn't novelty. It's that it solves the concrete problems that show up in real production deployments:

Scalability — New agents can be added without rewriting orchestration logic. The registry discovers new capabilities automatically. The routing classifier routes to them without hardcoded rules.

Debuggability — Proper observability and persistent state mean that when something fails, you can trace exactly what happened. Every agent action is logged. Every state transition is recorded. Failure is reproducible.

Reliability — Persistent agent state means individual failures don't cascade. A crashed agent can be restarted and resume where it left off. The supervisor pattern in the orchestration layer catches local failures before they propagate.

Flexibility — Local and remote agent separation means different parts of the system can scale independently based on load and security requirements. The knowledge layer can be updated without touching the agent layer.

Key Takeaways

The failure is almost never in the model. The three failure modes — chaos (no orchestration), amnesia (no memory), black box (no observability) — are architectural failures. Upgrading to a better model doesn't fix them.
Build the storage layer first. Agent state persistence is the most commonly underfed layer and the hardest to retrofit. Design for it from day one, even if your first version is simple.
RAG and MCP are not interchangeable. One retrieves from static content via search. The other calls live systems via APIs. Using the wrong one for the job produces wrong answers that look right.
Remote agents need explicit security architecture. Authentication, authorization, trust boundary enforcement, and audit logging are not optional once you cross a network boundary. Plan the policy before you write the routing code.
Observability is not an afterthought. You cannot improve a system you cannot see. Trace every agent action. Measure token cost and latency per agent. Feed production failures back into your eval set within the week.

Closing Thought

The dinner party didn't fail because the guests were bad at cooking. It failed because there was no system — no shared plan, no handoff protocol, no one tracking dependencies.

Multi-agent AI systems fail the same way, for the same reason. Not because the models are weak. Because the architecture isn't there.

The question worth sitting with: which of these five layers is the weakest link in the system you're currently building?

If you've shipped a multi-agent system and hit one of these failure modes — or found a pattern that works better than what's described here — I want to hear about it. Drop it in the comments.

How AI Agents Will Reshape E-Commerce Architecture — And What Developers Need to Build Now

shakti mishra — Sun, 17 May 2026 22:01:13 +0000

Your Next Customer Won't Click. They'll Delegate.

The commerce interface is changing. And it's not another redesign.

Think about the last time you booked a business trip.

You opened six tabs. You compared prices on three airline sites. You checked your credit card rewards portal. You pulled up the hotel loyalty app. You looked at Google Maps for the office address. You Googled cancellation policies. You re-entered your corporate card number on the fourth checkout page.

Now consider this instead:

"Book my Chicago trip. Keep it under travel policy. Prioritize hotels near the board office. Use loyalty points only if the redemption value makes sense. Book dinner for four on Tuesday. Ask me before the final purchase."

One instruction. An AI agent handles the rest — comparing options, checking policy compliance, querying loyalty status, booking the restaurant, creating the itinerary, then surfacing a single approval screen.

You don't browse. You delegate intent.

That shift — from human navigation to agent execution — is the premise behind agentic commerce. And if you're an engineer building systems that touch retail, payments, loyalty, fulfillment, or customer identity, this isn't a future-state thought experiment. The protocol stack making this possible is being standardized right now.

Why This Is an Architectural Problem, Not a UX Problem

Most discussions about AI agents in commerce focus on the customer experience side: "What will shopping feel like?" That's the wrong frame for engineers.

The more useful question is: What does your system need to expose for an AI agent to trust and transact with it?

Your landing pages, checkout flows, and banner ads were designed for a human with 200ms of attention. An AI agent has different requirements:

Structured, machine-readable product data
Programmatic access to pricing, inventory, cancellation policies, and promotions
A secure, permissioned payment layer with auditable authorization
Real-time fulfillment status with clear SLA signals If any of these are locked inside a human-facing frontend with no API surface, you don't just have a bad UX — you're invisible to agents.

The Six-Layer Stack Enabling Agentic Commerce

The acceleration here isn't coming from one model. It's coming from a stack of interoperability protocols converging at the same time.

1. Model Context Protocol (MCP)

Agents need more than prompts. They need a standardized way to call tools, retrieve context, and operate across systems without every team building one-off integrations.

In a commerce context, MCP means an agent can check your live inventory, apply a loyalty rule, read a return policy, and create a support case — all through a consistent interface. Think of it as the "USB standard" for agent-to-system communication.

2. Agent-to-Agent Protocol (A2A)

Commerce won't stay as one agent talking to one website.

Customer agents will negotiate with merchant agents. Broker agents will aggregate options across providers. Fulfillment agents will coordinate between inventory, shipping, and support. A2A, pushed forward by Google and the broader ecosystem, defines how agents exchange capabilities, task status, context, and multimodal outputs across vendor boundaries.

Without A2A, every agent integration is a bespoke negotiation. With it, agents interoperate like services in a well-designed microservices architecture.

3. Agent Payments Protocol (AP2)

This is the clearest sign that agentic commerce is leaving demo territory and entering the transaction layer.

The core primitive is the mandate — a structured object that ties together user intent, cart details, spending limits, and payment authorization into a single auditable unit.

{
  "agent_id": "travel_agent_789",
  "customer_id": "customer_123",
  "allowed_category": "business_travel",
  "max_transaction_amount": 1200,
  "approval_required_above": 500,
  "valid_until": "2026-06-30",
  "purpose": "Chicago board meeting"
}

This isn't just payment data. It's identity + consent + policy + fraud control in one verifiable package.

Merchants need to know the agent had authority. Customers need revocation controls and spending floors. Payment networks need fraud signals. Enterprises need audit trails. The mandate solves all of these at once — without requiring a human to be in the loop for every transaction.

4. Computer-Use Agents

When APIs don't exist, agents use the browser.

Computer-use agents can click buttons, fill forms, and navigate UI flows — effectively automating workflows that were never designed for programmatic access. It's fragile at scale and hard to govern, but for niche domains where building a formal API isn't cost-justified, it's a working bridge to the agentic future.

The implication: even if you haven't invested in an API layer, agents will find a way in. Better to build the surface intentionally.

5. Contextual Personalization with Memory

Traditional personalization asks: "What did this user click last week?"

Agentic personalization asks: "What is this person trying to accomplish right now, given their current context, stated constraints, historical preferences, and active policy rules?"

The architectural shift is from collaborative filtering over past behavior to intent inference over live context. This requires memory architectures, preference stores, vector search, and policy-aware recommendation layers — not just a recommendation engine.

6. Dynamic Real-Time Planning

What makes the travel scenario actually work isn't the flight booking. It's the replanning.

If the flight changes, the itinerary adjusts. If the hotel sells out, the agent finds the next best option that still satisfies the policy constraints. If an expense crosses a threshold, the workflow escalates to a human approval step.

This is orchestration with conditional branching across external state — not a linear task list. The agent manages a live plan, not a static sequence.

The Three Ways Agents Interact With Your Business

┌─────────────────────────────────────────────────────────┐
│                 AGENTIC COMMERCE INTERACTION MODELS      │
├─────────────────┬──────────────────┬────────────────────┤
│  AGENT-TO-SITE  │ AGENT-TO-AGENT   │  BROKERED A2A      │
│                 │                  │                    │
│ Customer agent  │ Customer agent   │ Broker agent sits  │
│ visits your     │ talks directly   │ between customer   │
│ website via     │ to your business │ and multiple       │
│ browser or API  │ agent            │ merchants          │
│                 │                  │                    │
│ Your site is a  │ Negotiation and  │ Your business is   │
│ machine-readable│ real-time offer  │ one option in      │
│ decision surface│ exchange layer   │ someone else's     │
│                 │                  │ ranking system     │
└─────────────────┴──────────────────┴────────────────────┘

Agent-to-site is the starting point: an agent scrapes your catalog, reads your policies, and completes checkout via your existing interface or API. Your website stops being a purely human experience and becomes a machine-readable decision surface.

Agent-to-agent is where it gets commercially interesting. Instead of scraping, the customer's agent contacts your business agent directly:

Customer agent: "I need a hotel near downtown Chicago for three nights. My traveler has Gold status. Can you improve the offer if they add breakfast?"

Merchant agent: "Yes. 12% off if breakfast is bundled and the reservation is prepaid."

That's not a chatbot interaction. That's a commerce negotiation layer — requiring access to inventory, pricing, loyalty rules, margin thresholds, fraud signals, and payment authorization.

Brokered agent-to-site is the model every product leader should be paying attention to. A third-party broker agent sits between the customer and many businesses, comparing options and recommending a transaction. Think Expedia or Instacart, but generalized across categories and powered by agents instead of static search.

If the broker owns the customer relationship, your business becomes an option in someone else's ranking function. That ranking function will score on price, availability, reliability, refund flexibility, fulfillment confidence, and API quality — not brand loyalty.

The Six Domains to Audit Right Now

Agentic commerce isn't a marketing channel upgrade. It's an infrastructure readiness problem. Here's where to look first:

1. Agent Discoverability

Your catalog needs to be legible to agents — not just humans. That means structured metadata, accurate availability signals, machine-readable policies, and programmatic access via APIs or MCPs.

SEO doesn't disappear, but the question shifts from "can Google index this?" to "can an agent confidently determine when my offer is the best fit for a user's stated intent?"

2. Loyalty and Clienteling APIs

Most loyalty systems were built for humans checking points in a mobile app. Agents need to query loyalty tier status, unused reward balances, eligible promotions, redemption rules, and personalized offers — programmatically, with proper permissioning.

If your loyalty program is human-only, agents will skip it. If it exposes clean, secured APIs, agents can actively use it to make better decisions on behalf of the customer.

3. Payment Mandates and Fraud Defense

This is the hardest surface to get right.

Recommending a product is one thing. Spending money is another. Your payment system needs to answer: who authorized this agent, what is the spending limit, what categories are allowed, when is human approval required, and how does the merchant verify the agent had authority?

The mandate structure in the AP2 spec (see JSON above) gives you a template. The implementation details — token management, revocation, fraud signal integration, audit logging — are where most teams will underestimate the work.

4. API-First Commerce Platform

If core commerce actions (search, cart, pricing, promotions, checkout, order status, returns, support escalation) live only inside a frontend, agents will struggle or fall back to fragile browser automation.

The architectural shift is from website-first commerce to capability-first commerce. The website becomes one client. Agents become another. Both consume the same underlying APIs.

5. In-Store and Physical Handoff

Agentic commerce won't stay digital. A customer agent may pre-authorize a purchase, apply a loyalty benefit, and schedule a pickup — all before the customer walks into a store. If the in-store POS can't read that agent-initiated context, the handoff breaks.

Omnichannel isn't a new idea, but agentic commerce makes agent-initiated online journeys completing in physical channels a first-class requirement rather than a nice-to-have.

6. Real-Time Fulfillment and Returns APIs

Agents will track, change, cancel, return, and escalate on behalf of customers. That requires real-time visibility into order state, inventory, shipping, return eligibility, and replacement availability.

A slow or poorly documented fulfillment API doesn't just create a bad experience — it can cost you the sale before a human customer ever sees your brand, because the broker agent ranked you lower based on fulfillment confidence.

The Market Signal Behind This

McKinsey projects that by 2030, the US B2C retail market could see up to $1 trillion in orchestrated revenue from agentic commerce, with global projections reaching $3–5 trillion.

That's not a number to build a product roadmap around directly. But it's a signal worth taking seriously when deciding whether API-first commerce infrastructure is a 2-year investment or a 5-year one.

Claude's recent releases of finance-focused and small business agents is an early indicator of how quickly vertical specialization is happening. When vertical-specific agents exist and are being actively used, your industry isn't far behind on the adoption curve.

Key Takeaways

The customer interface is shifting from clicks to delegated intent. Agents browse, compare, negotiate, and transact on behalf of users. Your checkout button may become an artifact of the human-browsing era.
Three interaction models define agentic commerce: agent-to-site (machine-readable storefronts), agent-to-agent (negotiation layers), and brokered agent-to-site (third-party ranking systems you don't control).
MCP, A2A, and AP2 are the protocol foundation. These three specifications are converging to make agents interoperable, transactable, and auditable at scale.
Payment mandates are the trust primitive. AP2 mandates combine identity, consent, policy, and fraud control into a single auditable object — the unit of authorization for autonomous purchases.
Six audit domains matter most right now: agent discoverability, loyalty APIs, payment mandates, API-first platforms, physical POS integration, and real-time fulfillment. Falling behind on any one of them makes you invisible or low-ranked in the agentic commerce stack.

Closing CTA

If agentic commerce scales the way the protocol investment suggests, the question for most product and engineering teams isn't whether to prepare — it's which surface to expose first.

So here's what I'm curious about: If an AI agent were evaluating your system right now, what would it find hardest to interpret — your pricing, your policies, your loyalty system, or your payment layer? And is that the same problem your human customers complain about?

Drop your answer in the comments. I suspect the answer is the same surface in both cases.