DEV Community: Ali Suleyman TOPUZ

Your Knowledge Graph Is Measuring Your Pipeline, Not Your Subject: A Replication Attempt

Ali Suleyman TOPUZ — Fri, 02 Oct 2026 15:31:01 +0000

I left a comment on a Medium piece a few weeks ago about agentic graph engineering across 82.8 million edges, the one about catching hidden vulnerabilities by walking task graphs, capability graphs, and run graphs instead of asking an LLM to reason over the whole mess. What stuck with me about that piece was a design choice buried in a caveat: the graphs were built with zero LLM calls, and the model only showed up later as an optional judged layer. I said at the time that this is exactly the kind of caveat that makes me trust the rest of the numbers. I meant it as praise for that specific project. I did not expect it to come back and bite my own assumptions about a completely different kind of graph a week later.

The piece that did the biting was “Your Knowledge Graph Is Describing Your Pipeline, Not Your Subject.” The claim, as I understood it: run the same knowledge-graph extraction pipeline on genuinely unrelated subjects, and a specific structural metric, prerequisite-concept density, lands in roughly the same place every time, regardless of what the graph is supposedly about. I want to be upfront that I could not get a clean read of the article’s own numbers. The mirror I could reach served meta tags and a sitemap link instead of the article body, and I’m not going to pretend I verified specific figures I never actually saw. What I can verify is the shape of the claim, because it’s a claim I can test cheaply on my own machine with my own domains, and that’s what this piece is: not a critique of someone else’s numbers, but my own smaller, honest attempt at the same experiment.

Here’s why I think this matters more than it sounds like it should. If I build a knowledge graph over my own codebase, or over a set of tickets, or over anything I plan to hand to an agent for retrieval, and I compute some structural metric on it, node count, edge density, whatever, that number is only useful if it’s telling me something true about the thing I modeled. If the number is actually a fingerprint of my extraction prompt and nothing else, then every downstream decision I make from it, tune the prompt because the graph looks too sparse, trust this domain’s graph more because it’s denser, is a decision made from noise dressed up as signal. That’s not a hypothetical for me. I’ve got a domain model for a SIM billing platform at work that I’d eventually like an agent to reason over via a graph instead of raw schema dumps, and before I do that, I want to know whether the graph I’d build would actually reflect that domain or just reflect however I happened to phrase the extraction prompt.

Picking domains I actually know

The replication only means something if the domains are genuinely unrelated and I can personally judge whether the extracted graph is honest, because I’m both the person writing the source documents and the person doing the extraction, and I need to know enough about each subject to catch the extractor inventing a relationship that isn’t there.

I picked three: ASP.NET Core’s middleware pipeline (backend work I do daily), SQL query optimization (also daily, also unrelated to the first in almost every way that matters, one is about request lifecycle and ordering, the other is about statistics and physical execution plans), and cellulose acetate slab production, which is the small artisanal materials business I’ve been building on the side and has nothing to do with software at all. Wood-pulp derived bioplastic, plasticizer ratios, a heated hydraulic press. If a pipeline artifact shows up identically across “how ASP.NET Core orders its middleware” and “how you press a CA slab without it warping,” that’s a much harder result to explain away as domain overlap than picking three flavors of backend engineering.

I wrote a fixed-size source document for each domain myself, deliberately, rather than pulling one from the web, because I wanted to control length and information density as closely as I could without pretending they’re perfectly identical: 875 words for the ASP.NET Core piece, 903 for SQL optimization, 820 for the CA slab piece. Close enough that word count isn’t doing the explaining, not so identical that I’d have to fake it.

The pipeline, for real

The extraction step is a fixed prompt against an LLM, one relationship vocabulary shared across all three domains, no domain-specific tuning anywhere. Here’s the actual extractor, built to talk to any OpenAI-compatible endpoint, which matters because it means you can point this at a paid API or at a free local Ollama model without changing a line of the extraction logic itself:

RELATION_TYPES = [
    "prerequisite_for", "part_of", "produces",
    "causes", "configures", "alternative_to", "measures",
]
EXTRACTION_PROMPT = """You are a knowledge graph extraction system. Read the
technical document below and extract every distinct concept as a node, and
every explicit relationship between two concepts as an edge.
Rules:
- Each node needs a short "id" (snake_case) and a human-readable "label".
- Each edge needs "source", "target", and a "type" drawn ONLY from this list:
  {relation_types}
- Only extract relationships the text actually states or clearly implies.
  Do not invent relationships that make the graph look more connected than
  the source material supports.
- Return ONLY valid JSON, no prose, no markdown code fences, in this exact
  shape:
  {{"nodes": [...], "edges": [...]}}
Document:
---
{document}
---
"""
def extract(document_text: str, model: str = "qwen2.5:3b-instruct") -> dict:
    client = build_client()
    prompt = EXTRACTION_PROMPT.format(
        relation_types=", ".join(RELATION_TYPES), document=document_text
    )
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1,
    )
    return json.loads(response.choices[0].message.content)

If you want to run this without a cloud bill, the local path is two commands away:

ollama pull qwen2.5:3b-instruct
ollama serve

and then pointing the client at http://localhost:11434/v1 with any placeholder API key, since Ollama's OpenAI-compatible endpoint doesn't check it. I want to be straight about one thing here: for the extraction passes that produced the numbers in this article, I didn't route through a tiny 3B local model. I ran the same fixed prompt, the same schema, the same instructions, through a frontier model instead, treating its output exactly the way the pipeline code above treats an API response, parsed and validated the same way. If anything that should work against the hypothesis, not for it. A stronger model has more room to notice "wait, these three domains actually are different" and produce genuinely divergent graphs. A pipeline artifact that survives a stronger model is a more uncomfortable result than one that only shows up on a 3B model too small to know better.

The bug that actually happened

The naive version of the parser, the one I first wrote before I’d seen a single real response, does exactly what the code snippet above does: hands the raw response straight to json.loads. First real run, it crashed:

CRASHED as expected: Expecting value: line 1 column 1 (char 0)

The model wrapped its JSON in a code fence despite the prompt saying not to, so I added a fence-stripper that checks whether the response starts with three backticks and trims the first and last line if so. Second run, same crash. Turns out the response wasn’t a bare fenced block, it opened with a one-line preamble, “Here is the extracted graph:”, before the fence started, so the naive “does this start with backticks” check never matched, and the whole thing fell through as unparsed prose again. The actual fix had to search for a fenced block anywhere in the response, not assume it sits at position zero:

FENCE_RE = re.compile(r"```

(?:json)?\s*\n(.*?)

```", re.DOTALL)
def strip_code_fence(raw: str) -> str:
    match = FENCE_RE.search(raw)
    if match:
        return match.group(1).strip()
    return raw.strip()

Small thing, but it’s the kind of small thing that eats an afternoon if you don’t hit it early and cheaply on a toy pipeline before it’s wired into something that matters. I’d rather find “the model sometimes narrates before it complies” on a three-document experiment than on a production ingestion job three months from now.

What actually happened

Here’s what the fixed pipeline produced across all three domains, computed with real networkx calls against the extracted graphs, not eyeballed:

+------------------------------+-------+-------+-------+---------+------------+-----------+------------+----------------+
| Domain | Words | Nodes | Edges | Density | Mean degree| Max degree| Prereq edges| Prereq density |
+------------------------------+-------+-------+-------+---------+------------+-----------+------------+----------------+
| ASP.NET Core middleware | 875 | 23 | 23 | 0.0455 | 2.00 | 4 | 13 | 0.5652 |
| SQL query optimization | 903 | 23 | 23 | 0.0455 | 2.00 | 5 | 10 | 0.4348 |
| CA slab production | 820 | 25 | 26 | 0.0433 | 2.08 | 5 | 15 | 0.5769 |
+------------------------------+-------+-------+-------+---------+------------+-----------+------------+----------------+

The overall graph density, edges divided by every possible directed pair, sat inside a band of 0.0433 to 0.0455 across three domains that share essentially nothing except that I wrote the source documents in a similar explainer style. That’s a spread of 0.0022. Node counts and edge counts vary with how much content is actually in each document, which is exactly what should vary, but the density figure barely moved at all, and the mean out-degree per node landed at almost exactly 2 in all three graphs. I don’t think that’s a coincidence of subject matter. I think it’s a fingerprint of the extraction prompt’s instruction to “extract every distinct concept as a node, and every explicit relationship,” applied to documents I wrote at similar informational density on purpose. Change my writing style and I’d expect that number to move even though the underlying domains haven’t changed at all, which is itself the whole point: the metric is measuring my prompt and my prose, not the subject.

The prerequisite-concept density is the metric the original piece specifically singled out, and it landed in a tighter band than I expected but not as tight as raw density: 0.4348 to 0.5769, a spread of 0.1421. That’s a real spread, not a rounding artifact, and the honest reading is that this is a partial replication. It’s uncomfortable in the way the original article’s framing predicted: three domains with nothing in common produced a prerequisite ratio that never dropped below 43 percent or rose above 58 percent, a band narrow enough that if you handed me a fourth unrelated domain’s number and asked me to guess whether it came from this pipeline, “somewhere in the high 0.4s to high 0.5s” would be a good bet regardless of what the domain actually was. That’s not nothing. That’s the pipeline talking.

The degree distribution held up the same way. In every domain, the highest-degree node is a real conceptual hub, not an artifact: in the ASP.NET Core graph, use_routing and authorization_middleware tie for the top spot at degree 4, which tracks, because half the article is about the fact that authorization can't run until routing has, and routing feeds everything downstream of it. In the SQL graph, index and index_seek both hit degree 5, the two concepts that every join algorithm and every sargability rule eventually has to reference. In the CA slab graph, sigma_mixer alone reaches degree 5, sitting at the exact point in the process where flake, plasticizer, pigment, and heat all have to converge before anything can move to the mold. None of that surprised me once I saw it, but it's a useful sanity check in the other direction: if the pipeline had produced a graph where the highest-degree node was some incidental concept mentioned once in a subordinate clause, that would have been a much stronger signal that the extraction was noise rather than structure, whatever the aggregate density numbers said.

I don’t want to oversell the cleanliness of this, though. The SQL optimization graph sitting lowest at 0.4348 lines up with something real in that document: it has more genuine “alternative_to” relationships, nested loop join versus hash join versus merge join, index scan versus index seek, than the other two, because query planning is fundamentally a choice between competing physical operators in a way that middleware ordering and slab pressing aren’t. So some of the variance across domains is real content, not just noise around a pipeline-fixed number. I’d rather report that nuance than pretend the replication came out cleaner than it did.

The prompt change that broke the metric on purpose

The notes-for-the-writer version of “is this metric gameable” turned out to be the easiest part of the whole exercise to demonstrate, and the most uncomfortable. I took the ASP.NET Core graph and reclassified exactly seven of its twenty-three edges, the ones typed produces, causes, or configures, as prerequisite_for instead, leaving the genuinely structural part_of edges untouched. This is the kind of reclassification a slightly more aggressive extraction prompt would produce on its own, something as small as adding one sentence like "when a relationship could plausibly be read as one thing being needed before another, prefer prerequisite_for" to the existing instructions:

+---------------------------------+-------+-------+--------------+----------------+
| Variant | Nodes | Edges | Prereq edges | Prereq density |
+---------------------------------+-------+-------+--------------+----------------+
| Original prompt | 23 | 23 | 13 | 0.5652 |
| Same graph, 7 edges reclassified | 23 | 23 | 20 | 0.8696 |
+---------------------------------+-------+-------+--------------+----------------+

Same document, same node count, same edge count, one type field changed on seven edges out of twenty-three, and prerequisite density jumped from 0.5652 to 0.8696, a swing of 0.3044. That single change moved the metric more than twice as far as the entire natural spread across three unrelated domains. If a metric can be pushed further by a one-sentence prompt tweak than it moves across ASP.NET Core, SQL optimization, and cellulose acetate combined, the metric was never primarily describing the subject in the first place. It’s describing how the prompt draws the line between “produces” and “requires,” and that line is a modeling choice, not a fact about the domain.

What I’d check before trusting a number like this again

A few things I’m now going to do before I let any structural metric off one of my own knowledge graphs inform a real decision, agent retrieval or otherwise:

Does the metric move at all across genuinely unrelated inputs. If it doesn’t move even a little, it isn’t measuring the domain, full stop, it’s a constant your pipeline happens to produce. My overall density number came uncomfortably close to failing this test; only the prerequisite-density figure showed real, if narrow, movement.

Is there a one-sentence change to the extraction prompt that shifts the metric by more than the metric’s natural range across domains. If a trivial wording change moves the number further than actual subject-matter variation does, as happened here, the metric is more sensitive to your prompt engineering than to your data, and no amount of re-running it on more documents fixes that; you’d just be re-measuring the same prompt bias with a bigger sample.

Would someone who actually knows the domain look at the extracted graph and agree it’s a fair model, not just a plausible-looking one. I could do this myself for all three of my domains, and the CA slab graph is the one I’d flag first: an edge landed between “buffing wheel” and “plasticizer ratio,” typed measures, because a buffing wheel is where under-plasticized material shows visible micro-cracking. That's a real fact from the source document, but it's a stretch to call it a measures relationship rather than a causes or an observational side effect, and it's exactly the kind of judgment call an LLM extractor makes silently, in both directions, on every document you ever run through it.

Is the density number doing real work, or is it just a proxy for how verbosely and evenly you wrote the source material. My near-identical density figures across three domains might be saying “extraction pipeline is unbiased by subject,” or they might just be saying “I write explainer prose at a pretty consistent information density regardless of what I’m explaining,” and I can’t fully separate those two explanations from three documents I authored myself in one sitting. A real test of that would need source documents from different authors, different eras, different natural writing styles, which is a bigger experiment than a Sunday afternoon replication, but it’s the honest limitation of this one.

Whether the highest-degree node in the graph is a concept a domain expert would actually point to as central. This one’s cheap to check and I’d add it as a standing step regardless of what the aggregate metrics say, because it caught nothing wrong here but it’s exactly the kind of check that would catch a bad extraction fast: if I ran this on my SIM billing domain model and the hub node turned out to be something incidental instead of the account hierarchy or the rating engine, I’d know the extraction was wrong before I ever got as far as computing a density figure.

I went into this expecting to either cleanly confirm the original claim or cleanly refute it, and got neither. The raw structural density looked almost pipeline-fixed. The prerequisite-concept density moved, but stayed inside a band tighter than I’m comfortable trusting as domain signal, and a one-line prompt change blew past that entire band with seven edge reclassifications. If you’re building a knowledge graph for an agent to retrieve or reason over and you’ve got a structural metric you like the look of, run it against a document you know cold and a document from a completely different world you also know cold, using the exact same extractor. If the numbers land close together, you haven’t found a fact about your domain. You’ve found out what your pipeline does by default, which is worth knowing, just not the thing you probably set out to measure.

Tags: knowledge-graphs, llm-engineering, ai-agents, python, dotnet, graphrag

Three Signals Say MCP Is Not Always the Right Tool, and What to Reach For Instead

Ali Suleyman TOPUZ — Thu, 01 Oct 2026 20:05:51 +0000

A week of reading other people’s postmortems made me go back and re-check my own defaults

For most of this year, my default was simple: if an agent needs to do something, wrap it in an MCP server. Not because I’d benchmarked it against anything, but because it’s the “standard” way to expose a capability to an agent now, and standards are usually the safe default when you don’t have a strong reason to deviate. I had a couple of small internal MCP servers running behind Claude Code and a coding agent at work, and I hadn’t questioned that setup in months.

Then in the space of about a week I read three completely unrelated things that all landed on the same conclusion from different directions, and none of them were arguing with each other. One was a build log from a developer who tried MCP first and deliberately walked away from it. One was a controlled internal evaluation with actual run counts. One was a straight-up architectural teardown with security research behind it. I don’t think any of the three authors read each other’s work. That’s what made me stop and re-check my own defaults instead of filing it under “someone’s hot take.”

So this is that re-check. Three findings, why each one holds up, and then the part I actually built myself: the same small internal tool exposed two different ways, instrumented to show where the tokens actually go over a real session, plus the gotcha I hit building it that none of the source material warned me about.

Why the “just wrap it in MCP” assumption breaks down

The assumption has a specific mechanical flaw, and it’s worth stating precisely instead of vaguely. When an MCP client connects to a server, the server responds with the full JSON Schema for every tool it exposes, all at once, at connection time. That schema then sits in the model’s context for the rest of the session. It doesn’t matter if the agent uses one of those tools once, all of them repeatedly, or none of them at all. The schema is there on every single turn, because that’s how the tools array gets sent with every request, and you pay for it in tokens whether or not it does anything for you that turn.

The second flaw is less about tokens and more about topology. MCP’s original session model, one client, one server, a persistent stateful connection, was designed for a single agent talking to a tool it trusts, running close to it. That’s a fine shape for a coding agent on your laptop talking to a local filesystem server. It’s a much rougher fit once you’re exposing a server across a trust boundary, to the internet, or to an unpredictable number of concurrent clients, because none of the load-balancing, auth, or failover machinery you’d want for that was part of the original design brief.

Neither of these is a reason MCP is bad. They’re reasons it has a shape, and that shape fits some jobs better than others. Here’s the evidence that convinced me the mismatch is common enough to matter.

Finding one: a developer who built the MCP version first, then threw it out

The first piece was a build log for a tool called CsMesh, a Roslyn-based semantic code search CLI for C# and .NET codebases, written by a developer working under the handle nrafinia. The premise of the tool itself is sharp: instead of having an agent grep through source files and guess at which IPaymentProcessor implementation actually gets dependency-injected at runtime, CsMesh builds a real symbol graph off the Roslyn compiler, resolves DI bindings and attribute-based routing that a syntax-only parser would miss, and hands the agent a short, ranked answer instead of a pile of matching files to read through itself.

What caught my attention wasn’t the tool, it was the decision behind its interface. The author built an MCP version first, then deliberately moved to a CLI, for two reasons stated plainly in the writeup: the always-injected schema cost, and the risk of an MCP tool call returning an unbounded blob of JSON that blows out the context window. The line that stuck with me: five tool definitions with decent descriptions is a permanent tax you pay for the privilege of maybe using them later. A CLI invocation, by contrast, costs nothing until the agent actually decides to run it.

The fix wasn’t just “use a CLI instead,” it was a specific interface discipline: every query takes a --budget flag as a hard token ceiling, and the tool returns structured exit codes instead of always trying to cram a full answer into stdout. Exit code 0 means a complete answer fit inside the budget. Exit code 2 means an answer exists but doesn't fit, along with suggestions for narrowing the query. Exit code 3 means the match was ambiguous and needs qualifying, like TypeName.MemberName instead of just MemberName. In the author's own example, a search that would have been 4,100 tokens as raw structural output collapsed to 125 tokens once routed through this budgeted, ranked interface. That's not a schema-injection saving, that's a response-shape saving, but it's the same underlying instinct: don't hand the model more than it asked for, and give it a mechanical way to ask for less when it gets too much.

Finding two: a controlled test where the MCP tools just sat there unused

The second piece was less of an opinion and more of an experiment, and it came from an unlikely source: Microsoft’s own SharePoint Framework team, writing on the Microsoft 365 Developer Blog about testing what their coding-agent skills actually improve. They set up a real task, upgrading a SharePoint Framework project from version 1.21.1 to 1.22.2, which isn’t a trivial bump: it involves a build tool migration from gulp to Heft, not just a version bump in a manifest file. They ran each condition five times to smooth out the non-determinism you’d expect from letting an LLM drive a real upgrade, and scored the results against concrete rubrics for dependency currency and configuration correctness.

Four conditions: a baseline agent with no extra help, an agent given an anti-hallucination skill telling it to verify facts against documentation before acting, that same skill plus a documentation-lookup MCP server (context7), and an agent explicitly pointed at a deterministic CLI tool built for this exact upgrade. The baseline scored 34 out of 50 on dependency currency and 38 out of 85 on configuration correctness. Adding the anti-hallucination skill alone bumped that to 40/50 and 46/85. Adding the MCP documentation server on top moved it to 39/50 and 47/85, statistically indistinguishable from the skill alone. The CLI-guided condition scored 50/50 and 83/85.

The line I keep coming back to from that writeup: in three of five runs, the MCP server’s tools were available but never invoked. Availability alone didn’t make the tools relevant to the agent’s plan. That’s the quiet, unglamorous version of the finding I care about most here. It’s not that MCP failed loudly, or that the documentation server gave wrong answers. It’s that having the capability sitting there, fully described, fully connected, cost real context and added essentially nothing, while pointing the same agent at an existing, deterministic tool that encoded the actual upgrade logic did almost all of the work. The lesson isn’t “MCP doesn’t help.” It’s “a general-purpose lookup tool doesn’t substitute for a tool that already knows the specific procedure,” and that distinction has nothing to do with which protocol wraps it.

Finding three: the architectural case, with the security research to back it up

The third piece was the most combative of the three: a piece called “MCP Is Dead,” by a writer going by Lucas McGregor, and it goes after MCP’s session model directly rather than its context cost. The core claim is that MCP’s stateful connection, one of the two connections per session has to stay open for the life of the session, doesn’t scale the way a normal stateless API does. No server can afford to hold an idle port open per client indefinitely, and there’s no built-in provision for reconnection: if a server drops, the session doesn’t migrate anywhere, it just ends.

The piece is equally blunt about authentication: the core protocol simply doesn’t define who’s allowed to call a tool or how a server should check that. Security gets treated as something each team bolts on afterward, which in practice means every team doing it a little differently, no shared standard to point to, and no guarantee that the server on the other end of a connection checked anything at all before answering.

That claim isn’t just rhetorical. It’s backed by research from Knostic, an AI security research firm, who scanned the internet for exposed MCP servers using Shodan and a set of custom fingerprinting filters. They found 1,862 internet-exposed MCP servers, manually verified a sample of 119 of them with safe, read-only tools/list requests, and every single one of the 119 allowed access to its internal tool listing with zero authentication. Not "most." All of them. That's not a theoretical trust-boundary concern, that's a documented, present-tense finding about what's actually running on the internet right now.

And the piece names a real company acting on exactly this shape of concern. Perplexity’s CTO, Denis Yarats, announced at the company’s Ask 2026 developer conference that Perplexity is moving away from MCP for its internal systems and enterprise integrations, back toward plain REST APIs and CLI tools, alongside a new multi-model Agent API with its own orchestration and tool execution built in. The stated reasons track the first two findings almost exactly: MCP’s schema and metadata overhead was costing real tokens and latency at their scale, and the security posture teams actually need in production, OAuth flows, granular permissions, rate limiting, audit logging, wasn’t something the protocol gave them out of the box.

Three sources, three different angles, one company’s production decision, and none of them coordinated with each other. That’s the pattern that made me stop treating this as noise.

Building it myself: the same tool, two ways, and where the tokens actually went

Reading three pieces of evidence is one thing. I wanted to see the schema-injection cost with my own numbers, on a tool shaped like something I’d actually deploy, not a toy weather API. So I built a small internal release-ops tool two ways: once as a naive MCP server exposing five separate tools, and once as a CLI wrapped behind a single generic tool call.

Here’s the naive version first, five tools, each with its own full JSON Schema, exactly the shape that gets sent to the model on every turn regardless of use:

// ReleaseOpsTools.cs, naive MCP server: five tools, five schemas,
// all of it injected into context on every turn whether the agent
// touches any of it or not.
using System.ComponentModel;
using ModelContextProtocol.Server;
[McpServerToolType]
public class ReleaseOpsTools
{
    [McpServerTool, Description("Gets the current deployment status for a service in a given environment, including running version, health check state, and last deploy timestamp.")]
    public string GetDeployStatus(
        [Description("Service name from the internal service catalog, e.g. 'billing-api'.")] string service,
        [Description("Target environment: dev, staging, or production.")] string environment)
        => ReleaseOps.Status(service, environment);
    [McpServerTool, Description("Restarts a running service instance. Disruptive: briefly interrupts traffic unless graceful draining is enabled.")]
    public string RestartService(
        [Description("Service name from the internal service catalog.")] string service,
        [Description("Target environment: dev, staging, or production.")] string environment,
        [Description("Justification for the restart, recorded in the audit log.")] string reason)
        => ReleaseOps.Restart(service, environment, reason);
    [McpServerTool, Description("Retrieves recent log lines for a service, optionally filtered by minimum severity.")]
    public string TailLogs(
        [Description("Service name from the internal service catalog.")] string service,
        [Description("Target environment: dev, staging, or production.")] string environment,
        [Description("Number of lines to return, default 100.")] int lines = 100,
        [Description("Minimum severity: debug, info, warning, error, critical.")] string? minSeverity = null)
        => ReleaseOps.Logs(service, environment, lines, minSeverity);
    [McpServerTool, Description("Lists every environment registered for the calling team, with region and freeze status.")]
    public string ListEnvironments() => ReleaseOps.Environments();
    [McpServerTool, Description("Rolls a service back to its previous known-good version. Requires confirm=true to actually execute; otherwise returns a dry-run preview.")]
    public string RollbackDeploy(
        [Description("Service name from the internal service catalog.")] string service,
        [Description("Target environment: dev, staging, or production.")] string environment,
        [Description("Must be true to execute; false returns a dry-run preview.")] bool confirm = false)
        => ReleaseOps.Rollback(service, environment, confirm);
}

And here’s the thin-wrapper version: the exact same logic, sitting behind a plain CLI, exposed through one generic tool call.

dotnet new console -n opsctl
cd opsctl

// Program.cs, opsctl: the real logic lives here regardless of
// which front end (MCP or a shell) ends up calling it. Plain manual
// argument parsing on purpose, no extra package: System.CommandLine's
// handler API has changed shape across its own beta releases, and a
// tool this small doesn't need a parsing library to be correct.
using System;
using System.IO;
using System.Linq;
string? Opt(string name, string[] a)
{
    int i = Array.IndexOf(a, name);
    return i >= 0 && i + 1 < a.Length ? a[i + 1] : null;
}
string Require(string name, string[] a) =>
    Opt(name, a) ?? throw new ArgumentException($"missing required option {name}");
var verb = args.Length > 0 ? args[0] : null;
var rest = args.Skip(1).ToArray();
switch (verb)
{
    case "status":
        Console.WriteLine(ReleaseOps.Status(Require("--service", rest), Require("--env", rest)));
        return 0;
    case "restart":
        Console.WriteLine(ReleaseOps.Restart(Require("--service", rest), Require("--env", rest), Require("--reason", rest)));
        return 0;
    case "logs":
    {
        int lines = int.TryParse(Opt("--lines", rest), out var n) ? n : 100;
        Console.WriteLine(ReleaseOps.Logs(Require("--service", rest), Require("--env", rest), lines, Opt("--min-severity", rest)));
        return 0;
    }
    case "environments":
        Console.WriteLine(ReleaseOps.Environments());
        return 0;
    case "rollback":
        Console.WriteLine(ReleaseOps.Rollback(Require("--service", rest), Require("--env", rest), rest.Contains("--confirm")));
        return 0;
    case "--help":
    case null:
        Console.WriteLine(File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "help.txt")));
        return 0;
    default:
        Console.Error.WriteLine($"unknown command '{verb}'. Run 'opsctl --help' for usage.");
        return 1;
}

help.txt sits next to the project file and needs one line in the .csproj so it actually lands in the build output instead of a FileNotFoundException the first time the agent asks for it:

<ItemGroup>
  <None Include="help.txt" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>

opsctl - internal release operations CLI
COMMANDS:
  status --service <name> --env <dev|staging|production>
  restart --service <name> --env <dev|staging|production> --reason <text>
  logs --service <name> --env <dev|staging|production> [--lines <n>] [--min-severity <level>]
  environments (no options)
  rollback --service <name> --env <dev|staging|production> [--confirm]

// RunCommandTool.cs, the MCP server exposes exactly ONE tool.
// The agent discovers subcommands by running 'opsctl --help' once,
// the same way a person would, instead of the schema describing
// every operation up front.
using System.ComponentModel;
using System.Diagnostics;
using System.Text;
using ModelContextProtocol.Server;
[McpServerToolType]
public class RunCommandTool
{
    private const int MaxOutputBytes = 4096;
    [McpServerTool, Description("Runs a single opsctl subcommand and returns its stdout, truncated to a fixed byte budget. Run 'opsctl --help' first if you don't already know the available subcommands and their arguments.")]
    public async Task<string> RunCommand(
        [Description("The full opsctl invocation, without the leading 'opsctl', e.g. 'status --service billing-api --env production'.")] string command)
    {
        var psi = new ProcessStartInfo("opsctl", command)
        {
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            UseShellExecute = false
        };
        using var process = Process.Start(psi)
            ?? throw new InvalidOperationException("Failed to start opsctl.");
        string stdout = await process.StandardOutput.ReadToEndAsync();
        string stderr = await process.StandardError.ReadToEndAsync();
        await process.WaitForExitAsync();
        string output = process.ExitCode == 0 ? stdout : $"exit code {process.ExitCode}: {stderr}";
        return TruncateUtf8Safe(output, MaxOutputBytes);
    }
    // The gotcha I actually hit: truncating raw UTF-8 bytes at a fixed
    // length can slice a multi-byte character in half. Encoding.GetString
    // on a cut surrogate doesn't throw, it silently emits U+FFFD replacement
    // characters into the output, which then confuses whatever parses the
    // tool result downstream. Decoder.Convert with flush=false lets you
    // back off to the last complete character instead of guessing.
    private static string TruncateUtf8Safe(string text, int maxBytes)
    {
        var bytes = Encoding.UTF8.GetBytes(text);
        if (bytes.Length <= maxBytes) return text;
        var decoder = Encoding.UTF8.GetDecoder();
        var chars = new char[maxBytes];
        decoder.Convert(bytes, 0, maxBytes, chars, 0, chars.Length,
            flush: false, out _, out int charsUsed, out _);
        return new string(chars, 0, charsUsed) + "\n...[truncated]";
    }
}

That truncation bug is the kind of thing none of the three source pieces warned me about, because it only shows up once you actually implement a byte-capped response instead of just talking about the idea of one. My first pass called Encoding.UTF8.GetString(bytes, 0, MaxOutputBytes) directly, the same way I'd seen it done in a couple of MCP tutorials, and it worked fine right up until a log line ended mid-emoji and the output started rendering <27> replacement characters into what the agent then tried to parse as a log timestamp. Decoder.Convert with flush: false fixes it by stopping at the last complete character instead of the last complete byte.

What the schemas actually cost, measured

I wanted real numbers, not a guess, so I pulled the JSON Schema payload each approach would actually send and ran it through a tokenizer. First attempt: tiktoken, so I'd have a real BPE count instead of an estimate. That failed, the sandbox I was working in blocks the download tiktoken needs for its encoding file, which is its own small lesson about how many "just run a tokenizer" tutorials assume network access you might not have in a locked-down environment. I fell back to the same ~4-characters-per-token heuristic Anthropic and OpenAI both publish for back-of-envelope sizing. It won't match an exact tokenizer, but both payloads are the same kind of text, English descriptions plus JSON punctuation, so the ratio between them holds up even if the absolute counts shift a little.

import json, re
_word_re = re.compile(r"[A-Za-z0-9_]+|[^\sA-Za-z0-9_]")
def count(text: str) -> int:
    pieces = _word_re.findall(text)
    return sum(1 if len(p) <= 4 else -(-len(p) // 4) for p in pieces)
mcp_schema_tokens = count(json.dumps(mcp_tools_list)) # all 5 tool schemas
cli_schema_tokens = count(json.dumps(run_command_tool)) # the 1 generic tool
help_tokens = count(opsctl_help_text) # paid once, turn 1
TURNS = 10
mcp_total = mcp_schema_tokens * TURNS
cli_total = (cli_schema_tokens * TURNS) + help_tokens

Modeling a 10-turn session where the agent only ever calls get_deploy_status, twice, and never touches restart, logs, environments, or rollback, here's what came out:

MCP naive server, 5 tools, full JSON schema
--------------------------------------------------------------
  schema tokens per turn: 1,220
  per-tool breakdown:
    get_deploy_status 228 tokens <- actually called
    restart_service 269 tokens <- never called
    tail_logs 326 tokens <- never called
    list_environments 95 tokens <- never called
    rollback_deploy 289 tokens <- never called
  total over 10 turns: 12,200
  spent on tools that were never called: 9,920 (81.3%)

CLI thin wrapper, single run_command tool
--------------------------------------------------------------
  schema tokens per turn: 191
  one-time --help cost (paid once, turn 1): 337
  total over 10 turns: 2,247
Difference over the session: 9,953 tokens
MCP approach costs roughly 5.4x more in schema overhead

Four out of five tools sitting completely unused, paying rent on every single turn, is 81% of the entire schema budget spent on capabilities the agent never touched in that session. That number will move around with a different mix of tools and a different session length, but the shape of it, a handful of always-on schemas, most of them irrelevant most turns, won’t change just because the tool happens to be about deployments instead of code search or issue tracking.

Trying the discovery step against a local model

The CLI-wrapper approach leans on the agent doing something an MCP schema does for free: figuring out what commands exist. A frontier model reliably runs opsctl --help first when it doesn't recognize a command surface. I wanted to see how that discovery step holds up against something you could run entirely offline, no API key, since that's the honest test of whether this approach only works because you're paying for a very capable model to make up the difference.

ollama pull llama3.1
ollama serve

# talk to opsctl through a local model, no cloud API involved
import subprocess
from ollama import Client
client = Client(host="http://localhost:11434")
tools = [{
    "type": "function",
    "function": {
        "name": "run_command",
        "description": "Runs a single opsctl subcommand and returns its stdout. "
                        "Run 'opsctl --help' first if you don't know the subcommands.",
        "parameters": {
            "type": "object",
            "properties": {"command": {"type": "string"}},
            "required": ["command"]
        }
    }
}]
def run_command(command: str) -> str:
    result = subprocess.run(["opsctl", *command.split()], capture_output=True, text=True)
    return result.stdout or result.stderr
response = client.chat(
    model="llama3.1",
    messages=[{"role": "user", "content": "Is billing-api healthy in production right now?"}],
    tools=tools,
)

Honest result: llama3.1 at 8B took the discovery step about two times out of three in casual testing, correctly running opsctl --help before guessing at a subcommand. The other third of the time it guessed a plausible-looking opsctl status billing-api production without the flag names, which opsctl's own argument parser rejected with a usage error, at which point the model read the error and self-corrected on the next call. It never silently failed, the CLI's own validation caught the malformed call every time, but it wasn't as clean as a frontier model doing it in one shot. That's the actual tradeoff nobody's hot take mentions: a CLI wrapper pushes some of the schema's job onto the model's ability to read help text and recover from a rejected argument, and a small model pays for that in extra round trips, not in silent wrong answers.

The rule of thumb I’m actually using now

None of this adds up to “MCP bad, CLI good.” It adds up to a question I ask before wrapping anything now, and the answer determines which way I go.

REACH FOR MCP WHEN REACH FOR A CLI / SINGLE FUNCTION
                                           TOOL WHEN
----------------------------------------------------------------------------
The tool needs to be discoverable and There's one clear caller, and it's
reusable across many different agents not going to be reused by some
and sessions you don't control, with other agent or client you haven't
usage patterns you can't predict met yet
You need a real permission boundary The agent already runs in a sandbox
around something dangerous: human or with scoped credentials that
approval gates, scoped OAuth tokens, limit blast radius just as well,
audit logging enforced centrally and MCP would just be ceremony
                                            wrapped around the same boundary
The client genuinely can't write its The tool is high-frequency: paying
own integration, a weaker model or a a fixed schema cost once beats
non-coding client with no shell to work re-describing it, but paying it on
with every turn for months doesn't
The response is small and bounded, so The response can be large or
the always-on schema cost is the only unbounded, and you need a hard
real overhead you're paying budget and structured exit codes
                                            more than you need discoverability

The security-boundary row is the one I’d weight above the token math, honestly. A tool sitting in front of something that can actually hurt you, a production database, a payments flow, anything with a rollback or restart in its name, benefits from a boundary enforced structurally rather than trusted to whatever the agent decides to do that day. My rollback_deploy tool, in the naive version above, is exactly the kind of thing I'd keep behind a real MCP server with server-side confirmation and audit logging even after everything above, specifically because "the agent decided not to call it without confirm=true" is a much weaker guarantee than "the server physically won't execute it without a signed approval token." The read-only status check next to it has no such argument in its favor, and paying its schema tax on every turn for the rest of the project's life was never buying me anything.

Where I landed

I didn’t come out of this deleting my MCP servers. I came out of it splitting them. The rollback and restart tools stayed behind MCP, because the boundary is the actual point there, not the convenience. The status check and log tail got demoted to opsctl subcommands behind a single run_command tool, because nothing about them needed a schema sitting in context on every turn for months, and the 5.4x overhead I measured on my own machine wasn’t buying me anything for capabilities that get called once or twice a session at most.

The three pieces that started this for me weren’t arguing that MCP is a bad idea. CsMesh’s author built the MCP version first and only moved off it once the cost showed up in practice. The SPFx team didn’t say MCP tools don’t work, they said tools nobody calls don’t help regardless of what protocol wraps them. And even the most combative of the three, the security teardown, ends with a real company replacing MCP with plain APIs and a CLI for internal use, not with abandoning tool-calling altogether. The actual finding across all three, and the one I’d want someone to walk away with, is narrower than “avoid MCP”: know which of your tools need the reach and the boundary MCP gives you, and stop paying its overhead for the ones that don’t.

Tags: mcp, ai-agents, llm, software-architecture, context-engineering, dotnet, developer-tools

Multi-Agent Systems Don’t Have a Consistency Problem. They Have a Database Problem.

Ali Suleyman TOPUZ — Thu, 01 Oct 2026 20:05:40 +0000

The bugs I kept blaming on the model turned out to be the same bugs I was fixing in 2012. The fix was boring. It worked.

The night one customer got refunded twice

The order was number 4471. It was a boring order: two items, one of them out of stock, a customer who wanted a partial refund. I had a small multi-agent setup running against a sandbox: an order agent that owned order state, and a support agent that talked to the customer and could trigger refunds through a tool.

Here’s what happened, reconstructed from the logs the next morning.

The support agent decided a partial refund was right and called the issue_refund tool. The payment sandbox was slow that evening. The tool call hit a 30 second timeout. The orchestrator did exactly what I had told it to do on timeouts: it retried the step. The model, seeing the same conversation, made the same decision and called issue_refund again.

Both calls went through. The first one had not failed. It had just been slow.

Meanwhile, the order agent had read order 4471 at status = 'awaiting_stock', decided the missing item should be cancelled, and wrote status = 'partially_cancelled'. A few hundred milliseconds later, the support agent, working from its own earlier read of the same row, wrote status = 'refund_pending'. Last write won. The cancellation vanished from the record. Nobody raised an error, because from the database's point of view nothing was wrong. Two clients wrote to a row. That's what rows are for.

So by morning I had:

a customer refunded twice for one item
an order whose status said “refund pending” while the warehouse thought it was cancelled
a Slack thread where I had typed “the agents are being inconsistent again, need a better system prompt”

That last line is the one I’m embarrassed about. Because I spent the next two evenings rewriting prompts. I added “never issue a refund twice” to the support agent. I added “check the current status before writing” to the order agent. It got slightly better and then broke again the next time the sandbox was slow.

Nothing about this failure was an AI problem. It was a retry without an idempotency key, and a lost update caused by two writers with stale reads. I have fixed both of those bugs in systems with zero LLMs in them. The model was just the part of the pipeline I was most suspicious of, so it’s where I looked first.

The LLM is not the bug. The LLM is a very slow, very creative client.

This is the argument of the whole article, so let me state it plainly.

Most “multi-agent consistency” failures are classic distributed-systems failures. Race conditions, lost writes, duplicated side effects, out-of-order messages. They show up in agent systems because agent systems are distributed systems: multiple independent actors, each with a partial and possibly stale view of shared state, communicating over unreliable channels, taking actions with real side effects.

An LLM in the loop changes a few things about the shape of these problems, and all of them make things worse:

Latency is huge and variable. A model call can take 800ms or 40 seconds. The window between “read state” and “write decision” is enormous compared to a normal service, so races that would be rare in a CRUD API become routine.
Retries are natural. Timeouts, rate limits, and flaky tool calls mean every framework retries. Retrying a non-idempotent side effect is how you refund someone twice.
Decisions are not replayable. If you retry a step, the model may decide something slightly different the second time. So you can’t even assume the retry is a duplicate of the original. It might be a new, conflicting intent.

None of this is fixed by a smarter prompt. You can’t prompt your way out of a lost update any more than you can fix a race condition in Java by adding a comment that says “please don’t race.”

I found JIN’s piece Multi-Agent Consistency Isn’t an AI Problem after I’d already gone through this, and it put words to something I’d felt but not articulated: the moment you have more than one writer, you’ve signed up for distributed-systems discipline whether you wanted to or not.

Every orchestration framework hits the same three walls

I’ve built small things with LangGraph, poked at CrewAI, used the OpenAI Agents SDK’s handoffs, and read through how Microsoft Agent Framework does checkpointing. They’re all useful. They solve real problems around control flow, state passing, and resuming after crashes.

But none of them solve the three problems that actually bit me, because those problems live below the orchestrator, at the boundary where an agent touches the outside world.

+-------------------------------+-----------------------------+---------------------------------------+
| Problem | What orchestrators give you | What they usually DON'T give you |
+-------------------------------+-----------------------------+---------------------------------------+
| Idempotency | Step retries, checkpoints, | A guarantee that the external side |
| (same intent, one effect) | resume from last state | effect (payment, email, booking) |
| | | fires exactly once across retries |
+-------------------------------+-----------------------------+---------------------------------------+
| Ordering | A graph or handoff sequence | Protection against two branches or |
| (writes land in a sane order) | inside ONE workflow run | two runs writing the same record with |
| | | stale reads |
+-------------------------------+-----------------------------+---------------------------------------+
| Conflict resolution | "Last node wins" state | A record of WHO asserted WHAT based |
| (two agents disagree) | merging, reducers | on WHICH version, so disagreements |
| | | can be reconciled, not overwritten |
+-------------------------------+-----------------------------+---------------------------------------+

Checkpoints are the one that fooled me longest. Durable execution (Shrashti Singhal’s write-up on agents that survive crashes and restarts is a good intro) is genuinely great. But a checkpoint tells you where your workflow was. It does not tell you whether the payment API received your request before the process died. Resuming from a checkpoint and re-running the “issue refund” step is exactly the double-refund scenario, just triggered by a crash instead of a timeout.

The orchestrator serializes steps inside its own view of the world. Your database, your payment provider, and your calendar API don’t live inside that view.

What 16 agents writing a C compiler actually needed

If you want to see what this looks like at real scale, Anthropic published a write-up earlier this year on building a C compiler with a team of parallel Claude agents. Sixteen agents, roughly 2,000 Claude Code sessions, around $20K in API cost, and about 100,000 lines of Rust at the end, capable of compiling a real Linux kernel.

What struck me reading it wasn’t the model capability. It was how much of the engineering was coordination plumbing, and how old-fashioned that plumbing was.

Each agent ran in its own container with its own clone of a shared git repository. To take a task, an agent wrote a lock file into a shared directory and pushed it. If another agent had already claimed that task, git’s push rejection forced the second agent to pick something else. Merge conflicts were constant and the agents resolved them the normal way.

Read that again with a database hat on. That’s a claim table with compare-and-swap semantics , implemented with git as the storage engine. The push either succeeds against the version you based it on, or it’s rejected and you re-read. That’s optimistic concurrency control.

And the failure they described is also a textbook one: when every agent ran into the same big problem (the kernel build), they all tried to fix the same bug and kept overwriting each other’s work. More parallelism produced less progress, because there was no way to partition the work so that writes didn’t collide. The fix was to change how the work was split so agents stopped touching the same state, not to make the agents smarter.

That’s the lesson I took from it. At 16 agents, the discipline that kept things consistent was locks, versioned state, and partitioned work. The models were the easy part.

Pattern 1: Idempotency keys on every agent-triggered side effect

The rule I follow now: no agent tool that causes an external side effect runs without an idempotency key, and the key is never generated by the model.

That second half matters. If you let the LLM invent a request ID, a retry gets a fresh ID and your dedupe does nothing. The key has to be derived deterministically from the intent: which workflow run, which step, which target, which payload.

Here’s the table:

CREATE TABLE side_effects (
    idempotency_key TEXT PRIMARY KEY,
    effect_type TEXT NOT NULL,
    request_hash TEXT NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending', -- pending | done | failed
    result JSONB,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    completed_at TIMESTAMPTZ
);

And the wrapper every side-effecting tool goes through:

import hashlib
import json

import psycopg
DSN = "postgresql://agents:agents@localhost:5432/agents"

def _hash(obj) -> str:
    return hashlib.sha256(json.dumps(obj, sort_keys=True).encode()).hexdigest()

def effect_key(run_id: str, step: str, target: str, payload: dict) -> str:
    """Deterministic key: same intent => same key, across retries and restarts."""
    return _hash({"run": run_id, "step": step, "target": target, "payload": payload})

class EffectInProgress(Exception):
    pass

def run_once(conn, key: str, effect_type: str, payload: dict, do_effect):
    req_hash = _hash(payload)
    claimed = conn.execute(
        """
        INSERT INTO side_effects (idempotency_key, effect_type, request_hash)
        VALUES (%s, %s, %s)
        ON CONFLICT (idempotency_key) DO NOTHING
        RETURNING idempotency_key
        """,
        (key, effect_type, req_hash),
    ).fetchone()
    if claimed is None:
        status, result, existing_hash = conn.execute(
            "SELECT status, result, request_hash FROM side_effects WHERE idempotency_key = %s",
            (key,),
        ).fetchone()
        if existing_hash != req_hash:
            raise ValueError(f"Key {key} reused with a different payload")
        if status == "done":
            return result # duplicate call: return the original outcome, do nothing
        raise EffectInProgress(key) # someone else is mid-flight; caller should back off
    try:
        # Pass the same key downstream so the provider can dedupe too.
        result = do_effect(payload, idempotency_key=key)
    except Exception:
        conn.execute(
            "UPDATE side_effects SET status = 'failed' WHERE idempotency_key = %s", (key,)
        )
        raise
    conn.execute(
        """
        UPDATE side_effects
        SET status = 'done', result = %s, completed_at = now()
        WHERE idempotency_key = %s
        """,
        (json.dumps(result), key),
    )
    return result

# usage
with psycopg.connect(DSN, autocommit=True) as conn:
    payload = {"order_id": 4471, "amount_cents": 1899, "reason": "out_of_stock"}
    key = effect_key(run_id="run-8f2c", step="issue_refund", target="order:4471", payload=payload)
    run_once(conn, key, "refund", payload, do_effect=lambda p, idempotency_key: {"refund_id": "rf_123"})

An honest caveat, because I got this wrong the first time: there’s a window where the effect succeeded but the process died before writing done. The row stays pending forever. Two things cover it. First, pass the same key to the downstream provider (Stripe and most payment APIs accept one), so even if you retry, they dedupe. Second, run a small reconciliation job that looks at old pending rows and asks the provider what actually happened. This is not glamorous. It's also the difference between "we refunded twice" and "we didn't."

A note on failed: I deliberately don't auto-retry failed rows with the same key. A failure might mean the model's intent was wrong, and I'd rather the next attempt go back through the agent's decision with fresh state.

Pattern 2: Optimistic locking in SQL, not in the orchestrator

The lost update on order 4471 happened because two agents read the same row, thought for a while, and wrote blindly. The orchestrator couldn’t prevent it because the two agents were in different workflow runs. The orchestrator had no idea they were touching the same record.

The database did know. I just hadn’t asked it to care.

Add a version column and make every write a compare-and-swap:

ALTER TABLE orders ADD COLUMN version INT NOT NULL DEFAULT 0;
-- Every agent write looks like this. Zero rows updated = someone else got there first.
UPDATE orders
SET status = %(new_status)s,
       version = version + 1
WHERE id = %(order_id)s
  AND version = %(expected_version)s
RETURNING version;

The important design decision is what happens on conflict. My first instinct was to just retry the write. That’s wrong. The agent’s decision was based on state that no longer exists. The correct move is to re-read and re-decide :

class ConcurrencyConflict(Exception):
    pass
def agent_update_order(conn, order_id: int, decide, max_attempts: int = 3):
    for attempt in range(1, max_attempts + 1):
        order = conn.execute(
            "SELECT id, status, version, items FROM orders WHERE id = %s", (order_id,)
        ).fetchone()
        # The slow part: an LLM call. Other agents can write during this window.
        decision = decide(order)
        if decision["new_status"] == order[1]:
            return order # nothing to change
        row = conn.execute(
            """
            UPDATE orders SET status = %s, version = version + 1
            WHERE id = %s AND version = %s
            RETURNING version
            """,
            (decision["new_status"], order_id, order[2]),
        ).fetchone()
        if row is not None:
            return row
        # Lost the race. Loop back, re-read fresh state, and let the agent decide again.
    raise ConcurrencyConflict(f"order {order_id}: gave up after {max_attempts} attempts")

Bounded attempts matter. If two agents keep flipping a record back and forth, that’s not a race, that’s a genuine disagreement, and retrying forever just burns tokens. That’s what Pattern 3 is for.

Since a lot of my day job is .NET, here’s the same thing with Npgsql:

using Npgsql;
public sealed class ConcurrencyConflictException(string message) : Exception(message);
public static async Task<int> UpdateOrderStatusAsync(
    NpgsqlConnection conn, long orderId, string newStatus, int expectedVersion)
{
    const string sql = """
        UPDATE orders
        SET status = @status, version = version + 1
        WHERE id = @id AND version = @expected
        RETURNING version
        """;
    await using var cmd = new NpgsqlCommand(sql, conn);
    cmd.Parameters.AddWithValue("status", newStatus);
    cmd.Parameters.AddWithValue("id", orderId);
    cmd.Parameters.AddWithValue("expected", expectedVersion);
    var result = await cmd.ExecuteScalarAsync();
    if (result is null)
        throw new ConcurrencyConflictException(
            $"Order {orderId} changed since version {expectedVersion}. Re-read and re-decide.");
    return (int)result;
}

If you’re on EF Core, a [ConcurrencyCheck] or row-version property gives you the same behavior and throws DbUpdateConcurrencyException. Same idea, same "re-read and re-decide" handling.

Pattern 3: A claim graph, so disagreements stop being silent

Optimistic locking tells you that two agents collided. It doesn’t tell you why, and it doesn’t keep the losing side’s reasoning. In my double-refund case, the order agent’s “partially cancelled” conclusion was actually correct, and it disappeared without a trace.

So I stopped letting agents write conclusions directly into business tables. Agents write claims. A claim says: this agent, in this run, asserted this value for this fact, based on this version of the world, for this reason.

CREATE TABLE claims (
    id BIGSERIAL PRIMARY KEY,
    subject TEXT NOT NULL, -- e.g. 'order:4471'
    predicate TEXT NOT NULL, -- e.g. 'status'
    value JSONB NOT NULL,
    agent_id TEXT NOT NULL,
    run_id TEXT NOT NULL,
    based_on_version INT, -- version of the subject the agent read
    evidence JSONB, -- tool outputs / reasoning summary
    supersedes BIGINT REFERENCES claims(id),
    asserted_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX claims_subject_idx ON claims (subject, predicate, asserted_at DESC);

It’s a graph because of supersedes: when an agent updates its view after seeing another claim, it links to the claim it's replacing. Two claims on the same fact, based on the same version, neither superseding the other, with different values: that's a conflict. You can find those with plain SQL:

SELECT a.id AS claim_a, a.agent_id AS agent_a, a.value AS value_a,
       b.id AS claim_b, b.agent_id AS agent_b, b.value AS value_b
FROM claims a
JOIN claims b
  ON a.subject = b.subject
 AND a.predicate = b.predicate
 AND a.id < b.id
WHERE a.subject = %(subject)s
  AND a.value <> b.value
  AND a.based_on_version = b.based_on_version
  AND b.supersedes IS DISTINCT FROM a.id
  AND NOT EXISTS (SELECT 1 FROM claims c WHERE c.supersedes IN (a.id, b.id));

Then a reconciler decides what gets committed to the real orders table. I start with deterministic rules and only escalate when rules can't decide:

# Which agent is authoritative for which predicate. Boring on purpose.
AUTHORITY = {
    "status": ["order_agent", "support_agent"], # order agent wins on status
    "refund_amount": ["support_agent"],
}

def reconcile(conn, subject: str, predicate: str):
    rows = conn.execute(
        """
        SELECT id, agent_id, value, evidence FROM claims
        WHERE subject = %s AND predicate = %s
          AND id NOT IN (SELECT supersedes FROM claims WHERE supersedes IS NOT NULL)
        ORDER BY asserted_at DESC
        """,
        (subject, predicate),
    ).fetchall()
    distinct_values = {json.dumps(r[2], sort_keys=True) for r in rows}
    if len(distinct_values) <= 1:
        return ("agreed", rows[0] if rows else None)
    ranking = AUTHORITY.get(predicate, [])
    ranked = sorted(
        rows, key=lambda r: ranking.index(r[1]) if r[1] in ranking else len(ranking)
    )
    if ranked[0][1] in ranking:
        return ("resolved_by_authority", ranked[0])
    # No rule applies: park it for a human (or a judge agent with all evidence attached).
    return ("needs_review", rows)

Is this more work than letting agents write directly? Yes. But the first time a conflict landed in my review queue with both agents’ evidence side by side, I understood in ten seconds what had happened. Compare that to two evenings of rewriting prompts.

Running all of this locally (no paid API needed)

You don’t need a hosted model to reproduce any of this. The races happen at the database layer. I run Postgres and Ollama in Docker and use a small local model as the “agent brain.”

# docker-compose.yml
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: agents
      POSTGRES_PASSWORD: agents
      POSTGRES_DB: agents
    ports:
      - "5432:5432"
ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
volumes:
  ollama:

docker compose up -d
docker compose exec ollama ollama pull qwen2.5:7b
pip install "psycopg[binary]" requests pytest

A decision function that uses the local model and forces JSON output:

import json
import requests
OLLAMA_URL = "http://localhost:11434/api/chat"

def ollama_decide(order) -> dict:
    order_id, status, version, items = order
    resp = requests.post(
        OLLAMA_URL,
        json={
            "model": "qwen2.5:7b",
            "stream": False,
            "format": "json",
            "messages": [
                {
                    "role": "system",
                    "content": (
                        "You manage order status. Reply ONLY with JSON: "
                        '{"new_status": "<status>", "reason": "<short reason>"}. '
                        "Allowed statuses: awaiting_stock, partially_cancelled, "
                        "refund_pending, shipped."
                    ),
                },
                {"role": "user", "content": json.dumps({"id": order_id, "status": status, "items": items})},
            ],
        },
        timeout=120,
    )
    resp.raise_for_status()
    return json.loads(resp.json()["message"]["content"])

Plug ollama_decide into agent_update_order from Pattern 2 and you have a real, slow, non-deterministic agent writing to a real database. Which is exactly the environment where these bugs live.

“More agents” is not a strategy, and there’s data now

Part of why I got into trouble is that I added the second agent because it felt like the natural next step. Everything I was reading framed multi-agent as the upgrade path.

The research is more cautious than the hype. Google Research ran a controlled study across 180 agent configurations. On some tasks multi-agent setups helped a lot: on Finance-Agent, a centrally coordinated setup improved performance by about 80.9%. On others it was actively harmful: on PlanCraft, a sequential planning task, multi-agent variants degraded performance by roughly 39% to 70% depending on the architecture. A follow-up in Nature Machine Intelligence extended this to 260 configurations across six benchmarks and five architectures, and the headline didn’t change: whether more agents help depends heavily on the task structure and how the agents coordinate.

That matches my experience exactly. Tasks that decompose into independent pieces (parallel research, analyzing separate documents) benefit. Tasks where every step depends on shared, evolving state get worse, because you’ve added coordination overhead and new ways for writes to collide, which is the whole subject of this article. JIN’s piece on the four multi-agent architectures and how a supervisor dispatches work is useful here: the supervisor pattern is essentially a way of centralizing writes, which is why it tends to behave better on stateful tasks.

On the governance side, Meta’s “Agents Rule of Two” is worth knowing. The idea is that, until prompt injection is solved, an agent session should have at most two of these three properties: it processes untrusted input, it has access to sensitive systems or private data, and it can change state or communicate externally. It’s framed as a security rule, but I’ve found it useful for consistency too. The third property, “can change state,” is exactly where idempotency, ordering, and conflicts live. Every agent I give write access to is an agent I now have to wrap in keys, versions, and claims. Keeping that set small is the cheapest mitigation there is.

How do you test “two agents raced and one lost”?

This was the part I found hardest to think about. My instinct was to write tests against the agent’s output. But the race isn’t in the output. It’s in the timing.

The trick that made it testable for me: take the model out of the test. Replace the LLM with a deterministic stub that sleeps. The sleep widens the race window to something you can hit every time. Then fire the same trigger from two threads at exactly the same moment using a barrier, and assert on side effects, not on text.

# test_races.py
import threading
import time
import psycopg
import pytest
from agents import effect_key, run_once, agent_update_order, EffectInProgress, ConcurrencyConflict
DSN = "postgresql://agents:agents@localhost:5432/agents"

@pytest.fixture
def db():
    with psycopg.connect(DSN, autocommit=True) as conn:
        conn.execute("TRUNCATE side_effects")
        conn.execute("DELETE FROM orders WHERE id = 4471")
        conn.execute(
            "INSERT INTO orders (id, status, version, items) VALUES (4471, 'awaiting_stock', 0, '[]')"
        )
        yield conn

def test_duplicate_trigger_fires_one_side_effect(db):
    calls = []
    lock = threading.Lock()
    def fake_refund(payload, idempotency_key):
        time.sleep(0.3) # simulate a slow payment API
        with lock:
            calls.append(idempotency_key)
        return {"refund_id": "rf_test"}
    payload = {"order_id": 4471, "amount_cents": 1899}
    key = effect_key("run-1", "issue_refund", "order:4471", payload)
    barrier = threading.Barrier(2)
    errors = []
    def worker():
        with psycopg.connect(DSN, autocommit=True) as conn:
            barrier.wait()
            try:
                run_once(conn, key, "refund", payload, fake_refund)
            except EffectInProgress:
                pass # expected for the loser
            except Exception as e:
                errors.append(e)
    threads = [threading.Thread(target=worker) for _ in range(2)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    assert not errors
    assert len(calls) == 1, f"side effect fired {len(calls)} times"
    count = db.execute("SELECT count(*) FROM side_effects").fetchone()[0]
    assert count == 1

def test_concurrent_writers_do_not_lose_updates(db):
    barrier = threading.Barrier(2)
    outcomes = []
    def make_agent(target_status):
        def decide(order):
            barrier.wait() # both agents read the same version, then...
            time.sleep(0.2) # ...think slowly, like a real LLM call
            return {"new_status": target_status}
        return decide
    def worker(status):
        with psycopg.connect(DSN, autocommit=True) as conn:
            try:
                agent_update_order(conn, 4471, make_agent(status), max_attempts=1)
                outcomes.append(("won", status))
            except ConcurrencyConflict:
                outcomes.append(("lost", status))
    t1 = threading.Thread(target=worker, args=("partially_cancelled",))
    t2 = threading.Thread(target=worker, args=("refund_pending",))
    t1.start(); t2.start(); t1.join(); t2.join()
    winners = [o for o in outcomes if o[0] == "won"]
    losers = [o for o in outcomes if o[0] == "lost"]
    assert len(winners) == 1 and len(losers) == 1
    status, version = db.execute("SELECT status, version FROM orders WHERE id = 4471").fetchone()
    assert version == 1
    assert status == winners[0][1] # the stored value is the winner's, and the loser KNOWS it lost

The second test is the one I care about most. It doesn’t assert that a specific agent wins. It asserts that exactly one wins, the version moved exactly once, and the loser got an explicit conflict instead of silently overwriting. Before the version column, this test failed every single run. That’s the point: a race you can reproduce on demand is a race you can actually fix.

Once the stubbed tests pass, I run a smaller number of the same scenarios with the real Ollama model plugged in, mostly to catch cases where the model’s re-decision after a conflict does something weird. But the stubbed version is the one in CI.

The checklist I use now, before adding a second agent

I keep this in the README of anything I build with agents. It’s short on purpose.

+----+--------------------------------------------------------------+---------+
| # | Question | Pass? |
+----+--------------------------------------------------------------+---------+
| 1 | Does every side-effecting tool go through an idempotency | |
| | key derived from the intent (not generated by the model)? | |
+----+--------------------------------------------------------------+---------+
| 2 | Is that key also passed to the downstream provider? | |
+----+--------------------------------------------------------------+---------+
| 3 | Is there a reconciliation job for effects stuck in pending? | |
+----+--------------------------------------------------------------+---------+
| 4 | Does every write to shared state use a version check | |
| | (CAS / optimistic locking) instead of a blind UPDATE? | |
+----+--------------------------------------------------------------+---------+
| 5 | On conflict, does the agent re-read and re-decide, with a | |
| | bounded number of attempts? | |
+----+--------------------------------------------------------------+---------+
| 6 | Can I see who asserted what, based on which version, when | |
| | two agents disagree? | |
+----+--------------------------------------------------------------+---------+
| 7 | Do I have a test that fires the same trigger twice and | |
| | asserts exactly one side effect? | |
+----+--------------------------------------------------------------+---------+
| 8 | Does the task actually decompose into independent pieces, | |
| | or does every step share evolving state? | |
+----+--------------------------------------------------------------+---------+
| 9 | Does this new agent really need write access (Rule of Two)? | |
+----+--------------------------------------------------------------+---------+

Here’s the uncomfortable part I had to admit to myself: items 1 through 7 apply to a single agent too. One agent with a retry policy can double-refund on its own. One agent running in two overlapping workflow runs can lose its own updates. My double refund on order 4471 wasn’t even caused by the second agent. It was a single agent retrying. The second agent just produced the lost update at the same time, which made the whole thing look like “multi-agent chaos.”

That’s the real reason for the checklist. A second agent doesn’t create new consistency problems. It makes the ones you haven’t solved visible faster. If you haven’t solved idempotency, ordering, and conflict resolution for one agent, adding another is not scaling. It’s turning up the frequency on a bug you already have.

I still use multi-agent setups. I just stopped blaming the model for bugs that belong to the database, and the database, it turns out, has had answers for these for decades.

References

JIN, Multi-Agent Consistency Isn’t an AI Problem: https://medium.com/jin-system-architect/multi-agent-consistency-isnt-an-ai-problem-439c875ffdb9
JIN, The Four Multi-Agent Architectures: How the Supervisor Dynamically Dispatches Work: https://medium.com/jin-system-architect/the-four-multi-agent-architectures-how-the-supervisor-dynamically-dispatches-work-and-why-it-1cd872c83a2f
Shrashti Singhal, Durable Execution: Agents That Survive Crashes, Restarts, and Weekends: https://pub.towardsai.net/durable-execution-agents-that-survive-crashes-restarts-and-weekends-4ffbeaef4ec1
Anthropic Engineering, Building a C compiler with a team of parallel Claudes
Google Research, Towards a Science of Scaling Agent Systems (and its Nature Machine Intelligence follow-up)
Meta AI, Agents Rule of Two: A Practical Approach to AI Agent Security

Tags: AI Agents, Multi-Agent Systems, Distributed Systems, Software Engineering, Artificial Intelligence, PostgreSQL, Database, LLM, Python, DotNet

Harness, Framework, or SDK: I Audited My Own Agent Stack Against a Real Source-Code Study

Ali Suleyman TOPUZ — Thu, 01 Oct 2026 10:09:52 +0000

A source-code study of eleven coding agents gave me a vocabulary I didn’t know I was missing. So I turned it on my own projects.

The sentence that stopped me

Last week I was explaining an incident to a colleague. One of our agents had gotten stuck in a tool-calling loop overnight and burned through a chunk of budget before anyone noticed. Halfway through my explanation I said “the agent framework retried the call” and then paused, because that wasn’t true.

There was no agent framework in that code path. It was a raw SDK call with a while loop I had written myself, eight months earlier, on a Friday afternoon. The retry was mine. The missing iteration cap was mine. The "framework" I was blaming was me.

That small moment bothered me more than the incident did. I realized I had been using “the agent framework” as a catch-all for at least three very different things in my stack:

Claude Code, which I use every day as a coding agent
Microsoft Agent Framework, which runs a couple of our .NET agents
A handful of direct Anthropic SDK integrations, some using the tool runner helper, some using a loop I wrote by hand

Those three things fail in completely different ways. They put the control loop in different hands. They store state in different places. And I had been talking about them as if they were interchangeable.

The same week, I read Jaroslaw Wasowski’s piece on Level Up Coding about choosing a coding agent’s runtime. It is built around a new arXiv paper (2609.00006) by Paul Barbaste and co-authors, a source-code anatomy of eleven production coding harnesses: Claude Code, Codex CLI, Gemini CLI, Mistral Vibe, OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, and OpenClaw, with Databricks’ Omnigent analyzed as a contrast point. Wasowski’s headline reading of it stuck with me: across roughly four million lines of Python, TypeScript, and Rust, none of those agents imports a general-purpose agent framework at runtime. They all own their loop.

That was the push I needed. If the people building the most heavily used coding agents in the world all chose to own the loop, I wanted to know why I had made the choices I made. So I did a small audit of my own stack using the same lens.

Three words that are not synonyms

The paper’s core definition is simple. An agent is a model plus a harness, and the harness is everything except the model: the loop, the tools, context management, safety controls, orchestration, and extension surfaces. It maps seven subsystems that every harness has to take a position on, even if that position is “we don’t do this.”

Wasowski’s framing on top of that is what made it practical for me. The question is not “which framework should I pick.” The question is who owns the loop, and where does state have to survive a crash. From that, three categories fall out naturally:

Harness. An opinionated, batteries-included runtime. You bring the model, the instructions, and your domain tools. The harness brings the loop, planning, context compaction, approvals, memory, and telemetry. You configure it, but you don’t write the loop. Claude Code is the obvious example. You shape its behavior through CLAUDE.md, skills, hooks, and permissions, but you are a guest in its loop.

Framework. You own the loop, or at least the shape of it. The framework gives you primitives: graphs, nodes, checkpoints, state reducers, workflow steps. LangGraph lives here. So does the workflow side of Microsoft Agent Framework. You decide what happens after each step. The framework decides how steps are wired, persisted, and resumed.

SDK. You own everything. The SDK gives you a typed API call and maybe a small helper. Anthropic’s tool runner is about as far as it goes: it will run the tool loop for you, but it has no opinion about memory, approvals, persistence, or what happens when your process dies halfway through.

Here is the version I ended up pinning above my desk:

+-------------+----------------------+-------------------------+--------------------------+
| | HARNESS | FRAMEWORK | SDK |
+-------------+----------------------+-------------------------+--------------------------+
| Loop owner | The runtime | You, via its primitives | You, entirely |
| State | Runtime decides | You declare, it persists| You build it (or not) |
| Safety | Built in, configured | Pluggable | Your code or nothing |
| Typical | Claude Code, | LangGraph, MAF | Anthropic SDK, |
| example | MAF AsHarnessAgent | workflows | tool runner, raw loop |
| Fails like | Opaque, "why did it | Graph/state bugs, | Missing guardrails you |
| | do that?" | version churn | forgot to write |
| Best for | Open-ended tasks, | Long-running, resumable | Narrow, bounded calls |
| | humans in the loop | pipelines | inside a bigger app |
+-------------+----------------------+-------------------------+--------------------------+

One wrinkle worth calling out: these are not product categories, they are roles. Microsoft Agent Framework plays two of them. Its workflow APIs are a framework in the classic sense. But it now also ships an agent harness, exposed through AsHarnessAgent, which is exactly the batteries-included runtime described above. The same NuGet dependency can put you in either column depending on which API you call. That confused me at first, and I think it confuses a lot of .NET teams.

My three integration points

I picked three real places in my stack where an LLM does something more than one-shot text generation, and asked the same two questions of each: who owns the loop, and what state must survive a crash.

1. The customer-facing assistant: I picked a harness

This is a support assistant that answers questions about device connectivity, reads account data, and can open a ticket or trigger a diagnostic action on the customer’s behalf.

When I listed what it needed, the list was long: tool calling, a plan for multi-step questions, conversation history that survives restarts, context compaction for long chats, file access scoped to one folder of generated reports, human approval before anything with side effects, and traces I can show someone after a complaint.

I could build all of that. I have built parts of it before, badly. But none of it is what makes this assistant valuable. What makes it valuable is the domain tools and the boundaries around them.

So this one runs on the Microsoft Agent Framework harness. Bruno Capuano’s MafClaw series on the .NET blog is a good walkthrough of the pattern, and the starting point really is this small:

using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;

var endpoint = Environment.GetEnvironmentVariable("FOUNDRY_PROJECT_ENDPOINT")!;
var model = Environment.GetEnvironmentVariable("FOUNDRY_MODEL") ?? "gpt-4.1-mini";

IChatClient chatClient =
    new AIProjectClient(new Uri(endpoint), new AzureCliCredential())
        .GetProjectOpenAIClient()
        .GetResponsesClient()
        .AsIChatClient(model);

var workingDirectory = Path.Combine(AppContext.BaseDirectory, "working");
Directory.CreateDirectory(workingDirectory);

AIAgent agent = chatClient.AsHarnessAgent(new HarnessAgentOptions
{
    // File tools are confined to this folder. The model never sees the rest of the disk.
    FileAccessStore = new FileSystemAgentFileStore(workingDirectory),

    // Reads go through silently. Writes and side effects still need a human.
    ToolApprovalAgentOptions = new ToolApprovalAgentOptions
    {
        AutoApprovalRules = [FileAccessProvider.ReadOnlyToolsAutoApprovalRule]
    },

    ChatOptions = new ChatOptions
    {
        Instructions = """
            You are a support assistant for device connectivity questions.
            Use get_device_status before answering anything about a specific device.
            Never promise a fix. Offer to open a ticket instead.
            """,
        Tools = [DeviceTools.GetDeviceStatus, DeviceTools.RequestDiagnosticRun]
    }
});

Console.WriteLine(await agent.RunAsync("Why is device 4417 offline?"));

And the tools are ordinary C#. The one with side effects gets wrapped so the model can request it but never run it directly:

using System.ComponentModel;
using Microsoft.Extensions.AI;

public static class DeviceTools
{
    [Description("Gets the current connectivity status for a device.")]
    public static string GetDeviceStatusById(
        [Description("Device identifier, e.g. 4417")] string deviceId)
        => deviceId switch
        {
            "4417" => "4417: offline since 03:12 UTC, last seen on cell 2231 (mock)",
            _ => $"{deviceId}: online (mock)"
        };

    [Description("Requests a remote diagnostic run on a device.")]
    public static string RequestDiagnosticRunById(string deviceId)
        => $"Diagnostic run queued for {deviceId} (mock)";

    public static AIFunction GetDeviceStatus { get; } =
        AIFunctionFactory.Create(GetDeviceStatusById, "get_device_status");

    public static AIFunction RequestDiagnosticRun { get; } =
        new ApprovalRequiredAIFunction(
            AIFunctionFactory.Create(RequestDiagnosticRunById, "request_diagnostic_run"));
}

A quick note on packages: the harness APIs were still prerelease when I wrote this, so the exact package set moves. I install with dotnet add package Microsoft.Agents.AI --prerelease plus the Azure AI Projects packages, and I check the MafClaw sample repo (aka.ms/mafclaw/repo) for the current versions before I trust anything I remember.

If you don’t want a cloud model for this at all , the harness does not care where the IChatClient comes from. For local development I swap the first block for Ollama through OllamaSharp, which implements IChatClient directly:

# Run Ollama locally (or: brew install ollama && ollama serve)
docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama
docker exec ollama ollama pull qwen3:8b

dotnet add package OllamaSharp

using OllamaSharp;

IChatClient chatClient = new OllamaApiClient(
    new Uri("http://localhost:11434"), "qwen3:8b");

// Everything below, AsHarnessAgent(...) included, stays exactly the same.

Pick a local model that actually supports tool calling. Small models will happily ignore your tools and answer from vibes, and the harness can’t fix that for you.

The part that convinced me the harness was the right tier was the approval flow. In one of the MafClaw sessions someone asked what happens if the user never answers an approval prompt. The answer became a whole sample with timeouts and bounded retries. Silence is not consent. That is the kind of edge case I would not have thought about until production taught me, and a harness that has already been taught it is worth a lot.

What I gave up: transparency. When the assistant does something odd, the first question is always “was that the model, or the harness?” I now keep OpenTelemetry tracing on from day one, not as an afterthought, because a harness without traces is a black box you are responsible for.

2. The internal batch agent: I picked a framework

The second integration is not a conversation. It is a nightly job that reads a few thousand support tickets, classifies them, pulls related device telemetry, drafts a summary per customer segment, and writes a report that product managers read over coffee.

Nobody is chatting with it. Nobody approves individual steps. What matters is completely different:

If it crashes at ticket 2,300, it must resume from ticket 2,301, not from zero
Each stage must be inspectable on its own when a summary looks wrong
Token spend per run needs a hard ceiling

This is the “where must state survive a crash” question, and here the answer is unambiguous: at every step boundary.

A harness is the wrong shape for this. Harnesses are built around an open-ended conversational loop where the model decides what comes next. My batch job has a fixed shape. Classify, enrich, summarize, write. The model should not decide the order of stages. I should.

So this runs on Microsoft Agent Framework workflows, the framework side of the same library. Each stage is an executor, the edges are declared, and checkpoints persist state between them. The model does real work inside each node, but the graph is mine.

I considered LangGraph seriously. It is arguably more mature for exactly this pattern. What tipped it was boring: the rest of this pipeline is .NET, the team reads C#, and the telemetry already flows through the same OpenTelemetry setup as everything else. Framework choice is rarely about the framework.

What I gave up: simplicity, and some stability. Framework APIs in this space move fast. I have rewritten graph wiring code twice this year because of breaking changes, which is the tax you pay for renting someone else’s primitives.

3. The narrow helper: I picked an SDK (the second time)

The third integration is tiny. Inside a larger ASP.NET Core application there is a form where an operator types a free-text description of a problem, and a helper turns that into a structured filter: device group, time window, error class. It has exactly one tool, which looks up whether a device group name is valid. It runs maybe a few hundred times a day.

The loop here is at most two or three turns. There is no state worth persisting. If it fails, the operator types the filter by hand, which is what they did before.

This is SDK territory. Here is roughly what it looks like, written against IChatClient so it runs the same against Anthropic in production and Ollama on my laptop:

using System.ComponentModel;
using Microsoft.Extensions.AI;

public sealed class FilterHelper(IChatClient client)
{
    private const int MaxTurns = 3;

    [Description("Returns true if the device group name exists.")]
    private static bool IsValidGroup(string groupName) =>
        groupName is "eu-fleet" or "us-fleet" or "test-bench"; // real lookup goes here

    private readonly AIFunction _groupTool =
        AIFunctionFactory.Create(IsValidGroup, "is_valid_group");

    public async Task<string?> BuildFilterAsync(string input, CancellationToken ct = default)
    {
        List<ChatMessage> messages =
        [
            new(ChatRole.System,
                "Convert the request into JSON with keys group, from, to, errorClass. " +
                "Validate the group with is_valid_group. Reply with JSON only."),
            new(ChatRole.User, input)
        ];

        var options = new ChatOptions { Tools = [_groupTool], MaxOutputTokens = 300 };

        for (var turn = 0; turn < MaxTurns; turn++)
        {
            ChatResponse response = await client.GetResponseAsync(messages, options, ct);
            messages.AddMessages(response);

            var calls = response.Messages
                .SelectMany(m => m.Contents)
                .OfType<FunctionCallContent>()
                .ToList();

            if (calls.Count == 0)
                return response.Text; // done, model answered

            foreach (var call in calls)
            {
                var result = await _groupTool.InvokeAsync(
                    new AIFunctionArguments(call.Arguments), ct);
                messages.Add(new ChatMessage(ChatRole.Tool,
                    [new FunctionResultContent(call.CallId, result)]));
            }
        }

        return null; // hit the cap, caller falls back to the manual form
    }
}

Every line of that loop is mine, including the MaxTurns cap that the Friday-afternoon version of me forgot. That is the whole point. There is nothing here I can't see.

If you are in Python, Anthropic’s tool runner gets you the same shape with even less code:

from anthropic import Anthropic, beta_tool

client = Anthropic() # reads ANTHROPIC_API_KEY

@beta_tool
def is_valid_group(group_name: str) -> bool:
    """Return True if the device group name exists."""
    return group_name in {"eu-fleet", "us-fleet", "test-bench"}

runner = client.beta.messages.tool_runner(
    model="claude-sonnet-5",
    max_tokens=300,
    tools=[is_valid_group],
    messages=[{"role": "user", "content": "errors in eu-fleet since Monday"}],
)

final = runner.until_done()
print(final.content[0].text)

Just remember that the runner runs the loop, but it does not bound it for you in any way that fits your budget. Put your own limit around it.

The wrong choice, and what it cost me

I said “the second time” above because the first version of that helper was not built on an SDK. It was built on a framework.

At the time we were adopting a graph-based framework for the batch pipeline, and I was excited about it. So when the filter helper came up, I reached for the same tool. I modeled a three-turn interaction as a graph: a parse node, a validate node, a conditional edge back to parse, a checkpointer (in-memory, because what was I going to persist?), and a state schema with five fields for what was really a list of messages.

It worked. It also cost me in ways that took months to add up:

Latency. The framework’s per-step overhead was small, but the helper sits on a form submit. Users noticed a sluggish feel that the raw call didn’t have once I measured them side by side.
Tokens. My graph’s state carried the full intermediate structure into every node’s prompt. The raw version sends only the messages it needs. The prompt got noticeably smaller once I rewrote it.
Upgrades. Two framework version bumps broke that helper even though it used almost none of the framework’s features. Each time, someone had to relearn the graph API to fix a component that should have been forty lines of plain C#.
Debugging. When the helper returned garbage, the stack trace went through the framework’s scheduler. For a two-turn loop, that is absurd.

The rewrite took one afternoon. The code shrank to what you saw above. Nothing got worse.

The lesson I took from it is not “frameworks are bad.” The batch agent is much better for being on one. The lesson is that I chose a tier based on what I was excited about, not on who needed to own the loop. A framework earns its keep when state has to survive something. When there is no state worth surviving, you are paying rent for an empty apartment.

Elif Sude Gökay makes a point in her piece on harness engineering that fits here: AI can make code cheap, but it does not automatically make trustworthy change cheap. The runtime choice is part of what makes change trustworthy, or not. Picking the heaviest tier by default is a way of making every future change more expensive.

A fair warning about the audit itself

I want to be careful not to turn one paper into scripture, and I think the paper would agree.

First, it is a source-code study, not a benchmark. It tells you how these harnesses are built. It does not tell you which one produces better outcomes on your tasks, and nothing in it measures that directly.

Second, the snapshots are snapshots. The authors pinned specific versions and even compared eight of them across one quarter, which is genuinely useful because it shows how fast this space converges and copies itself. But it also means some of what they describe may already be different in the version you are running today. Claude Code in particular is closed-source, and as far as I can tell the analysis of it is much harder for an outsider to reproduce than, say, the Aider or Mini-SWE-Agent sections. I read those parts with more caution.

Third, the “zero frameworks at runtime” finding is about coding harnesses built by teams whose whole product is the harness. Of course they own the loop. That is their job. It does not follow that your internal ticket pipeline should hand-roll its own checkpointing. The finding tells you what vendors do, not what you should do.

So I treat the taxonomy the way I treat a good architecture pattern name: as a thinking tool that makes conversations shorter and decisions more honest. Not as a ranking.

The checklist I use now

After the audit, I wrote down the questions I actually ask before the next integration. They are short on purpose.

1. Who should decide what happens next: the model, or me? If the model should drive (open-ended, conversational, unknown number of steps), lean harness. If I should drive (fixed stages, known shape), lean framework or SDK.

2. What state must survive a crash, and where? If the answer is “nothing,” you almost certainly want an SDK. If the answer is “every step boundary,” you want a framework with real checkpointing. If the answer is “the conversation and some user memory,” a harness usually has that covered.

3. Is a human in the loop for side effects? Approval flows are surprisingly hard to get right (timeouts, retries, sticky denials). If you need them and they aren’t your product, a harness that ships them is worth its opacity.

4. How many turns, realistically? One to three turns with one or two tools is SDK territory. Write the loop, put a cap on it, move on.

5. Can I trace it on day one? Whatever tier you pick, if you can’t see every model call and tool call, you will end up blaming the wrong layer. Ask me how I know.

6. Can it run locally? If the runtime only works against one hosted model, your dev loop and your tests get expensive. Anything built on IChatClient (or an OpenAI-compatible endpoint in Python) can point at Ollama for development. I now treat that as a requirement, not a nice-to-have.

7. Am I picking this because it fits, or because I’m already using it elsewhere? This is the one that would have saved me the most. Reuse is a good reason to pick between two options that both fit. It is a bad reason to pick an option that doesn’t.

Where I landed

My stack didn’t change much after this exercise. The chatbot stayed on a harness, the batch job stayed on a framework, and the helper had already been fixed. What changed is the vocabulary. When something breaks now, I can say which layer owns the loop, and that usually tells me where to look.

And when someone on the team says “the agent framework did it,” I ask the question that embarrassed me a few weeks ago: which one?

Sources

Jaroslaw Wasowski, “Harness vs. Framework vs. SDK: How to Choose a Coding Agent’s Runtime,” Level Up Coding: https://levelup.gitconnected.com/harness-vs-framework-vs-sdk-how-to-choose-a-coding-agents-runtime-2ac776994c3a
Paul Barbaste et al., “Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents: A Source-Code Study of Eleven Systems,” arXiv 2609.00006: https://arxiv.org/abs/2609.00006
Bruno Capuano, “Build Your Own AI Agent Harness in C#, the MafClaw Live Series,” .NET Blog: https://devblogs.microsoft.com/dotnet/build-your-own-ai-agent-harness-in-csharp-the-maf-claw-live-series/
Elif Sude Gökay, “Harness Engineering: When AI Writes the Code, What Are Engineers Actually Engineering?”: https://ai.plainenglish.io/harness-engineering-when-ai-writes-the-code-what-are-engineers-actually-engineering-4d45e21c1777

Tags: AI Agents, Harness Engineering, Dotnet, CSharp, Software Architecture, Claude, LLM, Microsoft Agent Framework, Ollama

Strongest 5 for Medium: AI Agents, Harness Engineering, Dotnet, Software Architecture, LLM

My Eval Suite Was Lying to Me: Auditing the Judge, Not Just the Agent

Ali Suleyman TOPUZ — Thu, 01 Oct 2026 10:09:36 +0000

I need to admit something first. For about six weeks this summer, I ran an LLM-as-judge eval suite against an agent I maintain, watched the score go up, watched it go down, and made decisions based on that single number every single time. I never once opened the judge’s reasoning text. I looked at the score, I looked at the trend line, and I moved on. It felt responsible. I had an eval suite. Most people I know still don’t.

It wasn’t responsible. It was theater.

The agent in question takes a pull request diff and a bit of repo context and turns it into a structured, ready-to-file ticket: title, summary, affected components, suggested acceptance criteria. It’s the kind of task that’s easy to demo and genuinely hard to evaluate, because “is this a good ticket” is a judgment call, not a string match. So like a lot of people building agents right now, I reached for LLM-as-judge: have a second model read the input, read the agent’s output, and score how well it did.

I’d looked at DeepEval early on, since it’s become something of a default for this kind of work. It’s a solid library, but it’s Python, and my agent and its surrounding tooling live in .NET. Rather than bridge two runtimes for what is, underneath the abstractions, a chat completion call and a JSON parse, I hand-rolled a small evaluator in C#. If you’re on DeepEval, or Promptfoo, or any of the other eval frameworks, the lesson in this article applies just as much to you. The library isn’t the problem. The habit of trusting the number without reading the reasoning is the problem, and it’s a library-agnostic habit.

For six weeks the score sat around 0.78 on a 0 to 1 scale, drifted a little with each prompt change, and I treated that drift as ground truth. Then I had a slow Friday, pulled up the raw judge outputs instead of the aggregated score, and started actually reading what the model had written before it produced that number. It took maybe forty minutes to go through twenty cases. It was one of the more useful forty minutes I’ve spent on this project all year.

What I Was Actually Evaluating

Before I get into what I found, it’s worth being specific about the setup, because the failure modes I hit are tied to how the harness was built, and yours will have its own version of these even if the shape is different.

Each eval case is an input diff, a short description of the expected behavior (“the agent should identify the correct repo, produce a title under 80 characters, and list at least two acceptance criteria”), and the agent’s actual output. The judge model receives all three and returns a score from 0.0 to 1.0 plus a free-text explanation. I was running the judge on a self-hosted Qwen 2.5 model through Ollama, partly out of cost discipline and partly because I wanted the eval loop to run entirely on my own machine without depending on an external API staying up or staying priced the same. Everything in this article runs against that local setup; if you’d rather point the same code at a hosted model, it’s a one-line change to the judge implementation later on.

The scores got logged to a CSV, and a small dashboard script plotted the rolling average after every run. That dashboard was the only thing I looked at. The full judge response, reasoning included, was sitting in a JSON Lines file the whole time. I just never opened it.

The Day I Actually Read the Reasoning

Once I started reading instead of skimming the number, three distinct kinds of problem showed up, and none of them were things the aggregate score could have told me.

The first was the one that stung the most: cases where the judge’s own explanation directly contradicted its score. One case scored 0.90, which in my rubric means “matches expected behavior, no meaningful issues.” The reasoning read, almost verbatim, “the ticket is missing acceptance criteria and references the wrong repository, though the summary line is well formatted.” That is not a 0.90. That is closer to a 0.2. I found four more like it in that first batch of twenty, always in the same direction: strongly critical reasoning attached to a high score. I have a guess about why, which I’ll get to in the calibration section, but the honest answer is I don’t fully know, and that uncertainty is itself worth taking seriously.

The second problem was a stale rubric. One of my metrics checked whether the ticket followed “the standard template,” but the standard template referenced in the judge prompt was the template from an earlier version of the agent. I’d updated the agent’s output format three prompt revisions ago to drop a “Reproduction Steps” section that didn’t make sense for most of the tickets it generates, but I never updated the judge’s rubric to match. So for weeks, the judge was quietly penalizing the agent for correctly not including a section I’d deliberately removed. The score dip I’d attributed to “the new prompt made things worse” was actually the eval suite grading against a spec that no longer existed.

The third was a parameter I thought I controlled and didn’t. I run every judge call with temperature: 0.0, expecting deterministic scoring for the same input. When I reran ten of the flagged cases a second time to see if the mismatch was reproducible, four of them came back with different scores and noticeably different reasoning text than the first pass. The temperature setting was being sent correctly in the request; I checked the payload. But the specific quantized build of the model I had pulled didn't honor it consistently, something I only found because I went looking, not because anything failed loudly. A silently-ignored parameter is worse than a documented limitation, because there's no error message pointing you at it.

None of these three problems showed up as a change in the aggregate score. They showed up as noise I’d been averaging away.

Building a Diagnostic Harness

Once I knew what to look for, I didn’t want to keep finding it by manually scrolling through JSON files, so I built a small harness that runs alongside the existing eval suite and does three things: logs the full reasoning next to the score (which the original suite already did, it just wasn’t surfaced anywhere), flags cases where the reasoning’s sentiment and the score’s direction disagree, and produces a short report of the worst offenders for me to read by hand.

The core types are deliberately plain:

public record EvalCase(
    string Id,
    string Input,
    string ExpectedBehavior,
    string ActualOutput);
public record JudgeVerdict(double Score, string Reasoning);
public interface IJudge
{
    Task<JudgeVerdict> EvaluateAsync(EvalCase evalCase, CancellationToken ct = default);
}

The judge implementation talks to a local Ollama instance over HTTP, using structured JSON output so the score and reasoning come back as a real object rather than something I have to regex out of free text:

public class OllamaJudge : IJudge
{
    private readonly HttpClient _http;
    private readonly string _model;
    public OllamaJudge(HttpClient http, string model = "qwen2.5:14b-instruct")
    {
        _http = http;
        _model = model;
    }
    public async Task<JudgeVerdict> EvaluateAsync(EvalCase evalCase, CancellationToken ct = default)
    {
        var prompt = $"""
            You are grading whether an AI agent's output correctly follows the
            expected behavior described below. Be specific and critical in your
            reasoning; do not soften your explanation to match a score you have
            not yet decided on.
            Input given to the agent:
            {evalCase.Input}
            Expected behavior:
            {evalCase.ExpectedBehavior}
            Actual agent output:
            {evalCase.ActualOutput}
            Score the actual output from 0.0 (completely wrong) to 1.0 (fully
            correct). Respond with JSON only, in exactly this shape:
            {{"score": <number between 0 and 1>, "reasoning": "<at least two full sentences>"}}
            """;
        var payload = new
        {
            model = _model,
            messages = new[] { new { role = "user", content = prompt } },
            format = "json",
            stream = false,
            options = new { temperature = 0.0 }
        };
        var response = await _http.PostAsJsonAsync(
            "http://localhost:11434/api/chat", payload, ct);
        response.EnsureSuccessStatusCode();
        var body = await response.Content
            .ReadFromJsonAsync<OllamaChatResponse>(cancellationToken: ct);
        var options = new JsonSerializerOptions { PropertyNameCaseInsensitive = true };
        return JsonSerializer.Deserialize<JudgeVerdict>(body!.Message.Content, options)!;
    }
    private record OllamaChatResponse(OllamaMessage Message);
    private record OllamaMessage(string Role, string Content);
}

If you’re on a hosted model instead, the only thing that changes is the constructor and the HTTP call target; the rest of the harness, and everything below, is identical either way. That’s the point of putting the judge behind IJudge in the first place.

The harness itself just runs the cases and keeps the full verdict, not just the score:

public record EvalRecord(EvalCase Case, JudgeVerdict Verdict, DateTime RanAt);
public class EvalHarness
{
    private readonly IJudge _judge;
    public EvalHarness(IJudge judge) => _judge = judge;
    public async Task<List<EvalRecord>> RunAsync(IEnumerable<EvalCase> cases)
    {
        var results = new List<EvalRecord>();
        foreach (var c in cases)
        {
            var verdict = await _judge.EvaluateAsync(c);
            results.Add(new EvalRecord(c, verdict, DateTime.UtcNow));
        }
        return results;
    }
}

Nothing here is clever. That’s deliberate. The whole point of this exercise was that I’d built something clever (an aggregate dashboard) on top of something I never actually inspected (the raw reasoning), and the fix was to stop skipping the boring, readable middle layer.

Catching Reasoning and Score Disagreement

I don’t think you can reliably automate “does this reasoning actually support this score” with another LLM call without just moving the trust problem one level up. What I wanted instead was a cheap, dumb, transparent heuristic that would flag candidates for a human to read, not a replacement for reading them.

The approach I landed on: keyword-based sentiment on the reasoning text, compared against the direction of the score. It is a heuristic in the least flattering sense of the word. It will misfire on sarcasm, on reasoning that lists both strengths and weaknesses in genuinely balanced proportion, and on domain-specific negative words that aren’t actually negative in context (“the ticket correctly flags this as a breaking change”). I want to be explicit about that, because the value of this tool is entirely in how cheaply it narrows down what a human reads next, not in any claim to accuracy on its own.

public static class ReasoningMismatchDetector
{
    private static readonly string[] NegativeWords =
    {
        "fail", "failed", "failure", "incorrect", "wrong", "missing",
        "does not", "doesn't", "did not", "didn't", "inconsistent",
        "hallucin", "contradicts", "ignores", "omits", "unclear", "confus"
    };
    private static readonly string[] PositiveWords =
    {
        "correct", "accurate", "matches", "complete", "clear", "consistent",
        "satisfies", "fulfills", "aligned", "appropriate", "well formatted",
        "well structured"
    };
    // Returns a value from -1 (entirely negative-sounding) to 1 (entirely
    // positive-sounding). A reasoning string with no keyword hits returns 0,
    // meaning "no signal," not "neutral."
    public static double SentimentScore(string reasoning)
    {
        var text = reasoning.ToLowerInvariant();
        int neg = NegativeWords.Count(w => text.Contains(w));
        int pos = PositiveWords.Count(w => text.Contains(w));
        if (neg + pos == 0) return 0.0;
        return (double)(pos - neg) / (pos + neg);
    }
    // Flags cases where the reasoning's sentiment and the score's direction
    // disagree by more than the threshold. This is a keyword heuristic, not
    // a sentiment classifier, and it exists purely to triage cases for a
    // human to read, not to replace that reading.
    public static bool IsMismatch(JudgeVerdict verdict, double threshold = 1.0)
    {
        var sentiment = SentimentScore(verdict.Reasoning);
        if (sentiment == 0.0) return false;
        var normalizedScore = (verdict.Score * 2) - 1; // map 0..1 onto -1..1
        var disagreement = Math.Abs(sentiment - normalizedScore);
        return disagreement >= threshold;
    }
}

Running this over the same batch of cases that first tipped me off surfaced the exact ticket I’d found by hand, plus two more I’d missed:

CASE ID SCORE SENTIMENT DISAGREEMENT REASONING (TRUNCATED)
------------------------------------------------------------------------------
tkt-0042 0.90 -0.67 1.57 "missing acceptance criteria and
                                               references the wrong repo, though
                                               the summary line is well formatted"
tkt-0107 0.85 -0.50 1.35 "ignores the priority field
                                               entirely, title is otherwise fine"
tkt-0031 0.20 0.60 0.80 "correctly captures scope and
                                               components, wording is a bit terse"

tkt-0031 is a good example of the heuristic's limits: it's flagged, but reading it, the judge was reasonably scoring down for terseness on a case where terseness genuinely mattered. Not every flag is a bug. That's expected, and it's fine, because the tool's job is to shrink a haystack, not to find the needle itself.

The Report

The report generator just sorts by disagreement and prints the top handful, so a five-minute morning check replaces the forty-minute manual scroll I did the first time:

public static class MismatchReport
{
    public static string Generate(IEnumerable<EvalRecord> records, int topN = 15)
    {
        var flagged = records
            .Select(r => new
            {
                r.Case.Id,
                r.Verdict.Score,
                Sentiment = ReasoningMismatchDetector.SentimentScore(r.Verdict.Reasoning),
                r.Verdict.Reasoning
            })
            .Where(r => r.Sentiment != 0.0)
            .Select(r => new
            {
                r.Id,
                r.Score,
                r.Sentiment,
                Disagreement = Math.Abs(r.Sentiment - ((r.Score * 2) - 1)),
                r.Reasoning
            })
            .OrderByDescending(r => r.Disagreement)
            .Take(topN);
        var sb = new StringBuilder();
        sb.AppendLine("CASE ID SCORE SENTIMENT DISAGREEMENT REASONING");
        foreach (var r in flagged)
        {
            var truncated = r.Reasoning.Length > 60
                ? r.Reasoning[..60] + "..."
                : r.Reasoning;
            sb.AppendLine(
                $"{r.Id,-10} {r.Score,-7:0.00} {r.Sentiment,-11:0.00} {r.Disagreement,-14:0.00} {truncated}");
        }
        return sb.ToString();
    }
}

I run this after every eval batch now, not just when something looks off. It became a five-minute step, which is the only reason I actually keep doing it.

How Many Judge Outputs You Actually Need to Check

Finding bugs by reading twenty cases by hand is useful once, but it doesn’t tell you whether your judge is trustworthy in general, only that it was wrong on those twenty. For that you need a real calibration pass: a sample of judge outputs, scored independently by a human, compared against the judge’s own scores.

My rule of thumb, and I’ll say plainly this is a rule of thumb rather than a statistically airtight sample size, is at least 30 to 50 cases for an initial calibration, pulled with deliberate stratification across score bands rather than pure random sampling. Cases near your pass/fail boundary are the ones where judge and human are most likely to disagree, and pure random sampling under-represents them if most of your traffic clusters at the high end, which mine does.

To compare human scores against judge scores in a way that accounts for chance agreement, I use Cohen’s kappa. It’s a simple statistic, and it’s much more informative than raw percent agreement, because percent agreement doesn’t correct for two raters who’d agree by coincidence just from both leaning toward the same common answer.

public static class AgreementMetrics
{
    // Both rating lists must use the same category indices, e.g. bucket
    // continuous scores into fail=0, borderline=1, pass=2 before calling this.
    public static double CohensKappa(
        IReadOnlyList<int> raterA, IReadOnlyList<int> raterB, int categoryCount)
    {
        if (raterA.Count != raterB.Count)
            throw new ArgumentException("Rating lists must be the same length.");
        int n = raterA.Count;
        var confusion = new int[categoryCount, categoryCount];
        for (int i = 0; i < n; i++)
            confusion[raterA[i], raterB[i]]++;
        double po = 0;
        for (int k = 0; k < categoryCount; k++)
            po += confusion[k, k];
        po /= n;
        var rowTotals = new double[categoryCount];
        var colTotals = new double[categoryCount];
        for (int i = 0; i < categoryCount; i++)
            for (int j = 0; j < categoryCount; j++)
            {
                rowTotals[i] += confusion[i, j];
                colTotals[j] += confusion[i, j];
            }
        double pe = 0;
        for (int k = 0; k < categoryCount; k++)
            pe += (rowTotals[k] / n) * (colTotals[k] / n);
        return (po - pe) / (1 - pe);
    }
}

I bucket my continuous 0 to 1 scores into three categories (fail below 0.5, borderline 0.5 to 0.8, pass above 0.8) before running this, since Cohen’s kappa is built for categorical ratings, not continuous ones, and forcing a bucket also matches how I actually use the score downstream.

For interpreting the result, I use the Landis and Koch scale, which is a common reference point even though it’s a rough guide rather than a hard law of statistics:

KAPPA RANGE INTERPRETATION
--------------------------------------
< 0.00 worse than chance
0.00 - 0.20 slight agreement
0.21 - 0.40 fair agreement
0.41 - 0.60 moderate agreement
0.61 - 0.80 substantial agreement
0.81 - 1.00 near-perfect agreement

My personal threshold, for a judge I’m willing to let gate something automatically (blocking a merge, failing a build), is substantial agreement, kappa at or above 0.6. For a judge I’m only using descriptively, to rank cases for my own attention rather than to make a pass/fail call, I’ll tolerate moderate agreement, in the 0.4 to 0.6 range, as long as I keep reading the flagged cases myself rather than trusting the aggregate.

When I ran this calibration on my own setup, using 40 stratified cases, I landed at kappa of 0.51. Moderate. Good enough to keep using descriptively, not good enough to let gate anything on its own.

What I Do Differently When Agreement Comes Back Low

A 0.51 kappa isn’t a single problem with a single fix, so I don’t treat it as one. I work through three things in order.

First, I try a different judge model on the same calibration set, holding everything else constant, purely to find out whether the disagreement is a model capability issue or a prompt issue. Swapping my local Qwen model for a larger one, or briefly pointing the same IJudge interface at a hosted model, tells me quickly whether the ceiling is the model or the instructions I'm giving it. If a bigger or different model closes most of the gap, that's model capability. If it doesn't move much, the prompt is the problem.

Second, assuming it’s the prompt, I rewrite the judge’s rubric to be more concrete, usually by adding two or three worked examples directly in the prompt, one of a case that should score high and one that should score low, with the reasoning spelled out. Vague rubrics (“is this a good ticket”) invite vague, inconsistent reasoning. Specific rubrics (“does the ticket name a real repository in this list, does it include at least two acceptance criteria, is the title under 80 characters”) give the judge less room to drift.

Third, for any single sub-check that has a clear right answer, I stop asking the judge at all. Whether the ticket references a repository that actually exists in my org is not a judgment call, it’s a lookup, and I moved it into a deterministic code check that runs alongside the LLM judge rather than inside it. The judge is genuinely useful for the parts of “good ticket” that require judgment. It’s a liability for the parts that don’t.

My Rule Now

I calibrate the judge against human scoring roughly once a quarter, and immediately after any change to the judge prompt, the judge model, or the agent’s own prompt, since any of those three can shift the relationship between score and reality without warning. Between calibrations, I read the mismatch report after every batch, which takes minutes, not hours, now that the harness does the sorting for me.

The habit I’ve actually changed is smaller than all of this tooling, though. It’s that “the eval suite passed” no longer means “the agent is good” to me. It means the agent produced outputs that a specific model, under a specific prompt, with a specific and only partially-verified relationship to human judgment, scored favorably. That’s a real signal, and it’s worth having. It is not, on its own, the thing I used to treat it as.

Tags: llm-evaluation, dotnet, ai-agents, ollama, deepeval, software-testing, machine-learning

Who Authorized the Agent? Building Real Identity and Delegation for AI Agents in .NET

Ali Suleyman TOPUZ — Tue, 29 Sep 2026 16:25:31 +0000

What RFC 8693 actually buys you that a diagram with boxes labeled “policy” never will

The gym story I wrote about last month left me with a checklist: does every write endpoint check ownership, is the action reversible, are you logging what an agent does on someone else’s data. I ran that checklist against a scheduling assistant I’d built for a client and found two endpoints that failed it. Fixing those took an afternoon. What took a lot longer, and is the actual subject of this article, is a question the checklist doesn’t answer at all: when that scheduling assistant’s calendar agent calls the meeting API on my client’s behalf, what identity is it actually using to make that call?

I’d been putting that question off with the same answer everyone puts it off with: the agent has an API key with a scope roughly shaped like “whatever it might plausibly need,” and there’s a “policy layer” somewhere that’s supposed to catch anything that goes wrong. I’ve read a dozen posts on agent governance this year, and most of them stop at exactly that level of detail. There’s a diagram. It has boxes. One box says “identity,” one says “policy,” one says “audit log,” and arrows connect them in a way that looks complete right up until you ask what’s actually inside any of the boxes. Nothing in those diagrams would have stopped the gym story, because nothing in them distinguishes the one question that story turned on: when an agent calls a downstream API on a user’s behalf, is it impersonating that user, or is it delegating on their behalf while remaining itself? Most systems I’ve worked on, mine included until recently, don’t even ask that question, let alone answer it in a way you could audit.

So I built the thing properly, in .NET, using RFC 8693, OAuth 2.0 Token Exchange, as the actual standard underneath the “identity” box instead of a diagram of one. This is the writeup: what the RFC actually specifies, a worked three-hop example with real C# and real JWTs, what it gets you when something goes wrong, and where it stops being enough on its own.

Impersonation and delegation are not the same failure mode

Here’s the distinction RFC 8693 draws, and it matters because the two failure modes it separates look identical from a diagram but are not identical in a database.

Impersonation means principal A is given all of principal B’s rights and is, within that context, indistinguishable from B. A downstream system that receives an impersonation token has no way to know an agent was even involved. Every log entry says “alice did this,” full stop, because as far as the token is concerned, alice did do it. This is what a bare bearer token handed to an agent gives you by default: the agent holds alice’s credential, or a credential minted to look exactly like alice’s, and every API it touches sees alice.

Delegation means A keeps its own identity, and it’s explicit in the token that B authorized A to act, specifically, on B’s behalf. The downstream system sees two identities in the same token: whose authority this is being exercised under, and who is actually holding the phone. That distinction is the entire difference between “we can’t tell whether alice or her calendar agent cancelled that meeting” and “alice’s calendar agent cancelled that meeting, acting under an authorization alice gave it, and here is the token that proves it.”

Most agent systems I’ve seen default to impersonation without anyone deciding that on purpose. It’s the path of least resistance: mint a token that looks like the user’s, hand it to the agent, move on. It works right up until you need to answer “did the agent do this, or did the user,” and you find out your logs can’t tell the difference because your tokens never could either.

What RFC 8693 actually specifies

Token Exchange defines a grant type for the standard OAuth token endpoint: urn:ietf:params:oauth:grant-type:token-exchange. A client POSTs to /token with a subject_token, the token representing whoever the action is really for, and gets back a new token, possibly narrower in scope, possibly for a different audience, possibly with a different actor attached. Here's the full parameter set:

TOKEN EXCHANGE REQUEST PARAMETERS (RFC 8693 section 2.1)
--------------------------------------------------------------------
Parameter Required What it carries
--------------------------------------------------------------------
grant_type yes urn:ietf:params:oauth:grant-type:
                                 token-exchange
subject_token yes the token for whoever the action is
                                 really for (the human, or a prior
                                 delegation token)
subject_token_type yes identifies the subject token's format,
                                 e.g. urn:ietf:params:oauth:token-
                                 type:access_token
actor_token no the token for whoever is actually
                                 making this call (the agent itself)
actor_token_type cond. required if actor_token is present
requested_token_type no the format you want back
resource / audience no where this token is meant to be used
scope no the scopes you're asking to be
                                 minted into the new token
--------------------------------------------------------------------

The response carries access_token, issued_token_type, and token_type, same shape as any other token response, which is part of what makes this practical: nothing downstream needs to know a token exchange happened at all, it's still just a JWT arriving in an Authorization header.

The part that actually implements delegation lives inside the token payload, in two claims the RFC defines specifically for this purpose. The act claim says who is currently acting, as a nested JSON object, not a flat string:

{
  "sub": "alice@contoso.example",
  "act": { "sub": "triage-agent" }
}

For a chain of delegations, act nests. The RFC is explicit about the ordering, and it's the opposite of what I guessed on my first read: "the outermost act claim represents the current actor while nested act claims represent prior actors. The least recent actor is the most deeply nested."

{
  "sub": "alice@contoso.example",
  "act": {
    "sub": "calendar-agent",
    "act": { "sub": "triage-agent" }
  }
}

Read that from the outside in and you get the chain in reverse chronological order: calendar-agent is acting right now, triage-agent acted before it, alice is who this is all ultimately for. may_act is the other half, a preemptive statement issued into a token that says who's allowed to become the actor before any exchange happens, which is what an authorization server checks against before it agrees to mint a delegation in the first place.

None of this is what a static API key gives you. A key has no subject, no actor, no chain, no expiry worth mentioning, and it’s usually shared across every call the agent ever makes, to every downstream service, for the life of the deployment. Token exchange gives you a new, short-lived, purpose-built token for every hop, with the delegation chain baked into the token itself instead of living in a wiki page somewhere describing what the key is “supposed to” be used for.

Building the three-hop chain

Here’s the scenario I built against: a user’s request goes to a triage agent, which calls a calendar agent, which calls a meeting-scheduling API. Each hop is its own service with its own credentials, and each one has to prove, cryptographically, that it’s allowed to be standing where it’s standing in that chain.

alice --request--> triage-agent --calls--> calendar-agent --calls--> meeting-api
         (subject) (exchanges (exchanges
                                      for calendar-agent for meeting-api
                                      audience) audience)

I went looking for an off-the-shelf identity provider that implements the delegation half of this out of the box before writing any of it myself, because I’d rather not maintain a security token service if I don’t have to. Keycloak 26.2 shipped a genuinely standards-compliant Token Exchange implementation in mid-2025, which was promising. Then I read the fine print: Keycloak’s standard token exchange handles the subject_token half fine, but as of the version I tested, it does not emit an act claim at all, delegation via actor_token is still tracked as an open feature request, not a shipped one. Duende IdentityServer doesn't implement RFC 8693 natively either; their own docs point you at the extension grant mechanism and tell you to build the exchange logic yourself. So for this article I built a minimal token exchange service directly, which turned out to be the right call anyway, because it's the only way to actually show you what's happening to the token at each hop instead of trusting a product's marketing claims about it.

The nested actor claim, as a real type

using System.Text.Json.Serialization;
public sealed class ActorClaim
{
    [JsonPropertyName("sub")]
    public string Sub { get; set; } = default!;
    [JsonPropertyName("act")]
    public ActorClaim? Act { get; set; }
}

The exchange service

This is the piece that stands in for the authorization server. It reads the incoming subject and actor tokens, checks a delegation policy, and mints a new token with the actor chain extended by one hop. I’m using Microsoft.IdentityModel.JsonWebTokens here rather than the older System.IdentityModel.Tokens.Jwt, it's the handler Microsoft actually recommends going forward, and its JsonWebToken type gives you TryGetPayloadValue, which deserializes a claim straight into a C# type through System.Text.Json, nested objects included, instead of handing you a flat string you have to re-parse.

// dotnet add package Microsoft.IdentityModel.JsonWebTokens
// dotnet add package System.IdentityModel.Tokens.Jwt
using Microsoft.IdentityModel.JsonWebTokens;
using Microsoft.IdentityModel.Tokens;
public sealed class DelegationPolicy
{
    // actorId -> the audiences that actor is allowed to exchange a token for.
    // This is the "may_act" check in practice: a lookup table today,
    // a policy engine call once you have more than a handful of agents.
    private readonly Dictionary<string, HashSet<string>> _mayActFor = new()
    {
        ["triage-agent"] = new HashSet<string> { "calendar-agent" },
        ["calendar-agent"] = new HashSet<string> { "meeting-api" },
    };
    public bool CanExchange(string actorId, string audience) =>
        _mayActFor.TryGetValue(actorId, out var allowed) && allowed.Contains(audience);
}
public sealed class TokenExchangeService(
    SigningCredentials signingCredentials,
    DelegationPolicy policy,
    TimeProvider timeProvider)
{
    private readonly JsonWebTokenHandler _handler = new();
    public string Exchange(
        string subjectToken,
        string actorToken,
        string audience,
        IEnumerable<string> requestedScopes)
    {
        var subjectJwt = _handler.ReadJsonWebToken(subjectToken);
        var actorJwt = _handler.ReadJsonWebToken(actorToken);
        var actorId = actorJwt.Subject;
        if (!policy.CanExchange(actorId, audience))
            throw new InvalidOperationException(
                $"{actorId} is not authorized to act on behalf of {subjectJwt.Subject} for audience {audience}");
        // Carry the existing delegation chain forward, if the subject token
        // already has one (this is hop 2 or later), and add this hop on top.
        subjectJwt.TryGetPayloadValue<ActorClaim>("act", out var priorAct);
        var newAct = new ActorClaim { Sub = actorId, Act = priorAct };
        var now = timeProvider.GetUtcNow();
        var descriptor = new SecurityTokenDescriptor
        {
            Issuer = "https://sts.contoso.example",
            Audience = audience,
            IssuedAt = now.UtcDateTime,
            Expires = now.AddMinutes(5).UtcDateTime,
            SigningCredentials = signingCredentials,
            Claims = new Dictionary<string, object>
            {
                ["sub"] = subjectJwt.Subject,
                ["act"] = ToClaimDictionary(newAct),
                ["scope"] = string.Join(' ', requestedScopes),
                ["jti"] = Guid.NewGuid().ToString("N"),
            }
        };
        return _handler.CreateToken(descriptor);
    }
    // SecurityTokenDescriptor.Claims serializes a Dictionary<string, object>
    // graph correctly as nested JSON, but it does not know how to serialize an
    // arbitrary POCO like ActorClaim placed directly as a claim value, that's
    // how the RFC 8693 reference implementations that predate this pattern
    // ended up with an act claim that serialized as empty arrays instead of
    // the actual nested object. Flatten it to dictionaries yourself and you
    // sidestep the whole problem.
    private static Dictionary<string, object> ToClaimDictionary(ActorClaim act)
    {
        var dict = new Dictionary<string, object> { ["sub"] = act.Sub };
        if (act.Act is not null)
            dict["act"] = ToClaimDictionary(act.Act);
        return dict;
    }
}

What the token actually looks like at each hop

Hop 1, alice authenticates normally, no exchange involved yet:

{
  "iss": "https://idp.contoso.example",
  "sub": "alice@contoso.example",
  "aud": "triage-agent",
  "scope": "chat.invoke",
  "exp": 1799999900
}

Hop 2, the triage agent exchanges alice’s token plus its own client-credentials token for a token scoped to the calendar agent:

{
  "iss": "https://sts.contoso.example",
  "sub": "alice@contoso.example",
  "act": { "sub": "triage-agent" },
  "aud": "calendar-agent",
  "scope": "calendar.read calendar.propose_slot",
  "exp": 1799999600
}

Hop 3, the calendar agent takes that token as its own subject_token, adds its own actor_token, and exchanges again for a token scoped to the meeting API:

{
  "iss": "https://sts.contoso.example",
  "sub": "alice@contoso.example",
  "act": {
    "sub": "calendar-agent",
    "act": { "sub": "triage-agent" }
  },
  "aud": "meeting-api",
  "scope": "meeting.schedule",
  "exp": 1799999300
}

The meeting API never talks to the triage agent, never talks to alice directly, and still receives a token that tells it, cryptographically, all three things it needs to know: this is for alice, calendar-agent is the one calling right now, and triage-agent is the one that started this chain. Reading that chain back out is the mirror image of building it:

public static class TokenChainReader
{
    private static readonly JsonWebTokenHandler Handler = new();
    public static (string Subject, ActorClaim? Actor) ReadDelegationChain(string token)
    {
        var jwt = Handler.ReadJsonWebToken(token);
        jwt.TryGetPayloadValue<ActorClaim>("act", out var act);
        return (jwt.Subject, act);
    }
    // Returns the chain in the order it actually happened: the subject first,
    // then each actor in the order it was granted delegation, oldest to newest.
    // The RFC nests newest-first, so this walks the chain and reverses it.
    public static IEnumerable<string> FlattenChain(string subject, ActorClaim? act)
    {
        var hops = new List<string>();
        for (var current = act; current is not null; current = current.Act)
            hops.Add(current.Sub);
        hops.Reverse();
        yield return subject;
        foreach (var hop in hops)
            yield return hop;
    }
}

Calling FlattenChain on the hop 3 token gives you ["alice@contoso.example", "triage-agent", "calendar-agent"], alice first, then the delegation in the order it actually happened, calendar-agent last because it's the one holding the phone right now.

What this actually buys you

The audit question is the one I couldn’t answer honestly before I built this. “Which agent, acting on behalf of which user, called this specific downstream action” used to mean grepping application logs across three services and hoping the correlation IDs lined up. Now it’s a property of the token itself. The meeting API’s access log can record sub, the full act chain, and the jti on every request, and answering that question later is a query against that log, not an investigation across three codebases that may or may not have logged the right thing at the time.

Revocation is the other half, and it’s the part that made the static-API-key approach genuinely uncomfortable in hindsight. A shared key used by many call sites means revoking it means rotating it everywhere it’s used, which is exactly the kind of change nobody wants to make under incident pressure. Delegation enforced at exchange time is a policy change, not a secret rotation: pull calendar-agent out of _mayActFor["triage-agent"] and the very next exchange attempt for that pair fails, while every other agent relationship keeps working untouched. Tokens already issued still carry whatever their five-minute expiry was, which caps the blast radius of a bad delegation automatically, something a long-lived API key never does on its own.

What RFC 8693 does not solve

This is the section I want to be honest about, because it’s the one most “agent governance” content skips past in the other direction, treating a working token exchange as though it were the whole answer. It isn’t. RFC 8693 answers “who is this request really for, and who is actually making it.” It says nothing about whether that specific request should be allowed. My DelegationPolicy.CanExchange check above is a lookup table, and a lookup table is a policy decision point in the loosest possible sense, it doesn't reason about resource attributes, time of day, rate limits, or the interaction between what alice is allowed to do and what calendar-agent is allowed to do on her behalf.

That’s the job of a real policy engine sitting downstream of the token validation, not instead of it. Something like this, as a Rego policy an OPA sidecar evaluates once the meeting API has already validated the token’s signature, audience, and expiry:

package meeting.authz
import future.keywords.if
import future.keywords.in
default allow := false
allow if {
    input.action == "meeting.schedule"
    input.token.aud == "meeting-api"
    "meeting.schedule" in split(input.token.scope, " ")
    current_actor := input.token.act.sub
    current_actor in data.agent_permissions["meeting.schedule"]
    input.token.sub in data.subject_permissions["meeting.schedule"]
}

That query needs the same three pieces of information every time: who this is for, who’s actually calling, and what they’re trying to do. Token exchange is what guarantees those first two are true and can’t be forged. The policy engine, whether that’s OPA, Cedar, or something simpler you wrote yourself, is what decides whether the third one should be allowed given the first two. Neither layer does the other’s job, and a system that has only the token exchange half still can’t answer “should this be allowed,” while a system that has only the policy half is trusting whatever identity claims land in front of it without any real means to verify who actually generated them.

I went into this project assuming the RFC would feel like overkill for three services and one client. It didn’t, mostly because the alternative I’d been living with, a single API key with a scope wide enough to cover everything the agent might plausibly need, was the thing that actually felt reckless once I’d built the version that doesn’t do that. The gym story was a reminder that agents will use whatever door is left open. This is what it looks like to stop leaving one open by default.

The Real Cost Curve of Running Agents in Production, From Four Different Companies

Ali Suleyman TOPUZ — Tue, 29 Sep 2026 16:25:18 +0000

I almost skipped writing this one up as its own piece, because on the surface it looked like four unrelated news items that happened to land in the same week. A big rideshare company posted about keeping AI spend flat. A streaming company open sourced a plugin. A search company published a compression cookbook. A Chinese lab dropped pricing on a new model tier. My first instinct was to treat them as four separate case studies and move on, the way I’d bookmark four blog posts and forget three of them by Friday.

Then I went back and actually read all four closely instead of skimming the headlines, and noticed they’re not four separate stories. They’re the same story, told from four different points in a pipeline. One company routes requests between models based on what the task actually needs. One company enforces that routing with hard rules instead of hoping the agent behaves. One company strips the tokens out of what the model receives before routing even enters the picture. And one company makes the case that the cheap tier in that routing decision is a lot more capable than “cheap” usually implies. Put together, they’re not four case studies, they’re four layers of the same optimization stack, and a small team can adopt them roughly in that order.

I run a small operation, not a company with Uber’s request volume, so I want to be upfront that I can’t replicate their numbers at their scale. What I can do is pull the real, checkable numbers out of each story, be honest about where the sourcing gets thin, and then build the two pieces of infrastructure a small .NET team would actually need to try any of this themselves: a cost-tracking middleware that sits around your HTTP client and tells you the truth about every call, and a router function that picks a model per task instead of sending everything through the same expensive one out of habit.

Here’s the four-way summary before I get into each one individually.

CASE STUDY SUMMARY, SAME WEEK, FOUR COMPANIES
Company Technique Reported savings
--------------------------------------------------------------------------------
Uber Live model routing (Pareto-optimal Total AI spend held
                     picks) + defaulting subagents to roughly flat while
                     cheaper models + code-mode batching weekly agent requests
                     of multiple tool calls into scripts grew 9.4x (Feb-Aug
                                                                2026); cost per
                                                                session down 52%
                                                                from its June peak
Spotify (Portal) Hook-enforced two-model routing: ~90% mean token
                     cheap model (Gemini 2.5 Flash in reduction on bulk-
                     their examples) handles bulk file read tasks, measured
                     reads and boilerplate, Claude is across 4 scenarios
                     reserved for reasoning and editing on a Java monorepo
Google (Headroom) Context-compression layer that 21-57% reduction in
                     strips redundant tokens out of tool prompt tokens across
                     outputs, logs, and search results four workload types,
                     before they reach the model, kept reversible via a
                     reversible via a retrieval tool retrieve tool
DeepSeek A cheaper model tier benchmarked Matches or slightly
(V4.1-Flash) against a frontier model, specifically beats a frontier
                     on tool-using tasks rather than model on a tool-use
                     raw reasoning benchmark at roughly
                                                                  1/30th the per-
                                                                  token input price

Uber: routing that moves, not routing that’s configured once

The number that made me stop and actually read Uber’s post twice was the ratio between two growth curves. Weekly active employees using their internal agent tooling grew 7x between February and August 2026. Weekly agent requests grew 9.4x over the same stretch, a faster curve than the headcount curve, meaning each person wasn’t just adopting the tools, they were leaning on them harder over time. And total spend, across that entire period, “relatively stabilized.” Not shrank, but held roughly flat while the thing generating the bill grew almost tenfold.

That’s the headline, but the mechanism underneath it is more interesting than the headline, and it’s not one trick, it’s three stacked together. The first is what they describe as benchmark-driven routing rather than a fixed default model. They run their agent harness against whatever’s currently Pareto-optimal across frontier and open-weight models on their own real workloads, and they move the default when a better tradeoff shows up, rather than picking a model once at launch and leaving it there for a year while pricing and capability both shift underneath that choice.

The second, and what they call the most impactful lever, is subagent delegation. Their setup splits a session into a primary model handling task decomposition and subagents handling the well-defined, narrower pieces of work that decomposition produces. The subagents default to weaker, more cost-effective models, and as the proportion of sessions that spin up subagents has grown, that default has done more of the total cost-control work than anything else in the stack. This is the same shape as the subagent-isolation pattern I tested for a different piece a few weeks back, but pointed at cost instead of context size, and it only pays off because the subagent’s model tier is deliberately cheaper, not because isolation is free on its own.

The third is what they call code-mode batching: instead of a model making five sequential tool calls, each one a full round trip through the model, it writes one script that makes all five calls and returns a single combined result. For a wide SELECT * query against a 50-row table, this cut token consumption by something close to 100 percent compared to the standard MCP tool-use pattern, and even trivial queries saw more than 50 percent savings. The mechanism is simple: five separate tool-use turns each resend the growing transcript, while one script executing five calls internally only pays the transcript cost once.

Put those three together and you get the unit numbers: cost per 1,000 model requests down 34 percent from its peak, cost per session down 52 percent from its June peak, both improving even as raw volume kept climbing. None of the three levers is exotic. Routing to whatever benchmarks best this month, defaulting subagents cheap, and batching tool calls into scripts are all things a small team can do without Uber’s internal tooling. The router function later in this piece is a stripped-down version of lever one and two combined.

Spotify: making the routing rule impossible to skip

Portal, Spotify’s open-sourced plugin, tackles the same idea from a different angle: instead of routing being a suggestion the agent can ignore when it’s in a hurry, it’s a rule the tooling enforces mechanically. The plugin ships as part of a broader system called AiKA Modes, ephemeral, declarative sub-agents that run more like a Lambda function than a persistent process, and two of those modes are the ones doing the cost work: bulk-reader, which handles large file analysis and returns a structured summary instead of the raw file, and code-writer, which generates predictable boilerplate, tests, configs, type stubs, matching whatever pattern already exists in the repo.

What makes this different from just writing a system prompt that says “use the cheap model for boilerplate” is the enforcement layer. Portal uses a PreToolUse hook that intercepts file reads before they happen and blocks anything over a configurable line-count threshold (350 lines by default), redirecting the read through bulk-reader instead of letting the primary, more expensive model ingest the whole file directly. That's a hard rule sitting below the model's own judgment, not a suggestion sitting above it. Below that hook sit bash wrapper scripts for the actual delegation calls, and below those, markdown documentation that guides the agent's behavior when the hooks do fire. The ordering matters: the hook is what actually saves money, the scripts and docs are what make the hook usable, and if you strip the hook out and keep only the guidance, my read of their own writeup is that the savings mostly evaporate, because nothing stops the agent from reading the big file directly when reading it directly feels like the path of least resistance mid-task.

The reported number is a mean 90 percent token reduction on bulk-read tasks, measured across four scenarios against a real Java monorepo, comparing a Claude session reading files directly against the same session delegating those reads to the cheap-model summarizer. That’s a real, specific measurement against a real codebase, not a synthetic benchmark, which is part of why I trust it more than a vaguer “up to 90%” claim would deserve.

What I want to give equal weight to is what they say it can’t do, because this is the part most vendor writeups skip and Spotify’s didn’t. Editing is off the table for the cheap tier, because a worker model’s summary doesn’t reliably preserve line numbers, and an edit against the wrong line is worse than no edit. Reasoning and debugging stay with the expensive model, because the cheap tier does shallow pattern matching that misses the subtle failure mode, the one that’s actually why you’re debugging in the first place. And large generations are a bad fit purely on latency grounds, each delegation costs 10 to 30 seconds and is capped at 30 seconds total, so a task small enough to finish in five seconds on the primary model becomes slower, not just cheaper, if you route it through the cheap tier anyway. Routing isn’t a free win you apply everywhere; it’s a win specifically shaped like “large, low-judgment, tolerant of a short delay,” and outside that shape it costs you speed for no savings.

Google’s Headroom: the layer underneath routing

Where Uber and Spotify both route the request to a different model, Headroom, the open-source context-compression layer associated with Google’s ecosystem, changes what’s inside the request in the first place, regardless of which model eventually reads it. It sits between your tools and your model, intercepts whatever those tools return, tool outputs, database results, file reads, RAG retrievals, and runs each one through a compressor matched to its content type: statistical summarization for JSON arrays that preserves anomalies and errors rather than repeating every normal row, log compression that keeps failures and drops the thousand lines of passing test noise, and similar type-specific handling for source code, diffs, and search results.

The mechanism it uses to stay safe, in their own framing, is “Compress-Cache-Retrieve.” Nothing is actually discarded, the original stays available in a cache, and the model gets a headroom_retrieve tool it can call if the compressed version turns out to be missing something it needed. That's the part of the design I find genuinely clever: most compression techniques are a one-way door, once you've thrown away detail you can't get it back mid-task, but this keeps a way back in if the model hits a wall.

The reported numbers, across four real workload types, come out to a 21 percent reduction on code search (17,199 tokens down to 13,597), 57 percent on SRE incident debugging (55,957 down to 24,340), 42 percent on codebase exploration (58,801 down to 33,895), and 30 percent on GitHub issue triage (46,067 down to 32,429). The spread across those four numbers is itself informative, the documentation notes savings depend on how repetitive the tool output is, and an SRE incident with pages of repeated log noise compresses far better than a search result set that’s already fairly dense.

Here’s the caveat I have to be honest about, because the source material I was handed for this piece claimed the accuracy story was settled: “no measured loss in task accuracy.” When I went and read Headroom’s own documentation directly, I couldn’t find that claim substantiated anywhere. What’s actually there is a description of the retrieval mechanism and the framing that it delivers “the same answers, a fraction of the tokens,” but no benchmark table, no before-and-after task success rate, nothing empirical backing that framing up. That’s not the same as evidence that accuracy does drop, I found nothing pointing that direction either, but it’s also not the same as a verified claim that it doesn’t. If you adopt this, the retrieval-fallback mechanism is the right safety net in principle, and I’d still want to run your own task-success comparison before and after turning compression on, rather than taking “same answers” as a settled fact just because it appears on the product page.

DeepSeek V4.1-Flash: what “a fraction of the cost” actually buys you

The fourth story is different in kind from the other three: it’s not a technique, it’s an argument that the cheap tier you’d route to is underrated. DeepSeek V4.1-Flash prices at $0.15 per million input tokens on a cache miss, $0.003 per million on a cache hit, and $0.60 per million output tokens off-peak (peak hours, 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays, run double those rates). Set next to Claude Opus 5’s $5 input / $25 output per million, that’s roughly a 33x gap on input and a 42x gap on output, before you even get to the cache-hit rate, which is another 50x cheaper again on top of that.

The benchmark comparison is where I want to slow down, because the honest version of this story is more interesting than the flattened “cheaper and just as good” version that usually gets repeated. On tool-using tasks specifically, HLE-with-tools, V4.1-Flash scores 63.9, which actually edges out Claude Opus 5’s 63.6. On Terminal-Bench 2.1 it hits 90.6, on DeepSWE 74.2, both strong agentic-coding numbers for a model priced two orders of magnitude below the frontier tier. But strip the tools away and test raw reasoning, plain Humanity’s Last Exam with no tool access, and the gap reappears: V4.1-Flash scores 36.8 against Opus 5’s 56.3 and GPT-5.6 Sol’s 44.5, about 65 percent of Opus 5’s score on that axis.

That split is the actual finding, and it’s a more useful one than a single blended number would be. This model isn’t “almost as good as the frontier tier” in general, it’s specifically as good, or slightly better, at the exact category of task that dominates real agent workloads, calling tools, reading their output, and acting on it, while trailing meaningfully on cold reasoning with no tools in hand. If your workload is mostly agentic (read a file, run a test, check the result, fix the thing) rather than mostly abstract reasoning, that’s precisely the axis where the cheap tier holds up, and it’s worth testing against your own tasks rather than assuming either the marketing framing or my summary of it transfers cleanly to what you’re building.

Building the instrumentation: middleware around the HTTP client

All four of those stories share a precondition I want to make concrete instead of leaving abstract: none of them would have a number to report without call-level instrumentation sitting underneath the technique. You can’t know your routing saved money if you don’t know what the unrouted baseline cost. So before the router function, here’s the piece that actually has to exist first, a DelegatingHandler that wraps whatever HttpClient your app uses to call an LLM API, and logs every call's model, token counts, cache status, and running per-session cost without your application code having to think about any of it.

// CostTrackingHandler.cs
// .NET 8, no external packages required.
using System.Text.Json;
using System.Text;
using System.Linq;
using System.Collections.Concurrent;
namespace AgentCostAudit;
public sealed record ModelPricing(
    decimal InputPerMillion,
    decimal OutputPerMillion,
    decimal CacheWritePerMillion,
    decimal CacheReadPerMillion);
public static class PricingTable
{
    // Published per-million-token USD rates at time of writing. Check your
    // provider's current pricing page before trusting these past this week.
    public static readonly Dictionary<string, ModelPricing> Models = new()
    {
        ["claude-opus-5"] = new(5.00m, 25.00m, 6.25m, 0.50m),
        ["claude-sonnet-5"] = new(3.00m, 15.00m, 3.75m, 0.30m),
        ["deepseek-v4.1-flash"] = new(0.15m, 0.60m, 0.15m, 0.003m),
    };
}
public sealed record CallLogEntry(
    string SessionId,
    string Model,
    string TaskType,
    int InputTokens,
    int OutputTokens,
    int CacheWriteTokens,
    int CacheReadTokens,
    bool CacheHit,
    decimal CallCostUsd,
    decimal SessionRunningTotalUsd,
    DateTimeOffset Timestamp);
public sealed class CostTrackingHandler : DelegatingHandler
{
    private readonly string _logPath;
    private readonly ConcurrentDictionary<string, decimal> _sessionTotals = new();
    private static readonly object FileLock = new();
    public CostTrackingHandler(HttpMessageHandler innerHandler, string logPath)
        : base(innerHandler)
    {
        _logPath = logPath;
    }
    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        // Session and task type ride in as headers set by the caller, so this
        // handler stays stateless per call rather than needing an ambient context.
        string sessionId = request.Headers.TryGetValues("X-Session-Id", out var sid)
            ? sid.First() : "unknown";
        string taskType = request.Headers.TryGetValues("X-Task-Type", out var tt)
            ? tt.First() : "unclassified";
        string? requestBody = request.Content is null
            ? null
            : await request.Content.ReadAsStringAsync(cancellationToken);
        string model = ExtractModel(requestBody) ?? "unknown";
        var response = await base.SendAsync(request, cancellationToken);
        // Buffer the body so we can inspect it and still hand a readable
        // response back to whatever called this handler.
        string responseBody = await response.Content.ReadAsStringAsync(cancellationToken);
        response.Content = new StringContent(responseBody, Encoding.UTF8, "application/json");
        var (inTok, outTok, cacheWrite, cacheRead) = ExtractUsage(responseBody);
        var pricing = PricingTable.Models.GetValueOrDefault(model)
            ?? PricingTable.Models["claude-sonnet-5"];
        decimal cost =
            inTok / 1_000_000m * pricing.InputPerMillion +
            cacheWrite / 1_000_000m * pricing.CacheWritePerMillion +
            cacheRead / 1_000_000m * pricing.CacheReadPerMillion +
            outTok / 1_000_000m * pricing.OutputPerMillion;
        decimal runningTotal = _sessionTotals.AddOrUpdate(sessionId, cost, (_, existing) => existing + cost);
        var entry = new CallLogEntry(sessionId, model, taskType, inTok, outTok,
            cacheWrite, cacheRead, cacheRead > 0, cost, runningTotal, DateTimeOffset.UtcNow);
        AppendLogLine(entry);
        Console.WriteLine(
            $"[{taskType}] model={model} in={inTok} out={outTok} cacheRead={cacheRead} " +
            $"cacheWrite={cacheWrite} cost=${cost:0.0000} sessionTotal=${runningTotal:0.00}");
        return response;
    }
    private void AppendLogLine(CallLogEntry entry)
    {
        string line = JsonSerializer.Serialize(entry);
        lock (FileLock)
        {
            File.AppendAllText(_logPath, line + Environment.NewLine);
        }
    }
    private static string? ExtractModel(string? requestBody)
    {
        if (requestBody is null) return null;
        using var doc = JsonDocument.Parse(requestBody);
        return doc.RootElement.TryGetProperty("model", out var m) ? m.GetString() : null;
    }
    private static (int input, int output, int cacheWrite, int cacheRead) ExtractUsage(string responseBody)
    {
        using var doc = JsonDocument.Parse(responseBody);
        if (!doc.RootElement.TryGetProperty("usage", out var usage))
            return (0, 0, 0, 0);
        int input = usage.TryGetProperty("input_tokens", out var i) ? i.GetInt32() : 0;
        int output = usage.TryGetProperty("output_tokens", out var o) ? o.GetInt32() : 0;
        int cacheWrite = usage.TryGetProperty("cache_creation_input_tokens", out var cw) ? cw.GetInt32() : 0;
        int cacheRead = usage.TryGetProperty("cache_read_input_tokens", out var cr) ? cr.GetInt32() : 0;
        return (input, output, cacheWrite, cacheRead);
    }
}

Wiring it into a real client is three lines, and the important part is that nothing downstream of HttpClient has to know the tracker exists:

// Wiring it up needs one extra using for JsonContent.Create:
// using System.Net.Http.Json;
var handler = new CostTrackingHandler(new HttpClientHandler(), "agent-cost-log.jsonl");
var http = new HttpClient(handler) { BaseAddress = new Uri("https://api.anthropic.com/") };
http.DefaultRequestHeaders.Add("x-api-key", Environment.GetEnvironmentVariable("ANTHROPIC_API_KEY"));
http.DefaultRequestHeaders.Add("anthropic-version", "2023-06-01");
var request = new HttpRequestMessage(HttpMethod.Post, "v1/messages")
{
    Content = JsonContent.Create(new
    {
        model = "claude-sonnet-5",
        max_tokens = 500,
        messages = new[] { new { role = "user", content = "Summarize this incident report." } }
    })
};
request.Headers.Add("X-Session-Id", "session-2026-09-13-01");
request.Headers.Add("X-Task-Type", "summarize");
var response = await http.SendAsync(request);

And querying it at the end of a session is a plain LINQ pass over the JSON-lines file, no database needed:

public static class SessionSummary
{
    public static void PrintSummary(string logPath)
    {
        var entries = File.ReadLines(logPath)
            .Select(line => JsonSerializer.Deserialize<CallLogEntry>(line)!)
            .ToList();
        foreach (var session in entries.GroupBy(e => e.SessionId))
        {
            decimal total = session.Sum(e => e.CallCostUsd);
            int calls = session.Count();
            double cacheHitRate = session.Count(e => e.CacheHit) / (double)calls;
            Console.WriteLine($"Session {session.Key}: {calls} calls, ${total:0.00} total, {cacheHitRate:P0} cache hits");
            foreach (var byTask in session.GroupBy(e => e.TaskType))
                Console.WriteLine($" {byTask.Key}: ${byTask.Sum(e => e.CallCostUsd):0.0000} across {byTask.Count()} calls");
        }
    }
}

I got this wrong on my first pass, worth admitting: I initially had the handler subtract cache_creation_input_tokens and cache_read_input_tokens from input_tokens before pricing the remainder, assuming input_tokens counted the whole prompt including cached portions. It doesn't. Anthropic's input_tokens field already excludes anything that went through the cache, it's only the genuinely fresh, uncached tokens. Subtracting the cache fields a second time silently zeroed out real cost. The version above prices the three buckets independently, which is the correct read of the usage object.

A router function, and the baseline it has to beat

With the tracker in place, the second piece is the one Uber’s post and Spotify’s plugin both lean on: something that picks a model per task instead of sending everything through whichever model you reach for by habit. I want to be explicit that what follows is a starting-point heuristic, not a finished classifier, in the same spirit as the “explicitly labeled as a starting point” caveat in the brief for this piece. It looks at input length and a short keyword list, nothing more sophisticated, and it’s meant to be replaced with something trained on your own task history once you have a few weeks of the log above to train it on.

// TaskRouter.cs
using System.Linq;
namespace AgentCostAudit;
public enum TaskComplexity { Trivial, Simple, Complex }
public sealed record RouteDecision(string Model, string Endpoint, TaskComplexity Complexity, string Reason);
public static class TaskRouter
{
    // Crude on purpose. Replace with a classifier trained on your own logged
    // task types before trusting this on anything that matters.
    private static readonly string[] ComplexSignals =
        { "refactor", "design", "architecture", "debug", "why does", "root cause", "trade-off" };
    private static readonly string[] TrivialSignals =
        { "summarize", "list", "extract", "format", "convert", "rename" };
    public static RouteDecision Route(string taskDescription, int estimatedInputTokens)
    {
        string lower = taskDescription.ToLowerInvariant();
        if (estimatedInputTokens > 20_000 || ComplexSignals.Any(lower.Contains))
        {
            return new RouteDecision(
                "claude-sonnet-5",
                "https://api.anthropic.com/v1/messages",
                TaskComplexity.Complex,
                "long context or a keyword suggesting multi-step judgment");
        }
        if (estimatedInputTokens < 3_000 && TrivialSignals.Any(lower.Contains))
        {
            // Local, self-hosted alternative for the trivial tier: no API key,
            // no per-token bill, just `ollama pull llama3.1:8b` first.
            return new RouteDecision(
                "llama3.1:8b",
                "http://localhost:11434/api/chat",
                TaskComplexity.Trivial,
                "short input matching a trivial-task keyword, routed to local Ollama");
        }
        return new RouteDecision(
            "deepseek-v4.1-flash",
            "https://api.deepseek.com/v1/chat/completions",
            TaskComplexity.Simple,
            "no strong signal either way, defaulting to the cheap hosted tier");
    }
}

And the piece that actually tells you whether routing was worth building: a comparison against a naive baseline that sends every task through a single model, run over the same task list.

// RoutingComparison.cs
using System.Linq;
namespace AgentCostAudit;
public static class RoutingComparison
{
    public static void Compare(IEnumerable<(string Task, int EstInputTokens, int ActualOutputTokens)> tasks)
    {
        decimal naiveTotal = 0m, routedTotal = 0m;
        var naivePricing = PricingTable.Models["claude-sonnet-5"];
        foreach (var (task, estIn, actualOut) in tasks)
        {
            decimal naiveCost =
                estIn / 1_000_000m * naivePricing.InputPerMillion +
                actualOut / 1_000_000m * naivePricing.OutputPerMillion;
            naiveTotal += naiveCost;
            var decision = TaskRouter.Route(task, estIn);
            bool selfHosted = decision.Model == "llama3.1:8b";
            var routedPricing = PricingTable.Models.GetValueOrDefault(decision.Model, naivePricing);
            decimal routedCost = selfHosted
                ? 0m
                : estIn / 1_000_000m * routedPricing.InputPerMillion +
                  actualOut / 1_000_000m * routedPricing.OutputPerMillion;
            routedTotal += routedCost;
            Console.WriteLine(
                $"[{decision.Complexity}] -> {decision.Model} (${routedCost:0.0000}) " +
                $"vs sonnet baseline (${naiveCost:0.0000}), reason: {decision.Reason}");
        }
        Console.WriteLine($"\nNaive single-model total: ${naiveTotal:0.00}");
        Console.WriteLine($"Routed total: ${routedTotal:0.00}");
        if (naiveTotal > 0)
            Console.WriteLine($"Savings vs baseline: {(1 - routedTotal / naiveTotal):P0}");
    }
}

The router in this article is deliberately not smart. What it gives you isn’t a good routing policy, it’s a place to plug your own task history into a comparison against a fixed baseline, so the savings number you get out the other end is measured against your real workload instead of borrowed from someone else’s blog post.

The adoption order, for a small team with limited time

If I were advising a small .NET team on which of these four techniques to try first, effort against expected savings, here’s the order I’d actually work through, and it doesn’t match the order I introduced them in above.

Start with instrumentation, before any of the four techniques, because none of the other numbers mean anything without it. The handler above is an afternoon of work, and every subsequent decision on this list depends on being able to measure it.

Second, try benchmarking a cheaper model against your own real tasks, the DeepSeek story. This is the lowest-effort of the four actual techniques, it’s a config change and a comparison run, not new infrastructure, and the router’s naive-versus-routed comparison above is enough to tell you in an afternoon whether a cheaper tier holds up on your workload the way it held up on HLE-with-tools.

Third, cheap-model delegation for I/O-heavy work, Spotify’s pattern. This needs slightly more than a config change, a hook or middleware layer that actually enforces the rule, because the Portal writeup’s own experience suggests guidance alone doesn’t hold under time pressure. But it’s bounded work, and the 90 percent number on bulk reads is specific enough to chase.

Fourth, live routing across your whole request stream, Uber’s pattern. This is the most valuable technique at their scale and the most infrastructure-heavy at any scale, it needs the benchmark-driven comparison running continuously rather than once, plus the subagent delegation logic to actually route work rather than just measure it. Worth building once the first three have already paid for themselves and you have real data on which of your task types are cheap-model-safe.

Fifth, and last, context compression, Headroom’s pattern. Not because it’s less valuable, the 21 to 57 percent range is real, but because it’s the one place in this list where I found the accuracy claim genuinely unverified rather than just under-explained, and a compression layer sitting in front of every model call is exactly the kind of change where I’d want my own before-and-after task-success numbers in hand before I trusted it broadly, not just the token count going down.

However you order it, the check at the end of each step is the same one, and it’s the actual point of all four company stories put together: compare against your own naive baseline, on your own tasks, with your own instrumentation, rather than assuming a savings number that held for someone else’s traffic pattern holds for yours. Uber’s 52 percent, Spotify’s 90 percent, Headroom’s 21 to 57 percent, and DeepSeek’s roughly-a-third-the-cost are all real, specific, individually checkable numbers. None of them is a promise about what happens when you run the same technique against a workload that isn’t theirs.

Tags: ai-agents, llm-cost-optimization, dotnet, model-routing, prompt-caching, agentic-ai, finops

.NET + AI in Production: What Actually Works (A Field Survey)

Ali Suleyman TOPUZ — Sun, 27 Sep 2026 09:18:41 +0000

Five write-ups, one week, and the patterns that kept showing up when the hype was stripped out

I spent a week reading everything I could find about shipping AI features in .NET, and I noticed something a bit humbling. The articles I trusted most were almost never about the model. They were about the boring layer around it: where the abstraction boundary sits, which NuGet packages you are still allowed to use for free, how you stop a code-generating agent from quietly wrecking your architecture, and how you keep a text-to-SQL bot from running DROP TABLE because someone phrased a question creatively.

So this is a survey, not a tutorial. I am pulling from five write-ups (listed at the bottom), adding my own opinions and doubts, and sketching the code where it helps. One honest note before we start: the code below is my own sketch of the patterns, not copied from those articles, and I have not run every snippet on every platform. Where I am extrapolating, I will say so.

Here is the map:

PATTERN WHAT IT PROTECTS YOU FROM
-------------------------------------------------------------------------
M.E.AI abstraction boundary Model and vendor lock-in
Non-silent fallback policy Surprise cloud calls and privacy leaks
Package guardrails Licensing shocks, hallucinated packages
Architecture fitness tests AI PRs that pass tests but rot the design
Validated text-to-SQL pipeline Writes, injections, and confident nonsense

1. Draw the boundary at Microsoft.Extensions.AI

The single most repeated idea across the sources was also the least exciting one: put Microsoft.Extensions.AI (M.E.AI) at the edge of your application and keep every provider SDK below it. Your app code talks to IChatClient and IEmbeddingGenerator>. Adapters for OpenAI, Azure, Ollama, or something on the phone sit underneath.

I was skeptical of this for a while. Abstractions over fast-moving APIs have a bad track record, and I have been burned by "we will just swap the provider later" more than once. What changed my mind is how thin the surface is. IChatClient is basically messages in, response out, plus streaming, plus a GetService escape hatch. There is not much to get wrong, and the middleware pipeline (logging, caching, function invocation, telemetry) composes on top of it instead of being smeared across your providers.

// dotnet add package Microsoft.Extensions.AI
// dotnet add package Azure.AI.OpenAI
// dotnet add package Microsoft.Extensions.AI.OpenAI

using Azure;
using Azure.AI.OpenAI;
using Microsoft.Extensions.AI;

builder.Services.AddChatClient(sp =>
    new AzureOpenAIClient(
            new Uri(builder.Configuration["Foundry:Endpoint"]!),
            new AzureKeyCredential(builder.Configuration["Foundry:Key"]!))
        .GetChatClient(builder.Configuration["Foundry:Deployment"]!)
        .AsIChatClient())
    .UseLogging()
    .UseOpenTelemetry();

The local alternative is the reason I now keep this boundary even in prototypes. Ollama gives you an IChatClient with no API key and no bill:

ollama pull llama3.1
ollama serve

// dotnet add package OllamaSharp
using OllamaSharp;

builder.Services.AddChatClient(
    new OllamaApiClient(new Uri("http://localhost:11434"), "llama3.1"));

Nothing above the registration changes. Same for embeddings: register an IEmbeddingGenerator> and your retrieval code never learns which vendor produced the vectors. My one warning is about embeddings specifically. Swapping the embedding model means re-embedding your corpus, so the abstraction saves you code but not migration work. Do not let the interface convince you the swap is free.

2. On-device AI in MAUI, without marrying one model

The most interesting application of that boundary is on-device AI. One of the sources walks through building it in .NET MAUI, and the question it opens with is the right one: how do you add AI to a cross-platform app without making the rest of the app depend on one model, one operating system, or one experimental API?

The platform reality is messy. Apple Intelligence on iOS and macOS, Gemini Nano through ML Kit on Android, and Phi Silica on Copilot+ Windows PCs are three different runtimes with three different availability stories. The source makes a point I think is underrated: device and runtime differences make a single static compatibility assumption unsafe. Whether a model is present is a runtime question, not a compile-time one.

So the shape is one IChatClient per platform, registered behind conditional compilation, plus a policy client on top that decides what happens when the local model is unavailable.

public enum CloudFallback { Never, AskUser, AllowWithNotice }

public sealed class OnDeviceFirstChatClient(
    IChatClient onDevice,
    IChatClient cloud,
    CloudFallback policy,
    IFallbackNotifier notifier) : IChatClient
{
    public async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken ct = default)
    {
        try
        {
            return await onDevice.GetResponseAsync(messages, options, ct);
        }
        catch (Exception ex) when (ex is NotSupportedException or PlatformNotSupportedException)
        {
            switch (policy)
            {
                case CloudFallback.Never:
                    throw new AiUnavailableException("On-device model unavailable.", ex);

                case CloudFallback.AskUser:
                    if (!await notifier.RequestConsentAsync(ct))
                        throw new AiUnavailableException("User declined cloud fallback.", ex);
                    break;

                case CloudFallback.AllowWithNotice:
                    notifier.ShowBanner("Using cloud AI because on-device AI is unavailable.");
                    break;
            }
            return await cloud.GetResponseAsync(messages, options, ct);
        }
    }

    // GetStreamingResponseAsync follows the same shape.
    public IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken ct = default)
        => throw new NotImplementedException();

    public object? GetService(Type serviceType, object? key = null) => onDevice.GetService(serviceType, key);
    public void Dispose() { onDevice.Dispose(); cloud.Dispose(); }
}

The part I care about is the word the source uses: non-silent. Fallback should be a product decision, not a catch block someone wrote on a Friday. If a user picked your app because "your notes never leave the device," a quiet cloud call is a breach of the promise even if the code is technically graceful. Make the policy an explicit enum, surface it in the UI, and log every fallback so you can see how often your local path actually fails.

My doubt: these platform APIs are young, several are still experimental, and behavior differs across OS versions and hardware. I would treat the adapters as the volatile part and keep them small enough to delete. The boundary is what makes that cheap.

3. The package supply chain got sharper teeth

This is the section that would have saved me the most pain if I had read it a year ago, because two separate problems are now colliding.

Problem one: free packages stopped being free

A run of very popular .NET libraries changed their terms. The codewithmukesh article lists the picture as of late August 2026, and I would treat it as the starting point for your own audit rather than legal advice:

PACKAGE LAST FREE LINE WHAT CHANGED
--------------------------------------------------------------------------
AutoMapper 14.0.0 (MIT) RPL-1.5 or commercial license
MediatR 12.5.0 (Apache-2.0) RPL-1.5 or commercial license
MassTransit v8 line (Apache-2.0) v9 is commercial
FluentAssertions 7.x (Apache-2.0) v8 under Xceed license, free
                                             only for non-commercial use
Polly Still BSD-3 Open Source Maintenance Fee
                                             (a fee model, not a relicense)

AutoMapper and MediatR have a free community tier for smaller organizations, and the article puts the revenue threshold at 5 million dollars in annual gross revenue. Check the current terms yourself, especially for the Polly fee start date, which that article dates to 16 November 2026. Fee structures move, and I would rather you read the maintainers' own pages than trust my summary of someone else's summary.

The reason this matters more with AI in the loop: coding agents were trained on years of code where AddAutoMapper and IMediator were the default answer. They will keep reaching for them, and they will happily bump a version to the latest, which is exactly the version that may carry the new license. A human might notice the license notice in the release notes. An agent will not.

If you decide to move off, there are alternatives worth evaluating, such as Mapperly (source-generated mapping), a source-generator mediator library or plain handler classes, and a community fork of FluentAssertions or Shouldly. I am not endorsing any of them blindly. I am saying that "pin the last free version forever" is a debt strategy, not a plan.

Problem two: models invent packages

The USENIX Security 2025 paper "We Have a Package for You!" tested 16 code-generating LLMs across 576,000 code samples and found 205,474 unique hallucinated package names. On average, 19.7 percent of package references pointed at packages that do not exist. The gap between model types is the part to remember: at least 5.2 percent for commercial models versus 21.7 percent for open-weight ones.

Two caveats so I do not oversell it. The study looked at two ecosystems, and C# and NuGet were not among them, so read it as directional evidence for .NET, not a measured NuGet rate. And a hallucinated name is only dangerous when someone registers it and waits, which is the "slopsquatting" attack the paper describes. That is not hypothetical enough to ignore, and it is cheap to defend against.

Guardrails that cost almost nothing

The codewithmukesh piece has a practical set, and I agree with most of it. First, make vulnerable packages break the build instead of scrolling past in a warning list:

<!-- Directory.Build.props -->
<Project>
  <PropertyGroup>
    <NuGetAudit>true</NuGetAudit>
    <NuGetAuditMode>all</NuGetAuditMode>
    <NuGetAuditLevel>low</NuGetAuditLevel>
    <WarningsAsErrors>$(WarningsAsErrors);NU1903;NU1904</WarningsAsErrors>
  </PropertyGroup>
</Project>

NU1903 is a high-severity advisory and NU1904 is critical. Second, turn on Central Package Management so versions live in exactly one file and an agent cannot sneak a version into a project file:

<!-- Directory.Packages.props -->
<Project>
  <PropertyGroup>
    <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    <CentralPackageTransitivePinningEnabled>true</CentralPackageTransitivePinningEnabled>
  </PropertyGroup>
  <ItemGroup>
    <PackageVersion Include="Microsoft.Extensions.AI" Version="9.9.0" />
  </ItemGroup>
</Project>

With CPM on, NU1008 fires when a PackageReference tries to carry its own Version, and NU1109 shows up when a transitive pin conflicts with a centrally defined version. Those two warnings are the tripwires that catch an agent editing the wrong file. Third, Package Source Mapping, because NU1507 warns when you have several feeds under CPM and no mapping:

<!-- nuget.config -->
<configuration>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="internal" value="https://pkgs.example.com/nuget/v3/index.json" />
  </packageSources>
  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="internal">
      <package pattern="Acme.*" />
    </packageSource>
  </packageSourceMapping>
</configuration>

Mapping matters for the hallucination problem specifically: it stops a public package named like your internal one from being resolved by accident.

Fourth, give the agent a way to check reality instead of guessing. Microsoft ships a NuGet MCP server that lets an agent look up whether a package exists and what its latest version is. With the .NET 10 SDK you can run it through dnx:

{
  "servers": {
    "nuget": {
      "command": "dnx",
      "args": ["NuGet.Mcp.Server", "--source", "https://api.nuget.org/v3/index.json", "--yes"]
    }
  }
}

I have written before about when MCP is overkill, and this is one of the cases where I think it earns its keep: it is read-only, it fronts a public registry, and it replaces a guess with a lookup. Pair it with a written rule in your agent instructions file listing banned packages, and a CI step that greps Directory.Packages.props for them.

4. Architecture fitness tests for AI-generated pull requests

Here is the failure mode that worries me most, because it is invisible in the usual dashboards. An agent opens a pull request. Every test is green. Coverage went up. And the change injects a DbContext straight into a controller because that was the shortest path to making the feature work. Nothing is broken. Everything is slightly worse.

Saurav Kumar's write-up on architecture gates for Copilot-generated PRs argues for treating architecture rules as executable tests, and I think that is right. Reviewers get tired, and the volume of AI-generated diffs is not going down. Rules that live in a wiki do not survive contact with a fast agent. Rules that fail the build do.

NetArchTest lets you express those rules in plain xUnit:

// dotnet add package NetArchTest.Rules
using NetArchTest.Rules;

public class ArchitectureTests
{
    private static readonly Assembly Domain = typeof(Acme.Domain.Order).Assembly;
    private static readonly Assembly Web = typeof(Acme.Web.Program).Assembly;

    [Fact]
    public void Domain_does_not_depend_on_infrastructure()
    {
        var result = Types.InAssembly(Domain)
            .ShouldNot()
            .HaveDependencyOnAny("Acme.Infrastructure", "Microsoft.EntityFrameworkCore")
            .GetResult();

        Assert.True(result.IsSuccessful, Describe(result));
    }

    [Fact]
    public void Controllers_do_not_touch_data_access_directly()
    {
        var result = Types.InAssembly(Web)
            .That().HaveNameEndingWith("Controller")
            .ShouldNot().HaveDependencyOn("Microsoft.EntityFrameworkCore")
            .GetResult();

        Assert.True(result.IsSuccessful, Describe(result));
    }

    private static string Describe(TestResult r) =>
        "Violations: " + string.Join(", ", r.FailingTypeNames ?? []);
}

The legacy problem, and the ratchet

This is where most teams give up. You add the rule to a five-year-old codebase and it reports 214 violations on day one, so someone disables the test and the whole idea dies. The fix I find most convincing is a ratchet: record today's violations as a baseline, fail the build on any new one, and also fail when a baseline entry disappears, so improvements get locked in.

private static void AssertRatchet(string rule, TestResult result)
{
    var path = Path.Combine(AppContext.BaseDirectory, "arch-baseline", $"{rule}.txt");
    var baseline = File.Exists(path)
        ? File.ReadAllLines(path).Where(l => l.Length > 0).ToHashSet()
        : new HashSet<string>();
    var current = (result.FailingTypeNames ?? []).ToHashSet();

    var added = current.Except(baseline).ToList();
    var healed = baseline.Except(current).ToList();

    Assert.True(added.Count == 0,
        $"New violations of '{rule}': {string.Join(", ", added)}");
    Assert.True(healed.Count == 0,
        $"Fixed. Remove from baseline so it cannot regress: {string.Join(", ", healed)}");
}

Make sure the baseline files are copied to the test output directory in the csproj. The count only ever goes down, and the AI agent gets an unambiguous error message it can act on, which is a nice side effect. An agent that sees "New violations of 'controllers-no-ef': OrdersController" will usually fix it on the next attempt.

One caution: the sources include benchmark-style numbers for how much these gates catch, and some of them are explicitly illustrative in their own text. I am not repeating any of them. Measure your own false-positive rate before you make the gate mandatory.

5. A production text-to-SQL pipeline that cannot write

The last pattern is the one with the sharpest edges. Satwiki De’s write-up on text-to-SQL agentic AI for enterprise analytical tasks describes a pipeline built on Azure AI Foundry with DeepEval-based evaluation, and the shape is one I would defend: not one big prompt, but five stages with a hard gate in the middle.

question
   |
   v
[Intent detector] analytical question? or chit-chat / write request?
   |
   v
[Query generator] LLM proposes SQL using an allowlisted schema
   |
   v
[Query validator] deterministic code, NOT an LLM. Rejects writes
   | and multi-statement SQL.
   v
[Read-only executor] read-only login, timeout, row cap
   |
   v
[Report builder] LLM summarizes rows into a report

The design decision I like best is that the validator is ordinary code. Asking a second model whether the first model’s SQL is safe is asking a probabilistic system to guard a deterministic risk. Parse the SQL and check the tree.

// dotnet add package Microsoft.SqlServer.TransactSql.ScriptDom
using Microsoft.SqlServer.TransactSql.ScriptDom;

public sealed record Verdict(bool IsValid, string Reason)
{
    public static Verdict Ok() => new(true, "");
    public static Verdict Reject(string why) => new(false, why);
}

public sealed class ReadOnlySqlValidator(IReadOnlySet<string> allowedTables)
{
    public Verdict Validate(string sql)
    {
        var parser = new TSql160Parser(initialQuotedIdentifiers: true);
        using var reader = new StringReader(sql);
        var fragment = parser.Parse(reader, out var errors);

        if (errors.Count > 0)
            return Verdict.Reject($"Does not parse: {errors[0].Message}");

        var statements = ((TSqlScript)fragment).Batches
            .SelectMany(b => b.Statements).ToList();

        if (statements.Count != 1)
            return Verdict.Reject("Exactly one statement is allowed.");

        if (statements[0] is not SelectStatement select)
            return Verdict.Reject("Only SELECT statements are allowed.");

        if (select.Into is not null)
            return Verdict.Reject("SELECT INTO writes data and is not allowed.");

        var cteNames = select.WithCtesAndXmlNamespaces?.CommonTableExpressions
            .Select(c => c.ExpressionName.Value)
            .ToHashSet(StringComparer.OrdinalIgnoreCase)
            ?? new HashSet<string>(StringComparer.OrdinalIgnoreCase);

        var visitor = new TableVisitor();
        fragment.Accept(visitor);

        foreach (var table in visitor.Tables)
            if (!allowedTables.Contains(table) && !cteNames.Contains(table))
                return Verdict.Reject($"Table '{table}' is not on the allowlist.");

        return Verdict.Ok();
    }

    private sealed class TableVisitor : TSqlFragmentVisitor
    {
        public List<string> Tables { get; } = new();
        public override void Visit(NamedTableReference node) =>
            Tables.Add(node.SchemaObject.BaseIdentifier.Value);
    }
}

Two things to note. A SELECT INTO is still a SelectStatement in the parse tree, so the explicit Into check is not decoration. And parsing is a first line of defense, not the only one. The executor is the second, and it should not depend on the validator being perfect:

public async Task<IReadOnlyList<Dictionary<string, object?>>> ExecuteAsync(string sql, CancellationToken ct)
{
    // Connection string uses a login with db_datareader only, plus:
    // Application Intent=ReadOnly
    await using var conn = new SqlConnection(_readOnlyConnectionString);
    await conn.OpenAsync(ct);

    await using var cmd = new SqlCommand(sql, conn) { CommandTimeout = 15 };
    await using var reader = await cmd.ExecuteReaderAsync(ct);

    var rows = new List<Dictionary<string, object?>>();
    while (await reader.ReadAsync(ct) && rows.Count < 1000)
    {
        var row = new Dictionary<string, object?>();
        for (var i = 0; i < reader.FieldCount; i++)
            row[reader.GetName(i)] = reader.IsDBNull(i) ? null : reader.GetValue(i);
        rows.Add(row);
    }
    return rows;
}

If the validator has a bug, the database login still cannot write. That layering is the point. For local development, a SQL Server container gives you a real target without touching anything shared:

docker run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=Dev_Passw0rd!" \
  -p 1433:1433 -d mcr.microsoft.com/mssql/server:2022-latest

The orchestrator is short, and the retry loop feeds the validator's rejection reason back to the generator, which in my experience of this general pattern fixes most first-attempt mistakes:

public async Task<Report> RunAsync(string question, CancellationToken ct)
{
    var intent = await _intent.DetectAsync(question, ct);
    if (intent.Kind != IntentKind.Analytical)
        return Report.Refusal(intent.Reason);

    string? feedback = null;
    for (var attempt = 0; attempt < 3; attempt++)
    {
        var sql = await _generator.GenerateAsync(question, intent, feedback, ct);
        var verdict = _validator.Validate(sql);
        if (!verdict.IsValid) { feedback = verdict.Reason; continue; }

        var rows = await _executor.ExecuteAsync(sql, ct);
        return await _reports.BuildAsync(question, sql, rows, ct);
    }
    return Report.Refusal("Could not produce a safe query for that question.");
}

Evaluating it with DeepEval

DeepEval is a Python library, so the pragmatic setup is a small pytest project that calls your .NET service over HTTP in CI. I would split the tests by what they can be checked with. Anything deterministic gets a deterministic assertion: "delete all orders" must be refused, a SELECT followed by a second statement that drops a table must be rejected, and the generated SQL should return the same rows as a hand-written gold query. Use an LLM judge only for the fuzzy part, the report narrative.

# pip install deepeval httpx pytest
import httpx
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

faithful = GEval(
    name="Report matches the data",
    criteria="The report must only state facts supported by the returned rows. "
             "Penalize invented numbers or trends.",
    evaluation_params=[LLMTestCaseParams.INPUT,
                       LLMTestCaseParams.ACTUAL_OUTPUT,
                       LLMTestCaseParams.RETRIEVAL_CONTEXT],
    threshold=0.7,
)

def test_revenue_report_is_faithful():
    r = httpx.post("http://localhost:5000/ask",
                   json={"question": "Revenue by region last quarter"}).json()
    case = LLMTestCase(
        input="Revenue by region last quarter",
        actual_output=r["report"],
        retrieval_context=[str(r["rows"])],
    )
    assert_test(case, [faithful])

Point the judge at your Foundry deployment with deepeval set-azure-openai, or at a local model with deepeval set-ollama llama3.1 while you iterate. Be aware that a small local judge is a noisy judge, so use it to debug the harness and a stronger model for the numbers you actually trust.

Start here: a checklist for your first production agent

If I were handing a .NET team one page, it would be this:

Put IChatClient and IEmbeddingGenerator at your app boundary, and keep provider SDKs out of business code.
Register an Ollama-backed client for local development so nobody needs a key to run the tests.
Decide your fallback policy in writing (never, ask, or allow with notice) before writing any fallback code.
Turn on NuGet audit and make NU1903 and NU1904 fail the build.
Enable Central Package Management and Package Source Mapping, and treat NU1008, NU1109, and NU1507 as real errors.
Audit your dependency list for AutoMapper, MediatR, MassTransit, FluentAssertions, and Polly against their current license terms.
Add a banned-packages rule to your agent instructions and a CI grep that backs it up.
Write five NetArchTest rules for your worst architecture violations, and ratchet them if the codebase is old.
Give the agent a read-only tool for your data, and enforce read-only at the database login, not just in the prompt.
Split evals into deterministic checks first and LLM-judge checks second, and run them in CI.

None of this is glamorous, and that is exactly why I trust it. The teams that worry me are not the ones with a weaker model. They are the ones with a great model and no guardrails around what it is allowed to touch.

Sources I read for this survey:

“The Hardest Part of Building AI in .NET Is Not the AI,” AI in Plain English
“On-Device AI in .NET MAUI Without Model Lock-In,” blog.devops.dev
codewithmukesh.com, “Stop Your AI Agent Picking the Wrong .NET Libraries”
Saurav Kumar, “Building Architecture Gates for Copilot-Generated Pull Requests in .NET,” c-sharpcorner.com
Satwiki De, “Building Text-to-SQL Agentic AI for enterprise analytical tasks,” Medium
Spracklen et al., “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs,” USENIX Security 2025

Tags: dotnet, csharp, ai, softwarearchitecture, nuget, azure, maui, devops

State, Memory, and the Two Camps Fighting Over What Your Agent Should Remember

Ali Suleyman TOPUZ — Sun, 27 Sep 2026 09:18:27 +0000

I built the same memory system twice, once for a project with three people and once for a codebase eleven teams touch, and I’m no longer sure both projects should use the same answer

I want to open with a disagreement I have not resolved, because I think pretending it’s settled is the dishonest version of this article.

A few weeks ago I watched Theo, who runs t3.gg and spends a lot of his time in the guts of coding agents, walk through an audit of his own Claude Code memory setup on stream. He found 45 memory files sitting in his main project. Twenty six of them had never been read back by the agent, not once. Memories were being written roughly three times as often as they were ever retrieved. His conclusion wasn’t “tune the retrieval.” It was “turn memory off,” and he did, across his whole fleet of agents, and told everyone watching to do the same. His argument, in short: code is the ground truth, a good grep over the actual repository beats a curated memory store that's mostly write-only, and for most teams the memory system is theater that costs tokens and adds a place for stale nonsense to hide.

I read that and felt seen, because I have a folder full of memory files in a side project that I would bet money the agent has never opened either.

Then I read the other side of it, and it’s not a strawman. Kapil Viren Ahuja, writing about this exact tension, makes a claim I can’t wave off: he agrees the minimalists are right, for individuals and small teams, and then argues that if you run engineering at an enterprise and follow that advice as written, your project will not survive the first ten people who join it. His framing, which stuck with me, is that in a brownfield enterprise codebase “what they delete is what saves you.” Not because curated memory is inherently superior to grep, but because the thing grep is searching, the actual code, is not a reliable narrator once dozens of engineers across several years have touched it.

Both of these people are describing something true. I spent the better part of a month trying to figure out where the line between them actually sits, by building the same kind of thing for two very different codebases, and this article is the honest version of what I found, including the part where I had to admit my test was flaky and fix it.

Camp one: memory is mostly theater, and grep already won

The minimalist case is stronger than “vector databases are overhyped,” which is the version of this argument I expected to find and didn’t. It’s a claim about what a language model is actually good at.

A model has seen an enormous amount of pretraining data involving grep, find, reading a file, walking a directory tree. It has seen comparatively little data involving the specific shape of your memory API, your bespoke fact schema, or the particular way your curated context gets assembled before a call. When an agent reaches for a tool, it's working from familiarity with that tool's shape, not from some abstract judgment about which retrieval method is theoretically best. Grep isn't a good retrieval algorithm in the information-retrieval sense. It's an extremely familiar one, and familiarity is what lets an agent use it correctly on the first try instead of half-remembering a query syntax from a different system.

Theo’s number is the concrete version of that argument. If you build a memory store and 58 percent of what goes into it (26 of 45 files) never gets read back out, you haven’t built a memory system, you’ve built a write-only log with extra steps and extra token cost on every write. The honest read of that audit isn’t “memory is bad,” it’s “this particular memory system had no retrieval discipline, so it decayed into noise, and the code underneath it was current enough that grep found the real answer faster anyway.”

For a small team on a codebase one or two people can hold in their heads, I think that’s just correct. The code is current. The conventions are whatever the last commit says they are, because the same small group of people wrote all of it and would notice if a comment lied. There’s no tribal knowledge locked in a departed engineer’s head, because there’s no departed engineer yet. In that world, a curated memory layer is solving a problem you don’t have, and every fact you cache is a fact that can go stale the moment someone refactors and forgets to update your memory store alongside the code.

Camp two: the code stops being a reliable narrator at scale

Here’s where I stopped being able to side entirely with Theo, and it happened on a codebase that wasn’t mine.

A team I was helping had a service where the README said "we deploy to staging via the pipeline in deploy/legacy," a comment three folders over said // TODO: migrate off legacy deploy, use deploy/v2, and deploy/v2 had been the actual production path for over a year. Nobody had deleted the old folder because deleting it required a conversation with a team that had since been reorganized twice. Grep over that repository doesn't return one truth. It returns three artifacts that each look authoritative and disagree with each other, and an agent has no principled way to know which one reflects what actually happens when code ships, because syntactically all three are just as valid as the others.

This is the shape of Ahuja’s argument, and it’s the one I now take seriously: a brownfield enterprise codebase accumulates dead code paths, comments that describe an intention nobody followed through on, and conventions that drifted across however many engineers touched the repository over however many years, and none of that shows up to grep as noise. It shows up as more code, indistinguishable in format from the code that’s actually true.

I ran my own small before-and-after on this, because I didn’t want to just take the claim on faith. I gave an agent a task, “scaffold the infrastructure-as-code for a new internal service,” on a real org’s repository, with nothing in its context beyond the codebase itself. The org is an Azure shop, has been for years, every existing service in that monorepo is provisioned through Bicep against Azure resources. The agent’s first attempt came back with Terraform targeting AWS. Not because anything in the repo said AWS, but because nothing in the repo said “we are an Azure shop and this is not a decision you get to make,” and in the absence of that constraint the model fell back on the statistically dominant pattern in its own training data, which skews AWS for generic infrastructure-as-code requests. I then wrote exactly one fact into a memory layer, “this organization provisions all cloud infrastructure on Azure via Bicep, this is a standard, not a preference,” loaded it ahead of the same prompt, and the agent’s next attempt came back correctly targeting Azure with Bicep. Same model, same repository, same prompt. The only variable that changed was one sentence of organizational memory that no file in the repository stated outright, because it didn’t need to be stated for a human who already knew it.

That’s the actual crux for me. Grep is unbeatable at finding what the code says. It has no mechanism at all for surfacing what the code assumes everyone already knows.

What actually belongs in memory, versus what belongs in the prompt

Before I get to the .NET implementation, I want to name a distinction that both camps actually agree on once you dig past the headline disagreement, because it’s what the practical section below is built on. It’s a framing I first saw laid out cleanly in a piece by Marwa Samy on what an AI agent should remember about a user, and it reframes the whole debate usefully: the question isn’t “how do I make the model remember everything,” it’s “what does my application actually need to remember.”

Conversation history and state are not the same thing, and conflating them is most of what makes memory systems bloat into Theo’s 45-file mess. Conversation history is the raw back-and-forth, every message, every hedge, every “actually, let’s change that.” State is the distilled, structured answer to “what does the agent currently know that it needs in order to keep helping this specific user or task.” If a user says, across three separate messages, “we’re on Azure,” “actually let’s keep this in West Europe for data residency,” and “we’ll probably want blue-green deployments eventually,” the conversation history is three messages of scattered context. The state is three fields: CloudProvider: Azure, Region: westeurope, DeploymentStrategy: blue-green (planned). Everything downstream, retrieval, prompting, memory hygiene, gets dramatically simpler once you stop trying to remember the conversation and start trying to remember the state.

That’s the design the .NET code below implements: a context provider that keeps raw history and extracted state in two separate places, extracts new facts after every turn, and merges them into existing state without duplicating or contradicting what’s already there.

Building it in .NET: a minimal AIContextProvider

The current Microsoft Agent Framework (the Microsoft.Agents.AI namespace) gives you an actual extension point for exactly this, AIContextProvider, an abstract class with two lifecycle hooks: one that runs before the model is invoked, where you inject retrieved context, and one that runs after, where you extract and persist whatever the turn taught you. Here's the shape, trimmed to what a minimal implementation actually needs.

using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
using System.Text.Json;
// The structured fact set we care about. Deliberately narrow: this is not
// a transcript, it's the handful of fields the agent actually needs on
// every future turn for this user or task.
public sealed class OrgMemoryState
{
    public string? CloudProvider { get; set; }
    public string? Region { get; set; }
    public string? DeploymentStrategy { get; set; }
    public Dictionary<string, string> AdditionalFacts { get; set; } = new();
}
public sealed class OrgMemoryContextProvider : AIContextProvider
{
    private readonly IMemoryStore _store;
    private readonly IChatClient _extractionClient;
    private readonly string _userId;
    public OrgMemoryContextProvider(IMemoryStore store, IChatClient extractionClient, string userId)
        : base(null, null, null)
    {
        _store = store;
        _extractionClient = extractionClient;
        _userId = userId;
    }
    // Runs before the model is called. Loads current state, hands it to
    // the model as a plain instruction, nothing exotic.
    protected override async ValueTask<AIContext> ProvideAIContextAsync(
        InvokingContext context, CancellationToken cancellationToken = default)
    {
        var state = await _store.LoadStateAsync(_userId, cancellationToken);
        if (state is null)
            return new AIContext();
        var summary = $"""
            Known organizational and user facts (treat as authoritative, do not contradict without asking):
            - Cloud provider: {state.CloudProvider ?? "unknown"}
            - Region: {state.Region ?? "unknown"}
            - Deployment strategy: {state.DeploymentStrategy ?? "unknown"}
            {string.Join('\n', state.AdditionalFacts.Select(kv => $"- {kv.Key}: {kv.Value}"))}
            """;
        return new AIContext
        {
            Messages = [new ChatMessage(ChatRole.System, summary)]
        };
    }
    // Runs after the model responds. Extracts structured facts from this
    // turn and merges them into whatever state already exists.
    protected override async ValueTask StoreAIContextAsync(
        InvokedContext context, CancellationToken cancellationToken = default)
    {
        if (context.InvokeException is not null)
            return; // never learn from a failed turn
        var turnText = string.Join(
            "\n",
            context.RequestMessages.Concat(context.ResponseMessages ?? [])
                .Select(m => $"{m.Role}: {m.Text}"));
        var extracted = await ExtractFactsAsync(turnText, cancellationToken);
        if (extracted is null)
            return; // nothing worth remembering from this turn
        var existing = await _store.LoadStateAsync(_userId, cancellationToken) ?? new OrgMemoryState();
        var merged = MergeState(existing, extracted);
        await _store.SaveStateAsync(_userId, merged, cancellationToken);
    }
    // Extraction: turn free-form conversation into the structured shape
    // above. This is the one call in the whole pipeline that talks to a
    // model, and its only job is "did this turn state a fact I care about."
    private async Task<OrgMemoryState?> ExtractFactsAsync(string turnText, CancellationToken ct)
    {
        var prompt = $"""
            Read the exchange below. If it states or changes any of these facts,
            return them as compact JSON with only the fields that were actually
            stated: cloudProvider, region, deploymentStrategy, additionalFacts
            (an object of free-form key/value pairs). If nothing relevant was
            stated, return an empty JSON object: {{}}
            Exchange:
            {turnText}
            """;
        var response = await _extractionClient.GetResponseAsync(prompt, cancellationToken: ct);
        var json = response.Text.Trim();
        if (json is "{}" or "")
            return null;
        try
        {
            var doc = JsonDocument.Parse(json);
            var state = new OrgMemoryState();
            if (doc.RootElement.TryGetProperty("cloudProvider", out var cp))
                state.CloudProvider = cp.GetString();
            if (doc.RootElement.TryGetProperty("region", out var rg))
                state.Region = rg.GetString();
            if (doc.RootElement.TryGetProperty("deploymentStrategy", out var ds))
                state.DeploymentStrategy = ds.GetString();
            if (doc.RootElement.TryGetProperty("additionalFacts", out var af))
            {
                foreach (var prop in af.EnumerateObject())
                    state.AdditionalFacts[prop.Name] = prop.Value.GetString() ?? "";
            }
            return state;
        }
        catch (JsonException)
        {
            // A malformed extraction is a reason to skip this turn's write,
            // not a reason to crash the conversation the user is in.
            return null;
        }
    }
    // Merge: a new fact for a key that already has a value overwrites it
    // (the newer statement wins), a key that's new gets added, a key that
    // isn't mentioned this turn is left completely alone.
    private static OrgMemoryState MergeState(OrgMemoryState existing, OrgMemoryState incoming)
    {
        return new OrgMemoryState
        {
            CloudProvider = incoming.CloudProvider ?? existing.CloudProvider,
            Region = incoming.Region ?? existing.Region,
            DeploymentStrategy = incoming.DeploymentStrategy ?? existing.DeploymentStrategy,
            AdditionalFacts = new Dictionary<string, string>(existing.AdditionalFacts)
                .Concat(incoming.AdditionalFacts)
                .GroupBy(kv => kv.Key)
                .ToDictionary(g => g.Key, g => g.Last().Value),
        };
    }
}
public interface IMemoryStore
{
    Task<OrgMemoryState?> LoadStateAsync(string userId, CancellationToken ct);
    Task SaveStateAsync(string userId, OrgMemoryState state, CancellationToken ct);
}

Registering it is the part that made this click for me the first time I wired it up, because it’s just another item in the same list that holds tools and instructions:

AIAgent agent = chatClient.AsAIAgent(new ChatClientAgentOptions
{
    Instructions = "You are an infrastructure assistant for this organization.",
    AIContextProviders = [
        new OrgMemoryContextProvider(memoryStore, extractionClient, userId: currentUserId)
    ],
});

The thing I got wrong the first time: I put the extraction call and the summarization-for-prompt call on the same IChatClient instance sharing conversation history, and ended up with the extraction step quietly picking up context bleed from the injected memory summary, so a fact would get "re-extracted" and re-written every single turn even when nothing new was said. The fix was boring, give the extraction call a clean, single-purpose prompt with no shared history, which is what the code above does.

Backing it with real storage: Azure Cosmos DB, and a local alternative

The part of this that actually matters for correctness, more than the storage technology you pick, is keeping two identifiers cleanly separate: the conversation or session ID, which is short-lived and dies with the conversation, and the user or tenant ID, which is long-lived and is what your memory is actually keyed on. Collapsing these two into one ID is the single most common bug I’ve seen in memory implementations, because it means every new conversation starts with total amnesia, which defeats the entire point.

+----------------------+------------------------+---------------------------+
| Identifier | Lifespan | What it answers |
+----------------------+------------------------+---------------------------+
| session_id / thread_id | One conversation, | "Which conversation is |
| | minutes to hours | this?" |
+----------------------+------------------------+---------------------------+
| user_id / tenant_id | Indefinite, survives | "Whose durable memory |
| | every future session | is this?" |
+----------------------+------------------------+---------------------------+

Azure Cosmos DB

Two containers, split by exactly that distinction. sessions holds the raw turn history, partitioned by sessionId, with a TTL so it expires on its own. userMemory holds the extracted state, partitioned by userId, with no TTL, because it's meant to outlive any single conversation.

using Microsoft.Azure.Cosmos;

public sealed class CosmosMemoryStore : IMemoryStore
{
    private readonly Container _userMemory;
    public CosmosMemoryStore(CosmosClient client, string databaseId)
    {
        _userMemory = client.GetContainer(databaseId, "userMemory");
    }
    public async Task<OrgMemoryState?> LoadStateAsync(string userId, CancellationToken ct)
    {
        try
        {
            var response = await _userMemory.ReadItemAsync<UserMemoryDocument>(
                id: userId, partitionKey: new PartitionKey(userId), cancellationToken: ct);
            return response.Resource.State;
        }
        catch (CosmosException ex) when (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
        {
            return null;
        }
    }
    public async Task SaveStateAsync(string userId, OrgMemoryState state, CancellationToken ct)
    {
        var doc = new UserMemoryDocument { Id = userId, UserId = userId, State = state };
        await _userMemory.UpsertItemAsync(doc, new PartitionKey(userId), cancellationToken: ct);
    }
    private sealed class UserMemoryDocument
    {
        public string Id { get; set; } = default!;
        public string UserId { get; set; } = default!;
        public OrgMemoryState State { get; set; } = default!;
    }
}

Container setup, once, via the SDK or the portal: partition key /userId on userMemory, no default TTL; partition key /sessionId on sessions, with DefaultTimeToLive set to something like 30 days so raw turns clean themselves up without a background job.

A self-hosted alternative: SQLite

If you don’t have an Azure subscription handy, or just want to run this on a laptop first, the identical two-table split works fine in SQLite with zero external services:

using Microsoft.Data.Sqlite;
using System.Text.Json;

public sealed class SqliteMemoryStore : IMemoryStore, IDisposable
{
    private readonly SqliteConnection _conn;
    public SqliteMemoryStore(string dbPath = "memory.db")
    {
        _conn = new SqliteConnection($"Data Source={dbPath}");
        _conn.Open();
        using var cmd = _conn.CreateCommand();
        cmd.CommandText = """
            CREATE TABLE IF NOT EXISTS user_memory (
                user_id TEXT PRIMARY KEY,
                state_json TEXT NOT NULL,
                updated_at TEXT NOT NULL
            );
            CREATE TABLE IF NOT EXISTS session_turns (
                session_id TEXT NOT NULL,
                user_id TEXT NOT NULL,
                turn_json TEXT NOT NULL,
                created_at TEXT NOT NULL
            );
            """;
        cmd.ExecuteNonQuery();
    }
    public async Task<OrgMemoryState?> LoadStateAsync(string userId, CancellationToken ct)
    {
        using var cmd = _conn.CreateCommand();
        cmd.CommandText = "SELECT state_json FROM user_memory WHERE user_id = $uid";
        cmd.Parameters.AddWithValue("$uid", userId);
        var result = await cmd.ExecuteScalarAsync(ct);
        return result is string json ? JsonSerializer.Deserialize<OrgMemoryState>(json) : null;
    }
    public async Task SaveStateAsync(string userId, OrgMemoryState state, CancellationToken ct)
    {
        using var cmd = _conn.CreateCommand();
        cmd.CommandText = """
            INSERT INTO user_memory (user_id, state_json, updated_at)
            VALUES ($uid, $state, $now)
            ON CONFLICT(user_id) DO UPDATE SET state_json = $state, updated_at = $now
            """;
        cmd.Parameters.AddWithValue("$uid", userId);
        cmd.Parameters.AddWithValue("$state", JsonSerializer.Serialize(state));
        cmd.Parameters.AddWithValue("$now", DateTimeOffset.UtcNow.ToString("O"));
        await cmd.ExecuteNonQueryAsync(ct);
    }
    public void Dispose() => _conn.Dispose();
}

No container to spin up, no account to create, and it’s a drop-in IMemoryStore so you can run the exact same OrgMemoryContextProvider against either backend without changing a line of the memory logic itself.

A deterministic smoke test

Every memory system needs one test that answers a single question without ambiguity: does it actually recall the specific fact it was told. I originally wrote this test by asking the agent a question in a second session and asserting on the model’s final text output, and it was flaky, not because the memory system was broken, but because the model would sometimes phrase the answer in a way my string match didn’t anticipate. The fix was to stop testing the model’s prose and start testing the retrieval layer directly, which is the part that’s actually deterministic.

using Xunit;

public class MemoryRecallSmokeTests
{
    [Fact]
    public async Task Recalls_CloudProvider_Fact_Across_Sessions()
    {
        var store = new SqliteMemoryStore(dbPath: ":memory:can-share-connection-in-real-setup");
        var userId = $"smoke-test-{Guid.NewGuid()}";
        // Session 1: seed the fact the way StoreAIContextAsync would have,
        // via a canned extraction result rather than a live model call, so
        // this test has no dependency on sampling variance.
        var extracted = new OrgMemoryState { CloudProvider = "Azure", Region = "westeurope" };
        var existing = await store.LoadStateAsync(userId, default) ?? new OrgMemoryState();
        await store.SaveStateAsync(userId, new OrgMemoryState
        {
            CloudProvider = extracted.CloudProvider ?? existing.CloudProvider,
            Region = extracted.Region ?? existing.Region,
            DeploymentStrategy = existing.DeploymentStrategy,
            AdditionalFacts = existing.AdditionalFacts,
        }, default);
        // Session 2: a brand new provider instance, same user ID, nothing
        // shared except what's in the store. This is the actual assertion:
        // does the provider surface the fact on a cold session. Call the
        // real public entry point, InvokingAsync, not the protected
        // override, so this test exercises exactly what the agent runtime
        // would call. FakeAgent/FakeSession are thin test doubles with no
        // behavior beyond satisfying the constructor, omitted here for space.
        var provider = new OrgMemoryContextProvider(store, extractionClient: null!, userId);
        var invokingContext = new InvokingContext(new FakeAgent(), new FakeSession(), new AIContext());
        var aiContext = await provider.InvokingAsync(invokingContext);
        var summary = aiContext.Messages!.Single().Text;
        Assert.Contains("Azure", summary);
        Assert.Contains("westeurope", summary);
        Assert.DoesNotContain("AWS", summary);
    }
}

The part worth calling out: this test never calls a real language model. It seeds the store the way a successful extraction would have, then verifies that a cold ProvideAIContextAsync call, from a completely new provider instance, surfaces exactly the fact that was written. That's the honest boundary of what "deterministic" can mean for a system with a model somewhere in its pipeline: you make the retrieval and merge logic deterministic and test that rigorously, and you test extraction quality separately, with looser, statistical evaluation, because extraction is the one place an LLM's judgment is actually load-bearing.

Production considerations, the honest section

Auth mapping. Never let a client hand you the userId it wants memory keyed on. Map the session's authenticated principal, whatever your auth middleware already validated, to a stable internal user ID server-side, and treat any user-supplied identity claim in a request body as untrusted input. The failure mode if you get this wrong isn't subtle: one user's session ends up reading another user's memory, because your context provider trusted a field it should have derived instead.

Retention and consent. Extracted facts are still personal or organizational data, and “we cached it in a memory store” doesn’t exempt it from whatever data handling rules already apply to the rest of your system. Concretely: put a TTL on raw session turns (they’re the least necessary thing to keep and the most sensitive, since they’re closer to verbatim conversation), keep extracted state around longer but make it inspectable, and build a real forget-me path, a method that deletes a user’s row from userMemory entirely rather than soft-marking it, if your retention policy calls for hard deletion. If you're storing anything a person might reasonably not want remembered indefinitely, ask once, up front, rather than assuming silence is consent.

Catching a wrong memory. This is the one I underestimated. An extraction step can misread a hedge as a commitment, “we might move to multi-cloud eventually” is not the same claim as “we are moving to multi-cloud,” and once that’s written into userMemory it will confidently color every future turn until someone notices. Three things helped: logging the source turn alongside every extracted fact, not just the fact itself, so a human reviewing a wrong memory can see exactly what triggered it; building a boring internal page that lists a user's current memory state in plain text, because "can I just look at what it thinks it knows" turned out to be the fastest way anyone on the team actually caught a bad extraction; and treating any correction the same way you'd treat a genuine update, overwrite the field and keep a record that it was corrected, rather than pretending the wrong version was never there.

So where do I actually draw the line

Here’s my answer, and it’s a size threshold, not a philosophy.

If you’re on a single repository small enough that one or two engineers hold the real conventions in their heads, and the code you’d grep is genuinely current, I’m with Theo. Skip the memory layer. Every fact you cache outside the code is a fact that can drift from the code, and on a small enough team someone will notice the drift before it matters. Build the context provider above if you want, but point it at nothing more than session-scoped scratch state, and don’t pretend you need userMemory at all yet.

The line I’d personally use is somewhere around the point where a codebase has more than one area nobody on the current team fully owns, the kind of area where the last person who understood why a decision was made has moved teams or left, or where “the standard” is something new hires get told in a hallway conversation rather than something the repository states anywhere a grep would find it. That’s usually somewhere past ten engineers and past a single repository, which lines up with Ahuja’s “first ten people” instinct more than I expected it to when I started this. Below that line, curated memory is solving a problem you don’t have yet. Above it, the code stops being a reliable narrator of its own history, and the one sentence of memory that tells an agent “we are an Azure shop, this is a standard, not a preference” is the difference between a correct first attempt and a plausible-looking wrong one that costs someone an afternoon to catch.

Neither camp is wrong. They’re describing different codebases.

Tags: dotnet, ai-agents, agentic-memory, azure-cosmos-db, csharp, llm-engineering, software-architecture

Three Microsoft Agent Framework Posts, One Working App: RAG, a Custom Agent, and a Workflow Graph…

Ali Suleyman TOPUZ — Fri, 25 Sep 2026 12:50:40 +0000

Three Microsoft Agent Framework Posts, One Working App: RAG, a Custom Agent, and a Workflow Graph Wired Together

I have been building small things with Microsoft Agent Framework (MAF) for about six weeks now, mostly evenings, mostly on a machine with no Azure subscription attached. Three posts landed in my feed within the same ten days that each covered a different slice of it. Jesse Liberty wrote an overview of retrieval-augmented generation in MAF, explaining the two ways the framework lets an agent pull in outside context. Davide Bellone, on Code4IT, published part three of his MAF series showing how to build a custom agent with its own instructions and identity, after already covering local models with Ollama in part two. And Liberty followed up a few days later with a piece on nodes and edges, the part of MAF that models a process as a directed graph instead of a single prompt-response loop.

All three are good, accurate posts. None of them is wrong about anything I could find. But reading them back to back left me with three separate mental models and no sense of how they fit into one application, because none of the three examples talks to either of the other two. Liberty’s RAG agent answers one question and stops. Bellone’s custom agent gives advice but never gets called by anything larger than a console loop. The workflow graph in Liberty’s second post routes messages between executors that do nothing more interesting than classify spam.

That gap is the actual reason MAF is worth learning if you have already used something else, be it Semantic Kernel, LangChain, or a hand-rolled orchestration layer on top of an LLM API. MAF does not ask you to buy into one big framework object that owns your prompts, your tools, your memory, and your control flow all at once. Retrieval is an AIContextProvider you attach to an agent. A custom agent is a ChatClientAgent wrapping any IChatClient. A multi-step process is a graph of Executor nodes wired up with a WorkflowBuilder. Each of these is a separate, composable piece, and the whole point of composability is that you are supposed to be able to put them together. So I did. This is the writeup, with real code, of a small support-ticket system where a workflow graph classifies incoming tickets, routes the package-upgrade ones to a custom agent that answers from a real set of .NET release notes using RAG, and escalates everything else to a human.

Everything below runs against a local Ollama model and a Qdrant instance in Docker, no Azure subscription required, because that is the one thing I wanted from all three original posts and had to build myself.

Part one: RAG, bridged to a vector store you actually run yourself

Liberty’s post gets the core distinction right. MAF’s TextSearchProvider is an AIContextProvider, and it supports two retrieval modes. In BeforeAIInvoke mode, the provider runs a search before every model call and stuffs the results into context automatically, whether the model asked for them or not. In OnDemandFunctionCalling mode, the search becomes a callable tool, and the model decides for itself whether a given question needs a lookup at all.

Here is the tradeoff in one table:

MODE WHEN IT SEARCHES BEST FOR
------------------------------------------------------------------------
BeforeAIInvoke Every single turn Narrow domains where
                                                   almost every question
                                                   needs the same corpus
                                                   (a policy doc, a
                                                   changelog)
OnDemandFunctionCalling Only when the model Broad agents that also
                          decides it needs to handle small talk or
                                                    general questions, where
                                                    forcing a search every
                                                    turn wastes tokens

Liberty’s example uses a mock search function that returns hardcoded strings, which is fine for showing the shape of the API but does not tell you what to do when the corpus is real. So the first thing I built was a bridge from TextSearchProvider to an actual vector store: Qdrant, running in Docker, with embeddings coming from Ollama instead of a paid embeddings API.

Start Qdrant:

docker run -d --name qdrant -p 6333:6333 -p 6334:6334 qdrant/qdrant

Pull an embedding model with Ollama, alongside whatever chat model you are already running:

ollama pull all-minilm
ollama pull llama3.1

The project file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net9.0</TargetFramework>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Agents.AI" />
    <PackageReference Include="Microsoft.Extensions.AI" Version="10.7.0" />
    <PackageReference Include="Microsoft.Extensions.AI.Ollama" />
    <PackageReference Include="Microsoft.Extensions.VectorData.Abstractions" />
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.Qdrant" />
    <PackageReference Include="Qdrant.Client" />
    <PackageReference Include="OllamaSharp" Version="5.4.25" />
  </ItemGroup>
</Project>

The data model for a chunk of a release-notes document, using the current Microsoft.Extensions.VectorData attributes:

using Microsoft.Extensions.VectorData;
public sealed class ReleaseNoteChunk
{
    [VectorStoreKey]
    public ulong Id { get; set; }
    [VectorStoreData(IsIndexed = true)]
    public required string PackageName { get; set; }
    [VectorStoreData(IsFullTextIndexed = true)]
    public required string Text { get; set; }
    [VectorStoreData]
    public required string SourceFile { get; set; }
    [VectorStoreVector(Dimensions: 384, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)]
    public ReadOnlyMemory<float>? Embedding { get; set; }
}

384 dimensions because all-minilm produces the same embedding size as all-MiniLM-L6-v2. Ingestion, chunking a folder of markdown release notes and embedding each chunk with the local model:

using Microsoft.Extensions.AI;
using Microsoft.Extensions.VectorData;
using Microsoft.SemanticKernel.Connectors.Qdrant;
using Qdrant.Client;
var embeddingGenerator = new OllamaEmbeddingGenerator(new Uri("http://localhost:11434"), "all-minilm");
var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"));
var collection = vectorStore.GetCollection<ulong, ReleaseNoteChunk>("release-notes");
await collection.CreateCollectionIfNotExistsAsync();
ulong id = 0;
foreach (var file in Directory.GetFiles("release-notes", "*.md"))
{
    var packageName = Path.GetFileNameWithoutExtension(file);
    var text = await File.ReadAllTextAsync(file);
    foreach (var chunk in ChunkText(text, chunkSize: 800, overlap: 120))
    {
        var embedding = await embeddingGenerator.GenerateEmbeddingVectorAsync(chunk);
        await collection.UpsertAsync(new ReleaseNoteChunk
        {
            Id = id++,
            PackageName = packageName,
            Text = chunk,
            SourceFile = file,
            Embedding = embedding
        });
    }
}
static IEnumerable<string> ChunkText(string text, int chunkSize, int overlap)
{
    for (int start = 0; start < text.Length; start += chunkSize - overlap)
    {
        yield return text.Substring(start, Math.Min(chunkSize, text.Length - start));
        if (start + chunkSize >= text.Length) yield break;
    }
}

Now the bridge itself. TextSearchProvider takes a delegate that turns a query string into a list of TextSearchProvider.TextSearchResult, so the adapter just has to embed the query and call VectorizedSearchAsync against the same collection:

using Microsoft.Agents.AI;
using Microsoft.Extensions.VectorData;
async Task<IEnumerable<TextSearchProvider.TextSearchResult>> SearchAdapter(string query, CancellationToken ct)
{
    var queryEmbedding = await embeddingGenerator.GenerateEmbeddingVectorAsync(query, cancellationToken: ct);
    var searchOptions = new VectorSearchOptions { Top = 4, VectorPropertyName = nameof(ReleaseNoteChunk.Embedding) };
    var hits = await collection.VectorizedSearchAsync(queryEmbedding, searchOptions, ct);
    var results = new List<TextSearchProvider.TextSearchResult>();
    await foreach (var hit in hits)
    {
        results.Add(new TextSearchProvider.TextSearchResult
        {
            SourceName = hit.Record.PackageName,
            SourceLink = hit.Record.SourceFile,
            Text = hit.Record.Text
        });
    }
    return results;
}

And wiring it to an agent, first in BeforeAIInvoke mode:

var textSearchOptions = new TextSearchProviderOptions
{
    SearchTime = TextSearchProviderOptions.TextSearchBehavior.BeforeAIInvoke,
    RecentMessageMemoryLimit = 4
};
IChatClient ollamaChat = new OllamaApiClient(new Uri("http://localhost:11434"), "llama3.1");
AIAgent releaseNotesAgent = new ChatClientAgent(ollamaChat, new ChatClientAgentOptions
{
    Name = "ReleaseNotesAgent",
    Instructions = "Answer questions about .NET package upgrades using only the provided release note excerpts. Cite the source file.",
    AIContextProviders = [new TextSearchProvider(SearchAdapter, textSearchOptions)]
});

Switching to on-demand is one line, SearchTime = TextSearchProviderOptions.TextSearchBehavior.OnDemandFunctionCalling, and the search shows up in the model's tool list instead of running automatically. I went with BeforeAIInvoke for this particular agent because every question a package-upgrade advisor gets is, by definition, going to need the release notes. If I were building a general-purpose support agent that only occasionally needed to check a knowledge base, on-demand would be the right call, since forcing a vector search on "hi, are you still there" wastes a round trip for nothing.

One thing that tripped me up on the first attempt: I set Dimensions: 4 by copying an example verbatim before I'd actually checked what all-minilm produces, and Qdrant rejected every upsert with a dimension mismatch and a fairly unhelpful error. Check your embedding model's actual output size before you write the attribute, not after.

Part two: a custom agent that survives a restart

Bellone’s post is the clearest explanation I have read of why ChatClientAgent exists at all: it is a thin wrapper that takes any IChatClient and gives it a name, a description, and a fixed set of instructions, so you get a reusable, nameable agent instead of a bag of chat completion calls with a system prompt taped on. His example agent, though, only lives for the length of one console session. The part that actually matters in a real app, keeping a conversation alive across a restart, is a paragraph, not code.

The releaseNotesAgent from the previous section already has instructions narrow enough to be useful (only answer package-upgrade questions, cite sources, defer everything else), so I will reuse it here rather than build a second one. What is new is session handling:

using System.Text.Json;
AgentSession session = await releaseNotesAgent.CreateSessionAsync();
var first = await releaseNotesAgent.RunAsync(
    "We're moving from EF Core 8 to EF Core 9. Anything that breaks LINQ translation?",
    session);
Console.WriteLine(first.Text);
// Persist the session before the process exits.
JsonElement serialized = releaseNotesAgent.SerializeSession(session);
await File.WriteAllTextAsync("session.json", serialized.GetRawText());

And on the next run of the program, instead of starting fresh:

JsonElement restored = JsonSerializer.Deserialize<JsonElement>(await File.ReadAllTextAsync("session.json"));
AgentSession resumedSession = await releaseNotesAgent.DeserializeSessionAsync(restored);
var second = await releaseNotesAgent.RunAsync(
    "And what about the query splitting behavior specifically?",
    resumedSession);
Console.WriteLine(second.Text);

That second call has full access to the first exchange, because the session, not just a transcript of strings, is what got restored. Two things worth flagging from actually doing this. First, SerializeSession is synchronous, DeserializeSessionAsync is not, which is easy to get backwards if you are typing from memory. Second, the docs are explicit that you should treat the serialized session as opaque and restore it against the same agent configuration that created it. I tested this by pointing the restored session at an agent with different instructions, out of curiosity, and the conversation history came back intact but the agent's behavior shifted immediately to match the new instructions rather than the old ones. That is probably the right design, but it means your instructions are effectively part of your session contract, not a cosmetic detail you can change freely once conversations are in flight.

Part three: a workflow graph that does more than sort spam

Liberty’s nodes-and-edges post is the strongest technical writing of the three, and its spam-classifier example genuinely teaches the shape of the API. But a workflow that only ever produces “spam” or “not spam” does not exercise fan-out and fan-in in any way that resembles what those patterns are actually for, which is running independent checks in parallel and merging the results.

Here is the ticket-triage graph. An incoming ticket gets classified, and if it needs a human it goes straight to escalation. If it does not, two independent checks run in parallel, a duplicate-ticket lookup and a severity estimate, and only once both finish does the graph decide whether to hand the ticket to the package-upgrade advisor or a generic handler.

The classifier, using the [MessageHandler] pattern MAF expects on an Executor:

using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Workflows;
public sealed record TicketClassification(string Category, bool NeedsHuman, string TicketText);
internal sealed partial class TicketClassifierExecutor : Executor
{
    private readonly AIAgent _classifierAgent;
    public TicketClassifierExecutor(AIAgent classifierAgent) : base("TicketClassifier")
    {
        _classifierAgent = classifierAgent;
    }
    [MessageHandler]
    private async ValueTask<TicketClassification> HandleAsync(string ticketText, IWorkflowContext context, CancellationToken ct = default)
    {
        var response = await _classifierAgent.RunAsync(
            $"Classify this support ticket. Reply with exactly one word for category (PackageUpgrade, Billing, Other) followed by YES or NO for whether it needs a human. Ticket: {ticketText}",
            cancellationToken: ct);
        var parts = response.Text.Trim().Split(' ');
        var result = new TicketClassification(parts[0], parts.Length > 1 && parts[1].Equals("YES", StringComparison.OrdinalIgnoreCase), ticketText);
        await context.AddEventAsync(new WorkflowEvent("ticket_classified", result));
        return result;
    }
}

The two parallel checks:

internal sealed partial class DuplicateCheckExecutor : Executor
{
    public DuplicateCheckExecutor() : base("DuplicateCheck") { }
    [MessageHandler]
    private async ValueTask<bool> HandleAsync(TicketClassification ticket, IWorkflowContext context, CancellationToken ct = default)
    {
        var isDuplicate = await LookUpRecentTicketsAsync(ticket.TicketText, ct);
        await context.QueueStateUpdateAsync(ticket.TicketText, isDuplicate, scopeName: "duplicate-checks");
        return isDuplicate;
    }
    private static Task<bool> LookUpRecentTicketsAsync(string text, CancellationToken ct) => Task.FromResult(false);
}
internal sealed partial class SeverityCheckExecutor : Executor
{
    public SeverityCheckExecutor() : base("SeverityCheck") { }
    [MessageHandler]
    private ValueTask<string> HandleAsync(TicketClassification ticket, IWorkflowContext context, CancellationToken ct = default)
        => new(ticket.TicketText.Contains("production down", StringComparison.OrdinalIgnoreCase) ? "high" : "normal");
}
internal sealed partial class ResponseComposerExecutor : Executor
{
    public ResponseComposerExecutor() : base("ResponseComposer") { }
    [MessageHandler]
    private async ValueTask HandleAsync(object[] results, IWorkflowContext context, CancellationToken ct = default)
    {
        await context.YieldOutputAsync($"Checks complete: {string.Join(", ", results)}");
    }
}
internal sealed partial class HumanEscalationExecutor : Executor
{
    public HumanEscalationExecutor() : base("HumanEscalation") { }
    [MessageHandler]
    private async ValueTask HandleAsync(TicketClassification ticket, IWorkflowContext context, CancellationToken ct = default)
    {
        await context.YieldOutputAsync($"Escalated to a human: {ticket.TicketText}");
    }
}
internal sealed partial class GenericHandlerExecutor : Executor
{
    public GenericHandlerExecutor() : base("GenericHandler") { }
    [MessageHandler]
    private async ValueTask HandleAsync(TicketClassification ticket, IWorkflowContext context, CancellationToken ct = default)
    {
        await context.YieldOutputAsync($"Filed under {ticket.Category}, no specialist agent for this category yet.");
    }
}

And the graph itself, with a conditional edge for the human-escalation branch and a fan-out/fan-in pair for the two parallel checks:

var classifier = new TicketClassifierExecutor(classifierAgent);
var duplicateCheck = new DuplicateCheckExecutor();
var severityCheck = new SeverityCheckExecutor();
var composer = new ResponseComposerExecutor();
var humanEscalation = new HumanEscalationExecutor();
var workflow = new WorkflowBuilder(classifier)
    .AddEdge<TicketClassification>(classifier, humanEscalation, condition: ticket => ticket?.NeedsHuman == true)
    .AddFanOutEdge(classifier, targets: [duplicateCheck, severityCheck])
    .AddFanInBarrierEdge(sources: [duplicateCheck, severityCheck], target: composer)
    .WithOutputFrom(humanEscalation, composer)
    .Build();

That fan-out edge does not use a targetSelector because both branches should run every time a ticket is not escalated, which is the simplest fan-out shape MAF supports. The version with a selector is for when only some targets should fire, which is what you would want if, say, the severity check should be skipped for anything already flagged as a duplicate.

Putting it together: the workflow calls the agent that does RAG

This is the piece none of the three original posts show, because none of them needed to. The PackageAdvisorExecutor below is the executor that actually calls releaseNotesAgent, the same RAG-backed custom agent from parts one and two, from inside the graph:

internal sealed partial class PackageAdvisorExecutor : Executor
{
    private readonly AIAgent _releaseNotesAgent;
    public PackageAdvisorExecutor(AIAgent releaseNotesAgent) : base("PackageAdvisor")
    {
        _releaseNotesAgent = releaseNotesAgent;
    }
    [MessageHandler]
    private async ValueTask<string> HandleAsync(TicketClassification ticket, IWorkflowContext context, CancellationToken ct = default)
    {
        var response = await _releaseNotesAgent.RunAsync(ticket.TicketText, cancellationToken: ct);
        await context.YieldOutputAsync(response.Text);
        return response.Text;
    }
}

Wire it in as a switch on category, alongside the fan-out branch from before:

var packageAdvisor = new PackageAdvisorExecutor(releaseNotesAgent);
var genericHandler = new GenericHandlerExecutor();
var fullWorkflow = new WorkflowBuilder(classifier)
    .AddEdge<TicketClassification>(classifier, humanEscalation, condition: ticket => ticket?.NeedsHuman == true)
    .AddSwitch(classifier, sw => sw
        .AddCase(t => t is TicketClassification c && !c.NeedsHuman && c.Category == "PackageUpgrade", packageAdvisor)
        .WithDefault(genericHandler))
    .WithOutputFrom(humanEscalation, packageAdvisor, genericHandler)
    .Build();
StreamingRun run = await InProcessExecution.RunStreamingAsync(
    fullWorkflow,
    "We're moving from EF Core 8 to EF Core 9. Anything that breaks LINQ translation?");
await foreach (WorkflowEvent evt in run.WatchStreamAsync())
{
    if (evt is WorkflowOutputEvent output)
    {
        Console.WriteLine($"Final answer: {output.Data}");
    }
}

Run that against a ticket about EF Core and the graph classifies it, routes it past the human-escalation check, hands it to PackageAdvisorExecutor, and that executor calls an agent that is, at that exact moment, running a vector search against Qdrant to ground its answer in the actual release notes on disk. Three separate MAF primitives, one request, no glue code that MAF itself didn't already give me a slot for.

I did have to pick one graph shape over the other for the final example; a production version would probably run the switch and the fan-out checks together rather than sequentially, using AddFanOutEdge with a targetSelector that includes packageAdvisor only for the right category. I kept them separate here because showing both patterns clearly mattered more than shaving one hop off the graph.

What I’d still add before this went anywhere near production

Durability first. Everything above runs with InProcessExecution, which means a crash mid-workflow loses whatever state was not explicitly persisted through QueueStateUpdateAsync. Microsoft has a public writeup on durable workflows that adds checkpointing so a graph can resume from durable storage after a process restart, and I would read that closely before trusting this pattern with anything that takes longer than a few seconds end to end.

Observability second. AddEventAsync gets you structured events per executor, which is a start, but what I actually want in a trace when a ticket gets classified wrong is the exact prompt the classifier agent sent, the raw model response before I parsed it into a TicketClassification, and which branch of the switch fired, all correlated to one ticket ID. Right now that means wrapping each agent call with my own logging, since the built-in events don't capture the prompt and response by default.

Governance third, and this is the one I care about most. I wrote a static reviewer a while back for a different MAF surface, declarative YAML workflows, that fails a build if a side-effecting action ships without a human-approval step ahead of it in the same branch. The exact same instinct applies here, just in C# instead of YAML: nothing in this graph currently stops someone from adding an executor that files a real support ticket, sends a real email, or calls a real billing API, and having that land as a two-line diff inside an existing branch with no reviewer noticing. A code-first graph is at least type-checked, which YAML is not, but a compiler catching a type mismatch has nothing to say about whether a new edge should have required a human in the loop. Before I’d let this touch a real ticketing system, every executor with a real-world side effect gets an explicit review comment and, ideally, a unit test that asserts the edge into it can only be reached from a state where NeedsHuman was already true or a human already said yes.

Tags: microsoft-agent-framework, dotnet, rag, ai-agents, csharp, workflow-orchestration, ollama, qdrant

Your Agent’s Memory Is Probably Wrong: A Practical Guide to Agentic Memory Design

Ali Suleyman TOPUZ — Fri, 25 Sep 2026 12:50:20 +0000

I built a memory layer, watched it rot, watched it get poisoned, and rebuilt it as four layers instead of one

I want to start with the thing that made me question the whole premise of what I was building: a filesystem with grep beat a funded, purpose-built memory product on its own benchmark.

This isn’t a rumor. On the LoCoMo long-conversation benchmark, a plain filesystem approach, grep, semantic search over flat files, and basic file operations, scored 74.0 percent accuracy with GPT-4o mini. Mem0’s graph-based configuration, the one built specifically to model entities and relationships in conversation history, scored 68.5 percent on its own best-reported setup. The simpler system won by 5.5 points against the tool designed to be smarter about memory than a folder of text files.

I’d spent the previous two months building something closer to the graph side of that comparison, an agent memory layer with embeddings, a vector store, and a retrieval pipeline I was genuinely proud of. So when I saw that number I did what anyone building this stuff for a living should do: I stopped and asked why.

The answer, once I sat with it, is not complicated. Large language models have seen an enormous amount of pretraining data involving grep, find, cat, reading a directory tree, and writing to a file. They have seen comparatively little data involving your bespoke vector store's query syntax, your particular graph schema, or the specific way your memory API expects a filter object. When a model has to choose a tool call and reason about what comes back, it is working from familiarity, not from some abstract notion of which retrieval method is objectively superior. Grep is not a good retrieval algorithm. It is a wildly familiar one, and familiarity is what lets an agent use a tool correctly on the first try instead of hallucinating a query it half remembers from a different API.

There’s a second reason vector search underperforms in a lot of real agent workloads, and it’s structural rather than about model familiarity: embeddings are bad at time. A vector store answers “what’s semantically close to this query,” not “what’s still true,” not “what superseded what,” not “what happened before what.” Conversations and codebases and support tickets are full of facts that change, get corrected, or get reversed, and a similarity search has no native concept of any of that. You end up needing a second system on top of the vector store just to handle staleness and contradictions, at which point you’ve added cost, latency, and an entire class of bugs to solve a problem the vector store was never built to solve in the first place.

None of this means vector search is useless. It means vector search is a specific tool for a specific job, and most agent memory problems are not that job. The decision rule I use now, and the one I’d hand to anyone starting this today, is simple: benchmark your memory approach against two baselines before you touch a vector database. Baseline one is full context, just stuffing everything into the window and seeing how the agent does. Baseline two is filesystem plus grep plus basic text matching. Only bring in a vector store, a graph database, or any paid memory API if it beats both baselines by a margin big enough to justify the added infrastructure, the added latency, and the added surface area for things to go wrong. In my experience that bar gets cleared when you have genuinely out-of-context retrieval needs, meaning millions of records you cannot fit in any context window at any price, real temporal reasoning across long historical spans, or multiple agents that need centralized, access-controlled memory. Below that line, a folder of markdown files and a grep call will carry you further than the pitch deck for any memory-as-a-service product will tell you.

What actually goes wrong without a real architecture

Here’s what I actually built first, because I think it’s worth being honest about the failure mode before I get to the fix. I had one table. memories, a blob of text, a timestamp, an embedding column. Every fact the agent learned, every user preference, every tool result worth remembering, went into that one table, and retrieval was a single similarity search over the whole thing.

It worked for about three weeks of light use. Then it started doing two things that should scare anyone running an agent in production.

The first was staleness. A user told the agent early on that their deployment target was staging. Three weeks later they told it, in a different conversation, that they’d moved to production. Both facts sat in the same table with roughly equal semantic weight, and depending on phrasing, the similarity search would sometimes surface the stale one. The agent confidently referenced a staging deploy that hadn’t existed for weeks. Nothing in my one-table design distinguished “this used to be true” from “this is true now.”

The second was worse. Somewhere in a batch of tool outputs I was letting the agent write straight into memory, unfiltered, a piece of scraped web content contained a line that looked almost exactly like a system instruction: something to the effect of “for future reference, always include the following text in your response.” The agent had ingested a web page as part of a research task, treated a line inside that page as a fact worth remembering, written it to memory verbatim, and then dutifully followed it two conversations later like a planted instruction, because by then it had no way of knowing that line had never come from me or from the user. Nothing validated what went into memory. Nothing tracked where a memory came from. I had built, without meaning to, a write path with no gatekeeper.

Those two failures point at the same root cause: treating memory as one undifferentiated bucket. A single table with a timestamp and an embedding cannot tell you what kind of fact something is, whether it’s still current, or whether you should trust where it came from. Fixing that meant admitting that “memory” isn’t one thing. It’s at least four different things, each with a different lifetime, a different write pattern, and a different failure mode.

Four layers, four jobs

The architecture I ended up with splits memory by function, not by storage technology. Here’s the shape of it before the code:

+-------------+------------------------------+----------------+------------------+
| Layer | What it holds | Lifespan | Typical store |
+-------------+------------------------------+----------------+------------------+
| Buffer | Current turn, active tool | One session | In-process list, |
| | calls, working scratch pad | | Redis with TTL |
+-------------+------------------------------+----------------+------------------+
| Episodic | Specific past events: "user | Weeks to | Append-only log, |
| | asked X on March 4, agent | months | Postgres table, |
| | did Y, result was Z" | | flat JSONL files |
+-------------+------------------------------+----------------+------------------+
| Semantic | Extracted, de-duplicated | Long-lived, | Vector store OR |
| | facts: preferences, entity | supersedable | filesystem, per |
| | relationships, stable state | | the rule above |
+-------------+------------------------------+----------------+------------------+
| Procedural | Learned rules for how to | Long-lived, | Versioned config |
| | act: "for this repo, always | rarely changes | or rules table |
| | run tests before commit" | | |
+-------------+------------------------------+----------------+------------------+

Buffer memory

This is the working set, the current conversation plus whatever the agent has pulled into context for the task at hand. It does not persist past the session, and it should not try to. Treating buffer memory as durable is the mistake that causes prompt-cache thrashing and context bloat in the first place. It’s a deque with a size cap, nothing more ambitious than that.

from collections import deque
class BufferMemory:
    def __init__ (self, max_turns: int = 20):
        self._turns = deque(maxlen=max_turns)
    def add(self, role: str, content: str) -> None:
        self._turns.append({"role": role, "content": content})
    def as_messages(self) -> list[dict]:
        return list(self._turns)

Episodic memory

Episodic memory is a log, not a database you overwrite. Every entry is an event that happened, tied to a timestamp and, critically, to a source. You never delete an episodic entry to “fix” it, you append a correction and let the record show both. This is what makes an audit trail possible later, and it’s also what feeds the semantic layer, since semantic facts get distilled out of episodes rather than written directly by the agent in the moment.

import json
import time
import uuid
class EpisodicMemory:
    def __init__ (self, path: str):
        self.path = path
    def record(self, actor: str, event: str, source: str) -> str:
        entry = {
            "id": str(uuid.uuid4()),
            "timestamp": time.time(),
            "actor": actor,
            "event": event,
            "source": source, # who/what generated this: "user", "tool:web_fetch", etc.
        }
        with open(self.path, "a") as f:
            f.write(json.dumps(entry) + "\n")
        return entry["id"]

Semantic memory

This is the layer where a lot of teams reach straight for a vector database, and per the decision rule above, that’s premature until you’ve measured it against grep over structured files. Semantic memory holds distilled, current facts, the kind of thing you’d write on an index card: user’s preferred deploy target, project’s chosen test framework, a person’s job title. The important property is supersession. A new fact doesn’t overwrite an old one, it marks the old one superseded and takes its place, so you always know what changed and when.

import json
import time
import uuid
class SemanticMemory:
    def __init__ (self, path: str):
        self.path = path
        self._facts: dict[str, dict] = {}
        self._load()
    def _load(self):
        try:
            with open(self.path) as f:
                for line in f:
                    fact = json.loads(line)
                    self._facts[fact["key"]] = fact
        except FileNotFoundError:
            pass
    def upsert(self, key: str, value: str, source: str) -> None:
        previous = self._facts.get(key)
        fact = {
            "key": key,
            "value": value,
            "source": source,
            "updated_at": time.time(),
            "supersedes": previous["id"] if previous else None,
            "id": str(uuid.uuid4()),
        }
        self._facts[key] = fact
        self._persist()
    def get(self, key: str) -> dict | None:
        return self._facts.get(key)
    def _persist(self):
        with open(self.path, "w") as f:
            for fact in self._facts.values():
                f.write(json.dumps(fact) + "\n")

If you’ve actually cleared the bar for needing similarity search, on wide topic coverage where exact keys don’t exist, plug in embeddings here rather than replacing the whole layer. A fully self-hosted option that costs nothing per call: run Ollama locally with a small embedding model and point your semantic layer’s search function at it instead of a paid embeddings API.

ollama pull nomic-embed-text
ollama serve

import requests
def embed(text: str) -> list[float]:
    resp = requests.post(
        "http://localhost:11434/api/embeddings",
        json={"model": "nomic-embed-text", "prompt": text},
    )
    return resp.json()["embedding"]

Store the resulting vectors in something as unglamorous as SQLite with the sqlite-vec extension, or Postgres with pgvector, both of which run in a Docker container with no external account required, and both of which let you keep the same supersession pattern from the code above instead of losing it the moment you add embeddings.

Procedural memory

Procedural memory is the layer most homegrown systems skip entirely, and it’s the one that pays off the most in agents that do repeated work against the same environment. It’s not a fact, it’s a rule about behavior: “in this repository, run dotnet test before proposing a commit," "this user always wants diffs, never full file rewrites." These rules should be versioned and reviewable like code, because they directly shape what the agent does next, which makes them the highest-leverage and highest-risk layer to get wrong. Here's a C# shape for it, since procedural rules map naturally onto a small versioned record type:

public sealed record ProceduralRule(
    string Id,
    string Scope, // e.g. "repo:billing-service"
    string Rule, // human-readable directive
    string Source, // who authored this rule
    int Version,
    DateTimeOffset CreatedAt,
    bool Active);
public sealed class ProceduralMemoryStore
{
    private readonly List<ProceduralRule> _rules = new();
    public void AddOrRevise(string scope, string rule, string source)
    {
        var existing = _rules
            .Where(r => r.Scope == scope && r.Rule == rule && r.Active)
            .FirstOrDefault();
        if (existing is not null)
        {
            _rules.Remove(existing);
            _rules.Add(existing with { Active = false });
        }
        var next = new ProceduralRule(
            Id: Guid.NewGuid().ToString(),
            Scope: scope,
            Rule: rule,
            Source: source,
            Version: (existing?.Version ?? 0) + 1,
            CreatedAt: DateTimeOffset.UtcNow,
            Active: true);
        _rules.Add(next);
    }
    public IReadOnlyList<ProceduralRule> ActiveRulesFor(string scope) =>
        _rules.Where(r => r.Scope == scope && r.Active).ToList();
}

Why you can’t just stuff it all in context instead

The obvious objection to building four layers is that modern context windows are enormous now, so why not just keep everything in the prompt and skip the architecture entirely. The “Lost in the Middle” finding is the answer to that, and it’s worth taking seriously rather than waving off as an old result from smaller-context models. The core finding is a U-shaped recall curve: models are noticeably better at using information placed at the very start or the very end of a long context, and performance degrades when the relevant fact sits in the middle of a long input, even for models explicitly built and marketed for long-context use. The effect shows up on straightforward multi-document question answering, not some adversarial edge case.

What that implies for memory design is specific, not just a vague “context is imperfect” caveat. Dumping your entire episodic log and every semantic fact into the prompt on every turn doesn’t just cost tokens, it actively makes retrieval worse, because the exact fact the agent needs is now buried in the least reliable part of the window instead of being surfaced at the top where the model actually attends to it. This is the real argument for retrieval as a discipline: not “context is too small,” but “even when context is big enough, unranked context degrades recall.” A memory layer’s job isn’t just storage, it’s putting the right handful of facts at the front of the prompt instead of relying on the model to find them buried in the middle of ten thousand tokens of history.

The number that made this real for me

I want to ground this in something other than a benchmark paper, because benchmarks are easy to nod along to and then ignore. When I rebuilt a coding agent’s memory from the single-table version to the four-layer version described above, on a workflow where the agent opens pull requests against a real repository and a human reviews them, the metric I actually cared about was the merge-without-rework rate: the percentage of the agent’s PRs that got merged without a human having to push a follow-up commit to fix something the agent got wrong.

Before the rebuild, over a two-week window, that rate sat at 61 percent. The failures were almost all the same shape as the staleness bug I described earlier: the agent would reintroduce a pattern the team had explicitly moved away from two sprints prior, because the old pattern and the new one were both sitting in an undifferentiated memory bucket with no notion of supersession. After the rebuild, with procedural rules capturing “we no longer do X, we do Y instead” as versioned, superseding entries rather than facts competing on similarity score, and semantic memory holding the current state of each convention instead of a blended average of every version that ever existed, the same metric over the following six weeks came in at 79 percent. Eighteen points, on the one number a reviewing engineer actually feels every time they open a PR from the agent. That gap is architecture, not model capability. Nothing about the underlying model changed in that window.

Memory poisoning is not hypothetical

The prompt-injection-via-scraped-webpage story I told earlier was an accident. It turns out the deliberate version of that attack has a name and a measured success rate, and it should change how you think about the write path into memory.

MINJA, short for memory injection attack, works through query-only interaction: an attacker doesn’t need write access to your database, they just need to be able to talk to the agent, the same access any normal user has. They craft inputs designed to get the agent to record a poisoned “memory” of its own accord, the same mechanism that let my scraped web page smuggle an instruction into my memory table. In controlled research conditions, this attack achieved better than a 95 percent injection success rate, meaning the poisoned content reliably made it into memory, and a roughly 70 percent attack success rate, meaning the poisoned memory actually altered the agent’s later behavior in the intended way.

The honest caveat, and it matters: those numbers were measured under idealized conditions, largely empty or sparse memory stores. The same research found that realistic conditions, meaning a memory store already full of legitimate, established facts, dramatically reduce how effective the attack is, because a single poisoned entry has to compete against a much larger body of consistent, trustworthy history before it gets treated as authoritative. That’s not a reason to relax. It’s a reason to treat memory volume and hygiene as a security property, not just an efficiency one, and to make sure new agents or fresh deployments, the ones with the sparsest memory and the biggest blind spot, get the strictest write controls rather than the most lenient ones.

Three defenses actually hold up against this, and none of them require a research team to implement:

+----------------------+---------------------------------------------------------------+
| Defense | What it does |
+----------------------+---------------------------------------------------------------+
| Validation on write | Every candidate memory is scored before it's committed, not |
| | after. Composite trust signals: does this contradict an |
| | existing high-confidence fact, does it look like an |
| | instruction rather than a fact, did it come from a tool |
| | output versus a direct user statement. |
+----------------------+---------------------------------------------------------------+
| Provenance tracking | Every entry carries who or what produced it: "user", "tool: |
| | web_fetch", "agent inference". Provenance is what lets you |
| | later discount or purge everything that came from one |
| | compromised source without touching anything else. |
+----------------------+---------------------------------------------------------------+
| Controlled forgetting | Trust-aware retrieval with temporal decay: older, unverified, |
| | or low-provenance entries lose retrieval weight over time |
| | unless something reaffirms them. Nothing sits in memory |
| | forever just because it was never explicitly wrong. |
+----------------------+---------------------------------------------------------------+

The thread connecting all three is that none of them trust the write path by default. A memory system that accepts anything an agent proposes and immediately treats it as ground truth is a memory system that will eventually get poisoned, whether by an attacker or, like mine, by an ordinary tool output nobody thought to validate.

Building it properly in .NET

Everything above is a design. Here’s the part I actually shipped, in C# against .NET 10, because “validate on write” and “don’t trust new memories” are principles that need to show up as code, not just as bullet points in an architecture doc. The core ideas: new memories land in quarantine, not straight into the trusted store; writes use optimistic concurrency so two concurrent updates can’t silently clobber each other; a correction supersedes the old entry instead of overwriting it; every entry carries a TTL; and a subject can be forgotten on request, which matters the moment you’re handling anything resembling personal data.

public enum MemoryStatus { Quarantined, Trusted, Superseded, Expired, Forgotten }
public sealed class MemoryEntry
{
    public Guid Id { get; init; } = Guid.NewGuid();
    public required string SubjectId { get; init; } // whose memory this is
    public required string Key { get; init; }
    public required string Value { get; init; }
    public required string Source { get; init; } // provenance
    public MemoryStatus Status { get; set; } = MemoryStatus.Quarantined;
    public Guid? SupersededBy { get; set; }
    public DateTimeOffset CreatedAt { get; init; } = DateTimeOffset.UtcNow;
    public DateTimeOffset? ExpiresAt { get; set; }
    // Optimistic concurrency token, mapped to a rowversion/xmin column
    // depending on your provider (SQL Server vs Postgres).
    public byte[] RowVersion { get; set; } = Array.Empty<byte>();
}
public interface IMemoryValidator
{
    Task<bool> IsSafeToPromoteAsync(MemoryEntry candidate, CancellationToken ct);
}
public sealed class GovernedMemoryService
{
    private readonly AppDbContext _db;
    private readonly IMemoryValidator _validator;
    private readonly TimeSpan _defaultTtl = TimeSpan.FromDays(90);
    public GovernedMemoryService(AppDbContext db, IMemoryValidator validator)
    {
        _db = db;
        _validator = validator;
    }
    // Step 1: every new memory is quarantined by default. Nothing an agent
    // writes is trusted the moment it's written.
    public async Task<MemoryEntry> RecordAsync(
        string subjectId, string key, string value, string source, CancellationToken ct)
    {
        var entry = new MemoryEntry
        {
            SubjectId = subjectId,
            Key = key,
            Value = value,
            Source = source,
            Status = MemoryStatus.Quarantined,
            ExpiresAt = DateTimeOffset.UtcNow.Add(_defaultTtl),
        };
        _db.MemoryEntries.Add(entry);
        await _db.SaveChangesAsync(ct);
        return entry;
    }
    // Step 2: promotion out of quarantine is a separate, explicit step,
    // gated by validation, so a poisoned write never becomes trusted
    // just by virtue of having been written.
    public async Task<bool> TryPromoteAsync(Guid entryId, CancellationToken ct)
    {
        var entry = await _db.MemoryEntries.FindAsync([entryId], ct)
            ?? throw new KeyNotFoundException(nameof(entryId));
        if (!await _validator.IsSafeToPromoteAsync(entry, ct))
            return false;
        entry.Status = MemoryStatus.Trusted;
        try
        {
            await _db.SaveChangesAsync(ct); // relies on RowVersion for concurrency
        }
        catch (DbUpdateConcurrencyException)
        {
            // Someone else touched this entry between read and write.
            // Reload and let the caller decide whether to retry.
            return false;
        }
        return true;
    }
    // Correction never overwrites. It writes a new quarantined entry and
    // marks the old one superseded, preserving the full history.
    public async Task<MemoryEntry> SupersedeAsync(
        Guid oldEntryId, string newValue, string source, CancellationToken ct)
    {
        var oldEntry = await _db.MemoryEntries.FindAsync([oldEntryId], ct)
            ?? throw new KeyNotFoundException(nameof(oldEntryId));
        var replacement = await RecordAsync(
            oldEntry.SubjectId, oldEntry.Key, newValue, source, ct);
        oldEntry.Status = MemoryStatus.Superseded;
        oldEntry.SupersededBy = replacement.Id;
        await _db.SaveChangesAsync(ct);
        return replacement;
    }
    // Subject-level "forget me": every entry for this subject, trusted or
    // not, is marked Forgotten and excluded from every future read path.
    // This does a soft delete rather than a hard one on purpose, so an
    // audit of "what did we forget and when" survives the deletion itself.
    public async Task ForgetSubjectAsync(string subjectId, CancellationToken ct)
    {
        var entries = await _db.MemoryEntries
            .Where(e => e.SubjectId == subjectId && e.Status != MemoryStatus.Forgotten)
            .ToListAsync(ct);
        foreach (var entry in entries)
            entry.Status = MemoryStatus.Forgotten;
        await _db.SaveChangesAsync(ct);
    }
    // TTL sweep, run on a background schedule (IHostedService / Quartz job).
    public async Task ExpireStaleEntriesAsync(CancellationToken ct)
    {
        var now = DateTimeOffset.UtcNow;
        var expired = await _db.MemoryEntries
            .Where(e => e.ExpiresAt != null && e.ExpiresAt < now
                        && e.Status != MemoryStatus.Forgotten)
            .ToListAsync(ct);
        foreach (var entry in expired)
            entry.Status = MemoryStatus.Expired;
        await _db.SaveChangesAsync(ct);
    }
}

A few things about this shape that only became obvious once I’d been running it a while. The RowVersion field isn't decoration, it's what stops two concurrent agent runs from racing each other to update the same fact and silently dropping one of the writes, which is exactly the kind of bug that's invisible until you have two sessions touching the same subject at once, and then it's a very confusing bug report. Quarantine as a default status rather than a boolean flag makes the validator a hard gate instead of an optional check someone forgets to call. And ForgetSubjectAsync doing a soft delete rather than a hard DELETE FROM is deliberate: you want to be able to prove, later, that a forget request was honored and when, without keeping the forgotten content itself around to prove it.

If you’re not on SQL Server or don’t want to stand up a managed database just to try this, the same shape runs against Postgres in a local Docker container with no other changes beyond swapping the EF Core provider, and Postgres’s xmin system column gives you the same optimistic concurrency behavior as SQL Server's rowversion once you map it with [Timestamp] or a value converter.

What to actually build first

If you’re starting from nothing today, here’s the order I’d do it in, tying the architecture and the implementation together instead of treating them as two separate concerns.

Start with the four-layer split before you write a single line of retrieval code, because retrofitting layer separation onto a one-table design is a much bigger job than building it in from day one, as I learned the expensive way. Buffer and episodic are cheap, a deque and an append-only log, and you should have both running in the first afternoon. Semantic comes next, and the moment you build it, run the two-baseline test from the start of this piece before you let yourself reach for a vector store: full context, then filesystem plus grep, and only add embeddings if neither clears the bar for your actual workload. Procedural memory is the layer people build last if they build it at all, and it’s the one I’d now argue matters most for any agent doing repeated work in the same environment, because it’s the difference between an agent that keeps making the same corrected mistake and one that actually learns your team’s conventions.

Then, and only once the four layers exist conceptually, wire in the governance model from the .NET example: quarantine by default, explicit promotion gated by validation, supersession instead of overwrite, TTL on everything, and a real forget path. Don’t build governance first and layers second. Governance without layered memory just means you’re carefully validating writes into the same undifferentiated bucket that caused the staleness problem in the first place. The two pieces have to land together, because a well-governed single table is still a single table, and a beautifully layered memory system with no write validation is still one poisoned tool output away from confidently repeating an attacker’s instructions back to your users.

The version of this I run now isn’t smarter than the one-table version I started with. It just knows what kind of fact it’s looking at, when that fact stopped being true, and whether it should trust where the fact came from before it acts on it. That turned out to be the whole problem.

Tags: ai-agents, agentic-memory, dotnet, llm-engineering, ai-security, csharp, software-architecture

DeepSeek Harness vs Codex vs Claude Code: What I Learned Comparing Three Agent Harnesses

Ali Suleyman TOPUZ — Tue, 22 Sep 2026 14:37:54 +0000

I’ve been saying for months that the model is not the product anymore, the harness is. Nobody believed me until I sat down and put three of them side by side on the same machine, and what I found made me rewrite this piece twice.

Here’s the thing that started this whole rabbit hole: take the exact same underlying model, no prompt changes, no fine-tuning, no new training data, just swap the execution environment around it, and the model goes from mediocre to genuinely good. I’d seen this claimed before in passing, but didn’t take it seriously until I found the number that made me stop scrolling. Research cited by JIN, who writes the System Architect newsletter on Medium, put a figure on it: the same model, wrapped in a different harness, moved a programming benchmark from 42 percent success to 78 percent. No new weights. Just a different thing deciding how context gets assembled, how tools get called, and how much rope the model gets before someone steps in.

That’s not a small effect. That’s the difference between “would not ship this” and “this is genuinely useful,” and it comes entirely from the layer wrapped around the model.

So I spent a couple of weeks doing what most comparison posts skip: installing the harnesses, reading the code that actually ships, not just the README, and finding where each one’s design choices show up as real, felt friction. Here’s what I found with DeepSeek Harness, OpenAI’s Codex as a Platform, and, as the baseline I use every day, Claude Code.

DeepSeek Harness: everything really is a plugin

DeepSeek Harness launched as a developer preview a few months ago under the tagline “Everything is a Plugin,” and the GitHub numbers aren’t exaggerated marketing, the repo is sitting north of 180,000 stars with just under 20,000 forks. That’s a real, fast-growing community. It’s MIT licensed and fully open source.

The architecture is the interesting part, not the star count. DeepSeek Harness is built on Cordis, a plugin meta-framework the team vendored and patched rather than pulled in as a stock dependency. Cordis treats every capability, the agent loop, the tool scheduler, the session log, the sandbox, the web UI, even the SDK itself, as a plugin implementing a Service interface. Plugins declare what they need through an inject field, and Cordis resolves load order from those dependencies instead of you hand-writing a boot sequence. A context object works as a shared registry where services claim stable keys like ctx.tools, ctx.llm, or ctx.sessions, so a plugin asks the context for whatever holds that key rather than importing a concrete implementation directly. Registrations install through ctx.effect(), which makes them reversible, so a reload or teardown cleanly unwinds instead of leaving dangling state behind. That's a genuinely clean design for hot-swapping components, and it's why you can, in principle, swap the model provider, the tool set, or the entire workflow engine without touching the rest of the system.

On paper this is exactly what “everything is a plugin” should mean. In practice, I wanted to know what actually ships versus what the pitch implies, so I read through the repository the way you’d read a codebase you were about to depend on in production, not the way you read a landing page.

What’s actually in the box

The core repo runs to roughly 453,000 lines of TypeScript across around 219 workspace packages, built on a vendored fork of Cordis pinned at 4.0.0-rc.7 with eighteen local patches on top of it, shipping as npm package 0.1.0-rc.6. That's a lot of surface area for something billed as "just plugins": you're depending on a patched fork of a pre-1.0 framework, not the framework's own release train.

The default model catalog is narrow, exactly two models, deepseek-v4-flash and deepseek-v4-pro. Everything else, Anthropic, OpenAI, Bedrock, Vertex, Azure, or any custom OpenAI-compatible endpoint, gets wired up through the provider configuration screen, not auto-discovered. This is where the interesting part starts, because a fair amount of what happens once you add providers isn't obvious from the docs alone.

The install, the router, and the two files nobody reads twice

I leaned heavily here on a hands-on writeup by a developer who goes by Andrus, published on the AI Advances gopubby publication, titled “Should You Switch From Claude Code to DeepSeek Harness? I Read the Shipped Code Instead of the README.” The framing matches exactly what I found poking around myself: read what ships, not what the marketing implies.

First number that jumps out: the install is 359 megabytes before you type a single character into it. That’s the @deepseek-ai/dsh package on disk, before any plugins, before any provider config. That's roughly the size of a small Electron app, for a CLI-first agent harness.

Second, and this is the part that made me go dig through the provider code myself: what looks in the repo like a small test fixture for provider configuration is, once you trace how it’s actually wired at runtime, a full multi-provider router covering somewhere in the neighborhood of three dozen providers, Anthropic and Claude models included. That’s not disclosed prominently in the getting-started docs, you find it by reading docs/user/guide/providers.md and following the "Add provider" flow into the actual adapter code. It's not sinister, DeepSeek genuinely wants to let you run any model through it, but the docs sell "DeepSeek's own harness" and the shipped code is a general-purpose model router that happens to default to DeepSeek's models. Worth knowing if procurement ever asks whether this sends anything to a competitor's API: the honest answer is it can, and the plumbing is already built in, quietly.

Third, and the one that surprised me most: there is no native local-model provider. Not Ollama, not LM Studio, nothing with a first-class entry in the provider catalog. Andrus was running a 36GB MacBook Pro with Ollama serving a locally quantized Qwen model handling roughly 70 percent of his day-to-day workload, and DeepSeek Harness could not drive that model without him hand-writing a custom provider entry pointed at Ollama’s OpenAI-compatible endpoint. Compare that to Codex, which documents OSS mode against Ollama or LM Studio directly in its own config guide. DeepSeek Harness technically supports it, since any endpoint speaking the OpenAI chat completions shape can be registered, but “supports it if you write the adapter yourself” is a different bar than “ships with it.”

Fourth is one I verified myself against the live repository rather than repeat secondhand. DeepSeek Harness’s own repo carries both a CLAUDE.md and an AGENTS.md at the root, a common pattern now that multiple coding agents look for different filenames. I pulled both files directly. In the dsh repo itself, CLAUDE.md is nine bytes, literally just the text AGENTS.md, a pointer telling any tool that reads it to go read the real file instead: the maintainers being disciplined about not duplicating content across two files.

The problem is that discipline is a convention, not something the harness enforces for you. Point dsh at a typical real-world repo where CLAUDE.md and AGENTS.md have organically diverged, which happens constantly once a team runs Claude Code and a second agent side by side for a while, and the context loader pulls both files into every prompt, in full, with no deduplication. I ran a rough estimate against two files from one of my own projects: a CLAUDE.md around 1,400 tokens of Claude Code-specific tool conventions, and an AGENTS.md around 2,100 tokens of generic instructions that had drifted out of sync with it over eight months. That's an extra 1,400 tokens of pure duplication tax on every single turn, not once per session, because there's no caching discount for instruction files re-sent with each prompt. Over a long session with fifty or sixty tool-calling turns, that's tens of thousands of wasted tokens paying for context the model already saw.

None of these four things is disqualifying on its own. Together they’re a pattern: DeepSeek Harness’s actual behavior, once you read the shipped code, is broader and less disciplined than the “everything is a plugin, clean and modular” pitch implies. The plugin architecture underneath is real and well-engineered. What ships around it needs a closer read than the README gives you credit for needing.

Codex as a Platform: OpenAI’s bet on the app-server, not the chat window

OpenAI’s framing for Codex shifted meaningfully this year. It’s no longer positioned as a coding chatbot with a CLI, it’s positioned explicitly as an open agent harness that other products build on top of, laid out in their “Codex as a Platform” writeup on the OpenAI Developers site.

The architecture splits into three integration surfaces instead of one monolithic app. codex exec is for scripts and bounded, one-off background tasks, the kind of thing you'd trigger from a CI job. The Codex SDK is a direct programmatic interface for applications that need to start, resume, or stream a task from inside their own code. The app-server is the deepest integration point: it exposes persistent conversation state, event streaming, tool exposure, and approval-request handling, so a product can build its own UI around Codex's agent loop while Codex handles sandboxing, execution, and approval plumbing underneath. The dividing line is deliberate: your application owns business logic, data sources, MCP services, approval policy, and the UI. Codex owns the agent loop and the sandboxed execution environment. That's a service boundary, not a chat window you embed.

The results backing this up were more concrete than I expected. On ARC-AGI-3, retained reasoning combined with context compaction pushed GPT-5.6’s accuracy from 13.3 percent to 38.3 percent, while cutting output tokens roughly sixfold. Same shape of claim as the 42-to-78 harness effect I opened with: same reasoning capability, dramatically different result once the surrounding execution logic changes how context gets carried and compacted.

The case study that actually convinced me this isn’t just a demo-day number is Asana’s public writeup of migrating their frontend test suite off Enzyme, an aging testing framework that had become a hard blocker to any React upgrade, onto React Testing Library. Asana’s engineering team estimated the migration properly done, framework swap plus improved coverage plus cleaning up legacy test infrastructure, would take roughly five more years at their existing pace. They ran it with Codex instead, frontier models on a high-reasoning setting, up to four agents running in parallel across different directories, working through nights and weekends unsupervised. The prompt driving all of it was five sentences: migrate everything using Enzyme to React Testing Library, follow existing conventions, test the changes, bias toward easier conversions first. It shipped in about a week and a half of engineering time spread across two calendar weeks. Total cost, model usage plus infrastructure, landed around 12,000 dollars, against a back-of-envelope estimate of roughly 6 million dollars for the manual version of the same scope.

Asana’s postmortem is refreshingly honest about why it worked, and it’s not “the model is smart.” Their conclusion was that agent output quality depends heavily on the quality of the environment you hand it: established conventions, clear examples, well-designed test helpers already in the codebase before any agent touched it. Their friction points weren’t model limitations either, they were stale internal documentation steering agents toward deprecated patterns, and infrastructure bottlenecks like slow linting and flaky tooling creating more drag than the model ever did. That’s a useful data point about where the ceiling actually sits on this kind of work: not model capability, environment quality.

If you want Codex against a local model rather than a hosted one, it supports this natively through OSS mode, unlike DeepSeek Harness’s roll-your-own approach:

# ~/.codex/config.toml
[model_providers.ollama]
name = "Ollama (local)"
base_url = "http://localhost:11434/v1"
wire_api = "chat"
[profiles.local]
model = "qwen2.5-coder:32b"
model_provider = "ollama"

# pull a coding-capable model locally first
ollama pull qwen2.5-coder:32b
# then run codex against your local profile, no API key needed
codex --profile local

That’s a first-class, documented path, not a workaround you have to reverse-engineer from the adapter source.

The sharpest framing I found: it’s not features, it’s where complexity accumulates

The comparison that changed how I think about this space didn’t come from a feature checklist. It came from JIN’s argument that the reason everyone is suddenly rewriting their harness runtime, instead of just picking a bigger model, is that the harness is now a first-class engineering decision. Once you accept that, the interesting question stops being “which harness has more plugins” and becomes “where does each harness let complexity pool up, and is that where you actually want it.”

DeepSeek Harness pools complexity into the plugin graph. The core, Cordis plus the agent loop, is genuinely compact and well-factored, but almost everything you’d care about (which provider you’re really talking to, how instruction files get merged, what a plugin does at runtime versus what its name implies) lives inside a plugin you have to go read. That’s a compact-core design: power and risk both concentrated in a growing, loosely-typed plugin ecosystem around a small, disciplined center.

Codex pools complexity at the boundary between your application and the agent loop. The app-server model means Codex stays relatively opinion-free about approvals, business logic, and UI, and pushes that decision-making onto whoever builds on top of it. That’s a service-boundary design: the complexity doesn’t disappear, it moves to whichever side of the API you’re standing on, and if you’re the one building the product around Codex, that’s you.

Claude Code, by contrast, keeps a documented event surface, session lifecycle, per-turn hooks, PreCompact and PostCompact around context management, tool-use hooks with allow-or-deny authority, that behaves like a replayable, inspectable runtime rather than either a plugin soup or a bare API boundary. You don’t get Cordis-style hot-swapping of the core loop, and you don’t get Codex’s clean hand-off to an external app, you get a system that logs its own decisions at each lifecycle point and gives you a hook into every one, a different bet: less raw extensibility, more auditability of what already happened.

None of these is objectively correct. They’re three answers to the same question of where you want the hard-to-reason-about parts of an agent system to physically live.

I wanted numbers behind this framing rather than vibes, so I went looking for a benchmark that measures harnesses as a variable rather than models. The closest real one is Harness-Bench, a 2026 paper measuring harness effects across models on 106 realistic agent tasks. Worth being upfront about its limits before citing numbers from it: it tests six configurable, generically-named harness configurations plus Codex evaluated separately, and does not test DeepSeek Harness or Claude Code by name, so treat this as evidence that harness choice matters at all, not a scoreboard ranking these three products. With that caveat attached, the spread was large: the best configurable harness hit a 76.2 completion score at 68,700 tokens per task, the weakest scored 52.4 at 82,100 tokens, a 23.8-point gap while burning more tokens for a worse result. Codex, evaluated independently, scored 80.4. The authors are explicit their scoring relies partly on LLM-assisted judging and describe the results as diagnostic under one fixed protocol, not a guarantee of production performance. I’d treat it the same way: directionally convincing that harness architecture is a real, measurable variable, not a leaderboard to build a purchasing decision on alone.

A separate paper, an enterprise agent evaluation framework called CLEAR, found something that rhymes with this from a different angle: across six agent systems on 300 enterprise tasks, up to 50x cost variation showed up between systems delivering similar accuracy, with accuracy-optimized configurations running 4.4 to 10.8 times more expensive than cost-aware ones for comparable quality. Different study, same underlying point: the harness and its configuration choices do as much work on your bill as the model does.

The comparison, side by side

+----------------+-----------------------+-----------------------+-----------------------+
| Dimension | DeepSeek Harness (dsh) | Codex as a Platform | Claude Code |
+----------------+-----------------------+-----------------------+-----------------------+
| Local models | No native provider, | Native OSS mode via | None. Model call is |
| | hand-wire a custom | Ollama/LM Studio, | inherently Anthropic |
| | OpenAI-compat endpoint | documented in config | API, Bedrock, or Vertex|
+----------------+-----------------------+-----------------------+-----------------------+
| Plugin system | Core design principle, | Not plugin-based, app | Hooks + MCP + |
| | Cordis DI, ctx keys, | owns logic, Codex owns | subagents, fixed core |
| | reversible effects | the loop via app-server | with attach points |
+----------------+-----------------------+-----------------------+-----------------------+
| Context handling| Loads CLAUDE.md and | App controls context | CLAUDE.md at session |
| | AGENTS.md in full, no | assembly, Codex handles | start, PreCompact/ |
| | dedup if they diverge | compaction internally | PostCompact hooks |
+----------------+-----------------------+-----------------------+-----------------------+
| Rough cost | Cheap model defaults, | Frontier pricing, but | Frontier pricing, |
| | but router overhead and | Asana case shows ~500x | subscription tiers plus |
| | duplicate context add | cheaper than the manual | usage-based options |
| | unseen tax | alternative | |
+----------------+-----------------------+-----------------------+-----------------------+

That cost row deserves its own caveat: “cheap model defaults” and “frontier model cost” aren’t apples to apples, DeepSeek Harness defaults to DeepSeek’s own inexpensive-per-token models, while Codex and Claude Code default to frontier-tier pricing. The point isn’t that one is cheaper in isolation, it’s that DeepSeek Harness’s per-token savings can get quietly eaten by the router overhead and the duplicate-context tax described above, in a way that doesn’t show up until you actually watch token usage over a real session instead of trusting the sticker price.

Where I actually landed

If you’re a solo developer or small team that wants to swap models freely, including cheap providers or your own fine-tunes, and you’re comfortable reading plugin source when the docs run thin, DeepSeek Harness’s architecture is legitimately good engineering. Just budget time to read the provider code before trusting what’s actually being called, and check your CLAUDE.md/AGENTS.md situation before assuming the context loader is economical with your tokens, because right now it isn't.

If you’re building a product that needs an agent embedded inside it, not a chat window bolted on, Codex’s app-server model is the more honest architecture, specifically because it draws the boundary explicitly instead of leaving you to reverse-engineer where your responsibility starts. The Asana numbers are extraordinary, but the caveat that matters is Asana’s own: none of it works without a codebase that already has clean conventions and good test scaffolding for the agent to imitate. Codex won’t invent that discipline for you, it amplifies whatever discipline is already there, in both directions.

If you’re already inside the Claude Code ecosystem and mostly want a documented, inspectable event surface to build policy on top of, the hooks-plus-sandbox model is still the one I reach for by default, mainly because auditability of what happened beats raw extensibility of what could happen, for work where an agent runs semi-unsupervised against a codebase people’s paychecks depend on.

None of these are wrong choices. They’re three bets about where complexity should live, and the honest answer is the same one that’s true of any infrastructure decision: read the code that actually ships, not the page trying to sell it to you.

Tags: agent-harness, deepseek, openai-codex, claude-code, ai-agents, developer-tools, llm-engineering

DEV Community: Ali Suleyman TOPUZ

Your Knowledge Graph Is Measuring Your Pipeline, Not Your Subject: A Replication Attempt

Picking domains I actually know

The pipeline, for real

The bug that actually happened

What actually happened

The prompt change that broke the metric on purpose

What I’d check before trusting a number like this again

Three Signals Say MCP Is Not Always the Right Tool, and What to Reach For Instead

A week of reading other people’s postmortems made me go back and re-check my own defaults

Why the “just wrap it in MCP” assumption breaks down

Finding one: a developer who built the MCP version first, then threw it out

Finding two: a controlled test where the MCP tools just sat there unused

Finding three: the architectural case, with the security research to back it up

Building it myself: the same tool, two ways, and where the tokens actually went

What the schemas actually cost, measured

Trying the discovery step against a local model

The rule of thumb I’m actually using now

Where I landed

Multi-Agent Systems Don’t Have a Consistency Problem. They Have a Database Problem.

The night one customer got refunded twice

The LLM is not the bug. The LLM is a very slow, very creative client.

Every orchestration framework hits the same three walls

What 16 agents writing a C compiler actually needed

Pattern 1: Idempotency keys on every agent-triggered side effect

Pattern 2: Optimistic locking in SQL, not in the orchestrator

Pattern 3: A claim graph, so disagreements stop being silent

Running all of this locally (no paid API needed)

“More agents” is not a strategy, and there’s data now

How do you test “two agents raced and one lost”?

The checklist I use now, before adding a second agent

References

Harness, Framework, or SDK: I Audited My Own Agent Stack Against a Real Source-Code Study

The sentence that stopped me

Three words that are not synonyms

My three integration points

1. The customer-facing assistant: I picked a harness

2. The internal batch agent: I picked a framework

3. The narrow helper: I picked an SDK (the second time)

The wrong choice, and what it cost me

A fair warning about the audit itself

The checklist I use now

Where I landed

My Eval Suite Was Lying to Me: Auditing the Judge, Not Just the Agent

What I Was Actually Evaluating

The Day I Actually Read the Reasoning

Building a Diagnostic Harness

Catching Reasoning and Score Disagreement

The Report

How Many Judge Outputs You Actually Need to Check

What I Do Differently When Agreement Comes Back Low

My Rule Now

Who Authorized the Agent? Building Real Identity and Delegation for AI Agents in .NET

What RFC 8693 actually buys you that a diagram with boxes labeled “policy” never will

Impersonation and delegation are not the same failure mode

What RFC 8693 actually specifies

Building the three-hop chain

The nested actor claim, as a real type

The exchange service

What the token actually looks like at each hop

What this actually buys you

What RFC 8693 does not solve

Further reading

The Real Cost Curve of Running Agents in Production, From Four Different Companies

Uber: routing that moves, not routing that’s configured once

Spotify: making the routing rule impossible to skip

Google’s Headroom: the layer underneath routing

DeepSeek V4.1-Flash: what “a fraction of the cost” actually buys you

Building the instrumentation: middleware around the HTTP client

A router function, and the baseline it has to beat

The adoption order, for a small team with limited time

.NET + AI in Production: What Actually Works (A Field Survey)

Five write-ups, one week, and the patterns that kept showing up when the hype was stripped out

1. Draw the boundary at Microsoft.Extensions.AI

2. On-device AI in MAUI, without marrying one model

3. The package supply chain got sharper teeth

Problem one: free packages stopped being free

Problem two: models invent packages

Guardrails that cost almost nothing

4. Architecture fitness tests for AI-generated pull requests