DEV Community: Mehdi Mohseni

Function Calling in C#: Building Tools Your AI Agent Can Call

Mehdi Mohseni — Sun, 25 Oct 2026 07:26:36 +0000

The gap between a chatbot and an agent is function calling. Without it, the model can only generate text, so you have to parse that text, guess the intent and hope the format holds. With function calling, the model emits a structured call: a tool name and arguments that match a schema you gave it. Your code runs it, the model sees the result and continues. That loop is the engine behind every serious agentic application.

This article builds that loop in C# with the official Anthropic SDK: defining a tool, running the loop, dispatching to real code, handling failures, and the production concerns around it. All the code here was compiled and run against Anthropic 13.2.0 and a PostgreSQL 18 database.

What Function Calling Actually Is

In the Messages API, tool use is part of the message exchange, not a side channel. When the model decides to call a tool, the response has stop_reason: "tool_use" and one or more tool_use content blocks. Each block has an id, a tool name and a JSON input that follows the schema you provided.

Your application runs the tool, locally, against a database or an internal API, and sends the outcome back as a tool_result block in the next user message. The model continues from there, maybe calling more tools, until it returns stop_reason: "end_turn" with a final answer.

This beats parsing text for three reasons. The input comes as structured JSON shaped by your schema, not as prose you have to regex. Failures are part of the protocol: you return an error result and the model can reason about it and try again. And the model can ask for several tools in one turn, which you can run in parallel.

Setting Up: The SDK

Since version 10, the Anthropic package on NuGet is the official Claude SDK for C#, from anthropics/anthropic-sdk-csharp. Versions 3.x and below of that package name were the community tryAGI SDK, which now lives at tryAGI.Anthropic. If a project has an old Anthropic reference, check which one it is before you copy any code, because the APIs are completely different.

dotnet add package Anthropic

The SDK models the API closely. Content blocks are union types (ContentBlock wraps a TextBlock, a ToolUseBlock and so on, and you reach the concrete block through .Value), and MessageParam accepts a UserMessageParam or an AssistantMessageParam. That's more ceremony than a hand-written client, but the compiler tells you when you build a message the API would reject.

You could use Semantic Kernel or Microsoft.Extensions.AI here instead. For learning the loop, or debugging a tool that isn't being called, it's worth being close to the wire first. Layering an abstraction on top is easy once you know what it sends.

Defining a Tool in C

A tool has a name, a description and a JSON Schema for its input. The description isn't documentation for people. It's what the model reads to decide whether and when to call the tool, and vague descriptions lead to missed or wrong calls.

var getOrderTool = new Tool
{
    Name = "get_order_details",
    Description = """
        Retrieves the current status, line items, and shipping information for a
        specific customer order. Use this when the user asks about order status,
        delivery dates, or what items are in an order.
        """,
    InputSchema = new InputSchema
    {
        Properties = new Dictionary<string, JsonElement>
        {
            ["order_id"] = JsonSerializer.SerializeToElement(new
            {
                type = "string",
                description = "The order identifier, e.g. ORD-20261234"
            }),
            ["include_line_items"] = JsonSerializer.SerializeToElement(new
            {
                type = "boolean",
                description = "Whether to include individual line items in the response"
            })
        },
        Required = ["order_id"]
    }
};

MessageCreateParams.Tools takes a list of ToolUnion, because the same list also holds Anthropic's built-in tools. A Tool converts to it implicitly, so [getOrderTool] works.

The Agentic Loop in C

The loop is simple in concept and easy to get wrong. The two common mistakes: handling only the first tool_use block when the model asked for several, and sending the results back in separate messages. The API expects one user message that contains a tool_result for every tool_use id of the previous turn.

using System.Text.Json;
using Anthropic;
using Anthropic.Models.Messages;

public static class AgentLoop
{
    public static async Task<string> RunAgentLoopAsync(
        AnthropicClient client,
        string userMessage,
        IReadOnlyList<ToolUnion> tools,
        Func<string, IReadOnlyDictionary<string, JsonElement>, CancellationToken, Task<object>> dispatcher,
        CancellationToken ct = default)
    {
        var messages = new List<MessageParam>
        {
            new UserMessageParam { Content = userMessage }
        };

        const int maxTurns = 10; // guard against infinite loops
        for (int turn = 0; turn < maxTurns; turn++)
        {
            var response = await client.Messages.Create(new MessageCreateParams
            {
                Model = "claude-sonnet-5-5",
                MaxTokens = 4096,
                Tools = tools,
                Messages = messages
            }, ct);

            // Send the assistant turn back as it came, tool_use blocks included
            messages.Add(new AssistantMessageParam
            {
                Content = response.Content.Select(ToParam).ToList()
            });

            if (response.StopReason == StopReason.EndTurn)
            {
                return string.Concat(response.Content
                    .Select(block => block.Value)
                    .OfType<TextBlock>()
                    .Select(text => text.Text));
            }

            if (response.StopReason != StopReason.ToolUse)
                throw new InvalidOperationException($"Unexpected stop_reason: {response.StopReason}");

            // Fan-out: execute all tool_use blocks in parallel
            var toolUseBlocks = response.Content
                .Select(block => block.Value)
                .OfType<ToolUseBlock>()
                .ToList();

            var resultTasks = toolUseBlocks.Select(async block =>
            {
                try
                {
                    using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
                    cts.CancelAfter(TimeSpan.FromSeconds(30)); // per-tool timeout

                    var result = await dispatcher(block.Name, block.Input, cts.Token);
                    return new ToolResultBlockParam
                    {
                        ToolUseID = block.ID,
                        Content = JsonSerializer.Serialize(result)
                    };
                }
                catch (Exception ex)
                {
                    // Return a structured error, do NOT throw, let the model recover
                    return new ToolResultBlockParam
                    {
                        ToolUseID = block.ID,
                        Content = JsonSerializer.Serialize(new { error = ex.Message }),
                        IsError = true
                    };
                }
            });

            var toolResults = await Task.WhenAll(resultTasks);

            // All results go back in one user message, one block per tool_use id
            messages.Add(new UserMessageParam
            {
                Content = toolResults.Select(r => (ContentBlockParam)r).ToList()
            });
        }

        throw new InvalidOperationException("Agent exceeded maximum turn limit without reaching end_turn.");
    }

    static ContentBlockParam ToParam(ContentBlock block) => block.Value switch
    {
        TextBlock text => new TextBlockParam { Text = text.Text },
        ToolUseBlock use => new ToolUseBlockParam { ID = use.ID, Name = use.Name, Input = use.Input },
        _ => throw new NotSupportedException($"Block type {block.Value?.GetType().Name} is not sent back here")
    };
}

A few details that matter:

The assistant turn goes back in the history. The response is a Message with ContentBlocks, but the request needs ContentBlockParams, so ToParam converts the text and tool_use blocks. Without them, the tool_result ids in the next message refer to calls the API has never seen. If you turn on extended thinking, the thinking blocks have to go back as well, and ToParam needs a case for them.
The error JSON is serialized, not concatenated. An exception message with a quote in it would otherwise produce invalid JSON in the tool result.
The per-tool timeout uses a linked CancellationTokenSource. A tool that hangs, because a downstream API is slow or a query has no index, would otherwise block the whole turn.

We ran this loop against a local server that answered in the Messages API format: a first response with two tool_use blocks, one for get_order_details and one for a tool that doesn't exist, then an end_turn. The second request carried the assistant turn with both tool_use blocks, followed by one user message with two tool_result blocks. The unknown tool came back with "is_error": true instead of crashing the loop.

Doing It Well: Validation, Errors, and Idempotency

The model follows your schema well, but the input is still untrusted. Validate order_id before it reaches the database, and apply a default when an optional field like include_line_items is missing.

When something goes wrong, return a result the model can reason about: "Order ORD-99999999 not found" is actionable, "NullReferenceException at line 42" isn't. The model will often retry with corrected arguments or explain the failure to the user.

Idempotency matters as soon as you have write tools. If the model retries cancel_order because a timeout made the first result unclear, the handler has to be safe to call twice. Design write tools with idempotency keys or check-then-act semantics from the start. Retrofitting that is painful.

Real Tools: Querying a Database

Here's a dispatcher that routes tool names to typed handlers, including an EF Core query:

public class OrderToolDispatcher
{
    private readonly OrderDbContext _db;
    private readonly ILogger<OrderToolDispatcher> _logger;

    public OrderToolDispatcher(OrderDbContext db, ILogger<OrderToolDispatcher> logger)
    {
        _db = db;
        _logger = logger;
    }

    public async Task<object> DispatchAsync(
        string toolName, IReadOnlyDictionary<string, JsonElement> input, CancellationToken ct)
    {
        _logger.LogInformation("Executing tool {ToolName} with input {Input}",
            toolName, JsonSerializer.Serialize(input));

        return toolName switch
        {
            "get_order_details" => await GetOrderDetailsAsync(
                input.TryGetValue("order_id", out var id) ? id.GetString() ?? "" : "",
                input.TryGetValue("include_line_items", out var ilv)
                    && ilv.ValueKind == JsonValueKind.True,
                ct),
            _ => throw new NotSupportedException($"Unknown tool: {toolName}")
        };
    }

    private async Task<object> GetOrderDetailsAsync(
        string orderId, bool includeLineItems, CancellationToken ct)
    {
        if (!Regex.IsMatch(orderId, @"^ORD-\d{8}$"))
            return new { error = "Invalid order_id format. Expected ORD- followed by 8 digits." };

        IQueryable<Order> query = _db.Orders.AsNoTracking();
        if (includeLineItems)
            query = query.Include(o => o.LineItems);

        var order = await query.FirstOrDefaultAsync(o => o.OrderId == orderId, ct);

        if (order is null)
            return new { error = $"Order {orderId} not found." };

        return new
        {
            order_id = order.OrderId,
            status = order.Status.ToString(),
            estimated_delivery = order.EstimatedDelivery?.ToString("yyyy-MM-dd"),
            line_items = includeLineItems
                ? order.LineItems.Select(li => new { li.Sku, li.Quantity, li.UnitPrice })
                : null
        };
    }
}

Two things in it are easy to get wrong:

A conditional Include has to be built outside the lambda. Include(o => includeLineItems ? o.LineItems : null) looks reasonable and compiles, but EF Core throws InvalidOperationException at runtime, because the include path must be a plain navigation. Add the Include to the query only when you need it.
input is a dictionary of JsonElement. That's what the SDK's ToolUseBlock.Input gives you. Read optional fields with TryGetValue, and check ValueKind instead of calling GetBoolean(), which throws if the model sent the value as a string.

Return anonymous objects or DTOs, which serialize cleanly. Don't leak entity graphs or stack traces into tool results.

Safety and Production Concerns

Least-privilege tools. Each tool should do one thing with only the permissions it needs. A read-only get_order_details tool should connect with a read-only database credential. Don't give a query tool a connection string that can write.

Authentication. Tools run in the context of your application, not the user's session. If your internal API needs a service token, inject it through DI when the tool is registered, and never let it flow through the tool schema or the conversation.

Log every dispatch. You need to know which tools the model called, with which inputs, how long they took and whether they succeeded. Structure those logs (tool name, duration, success or error) so you can alert on anomalies.

Cost. Every tool result goes back into the context window. A tool that returns 50 KB of JSON burns tokens on every later turn. Return the minimum useful data, and if the full result is large, offer a summary tool and a detail tool the model can call selectively.

Human approval for risky actions. For any tool that sends an email, charges a card, cancels an order or changes production data, put a human in the loop. A practical pattern: the tool returns {"status": "pending_approval", "approval_id": "..."}, the agent tells the user, and the action only runs after someone confirms.

Show the model good inputs. The SDK's Tool type also has an InputExamples property, where you can give a few realistic argument sets. For tools with a strict format, like the ORD- ids here, that's a way to show the format instead of only describing it.

Wrapping Up

Building real agentic .NET applications is engineering, not magic. The pieces that get you to a production-ready agent are the ones in this article: the official Anthropic SDK, a tool schema written for the model, a bounded loop that returns every result in one message, structured error results, per-tool timeouts and handlers that are safe to call twice. The rest is domain work: which tools your system actually needs to expose, and which invariants they have to keep.

PostgreSQL jsonb with EF Core 10: Mapping, Indexing and Querying Document Columns

Mehdi Mohseni — Sat, 24 Oct 2026 20:22:04 +0000

The Problem Worth Solving

You have a Product entity with fifteen optional specification fields. Half of them are null for most rows. Adding a new field means a migration. Querying by any combination of them means either a bloated WHERE clause or an EAV table that turns into a JOIN nightmare. PostgreSQL's jsonb column exists for exactly this case, and EF Core 10 with Npgsql.EntityFrameworkCore.PostgreSQL 10 maps it as a real part of the model.

This article covers the mapping, what LINQ translates to, which indexes the planner actually uses, and when a jsonb column beats a separate table. Every snippet here was run against PostgreSQL 18 with Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3, and a few things that look obvious turned out not to work. Those are called out where they come up.

Mapping: Complex Types with `ToJson()`

In EF Core 10 the recommended approach is to map a .NET complex type to a jsonb column with ComplexProperty(...).ToJson(). EF knows the structure of the document, so it can translate property access in LINQ into PostgreSQL JSON operators and it tracks changes to the document.

// Domain types
public record ProductSpecs(string? CPU, int? RamGB, string? FormFactor, decimal? WeightKg);

public class Product
{
    public int Id { get; set; }
    public string Name { get; set; } = default!;
    public decimal Price { get; set; }
    public string Category { get; set; } = default!;  // normalised column, queried constantly
    public ProductSpecs Specs { get; set; } = default!; // goes into jsonb
}

// DbContext configuration
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Product>(b =>
    {
        b.ToTable("products");
        b.Property(p => p.Category).HasMaxLength(100);

        // Emits: specs jsonb NOT NULL
        b.ComplexProperty(p => p.Specs, cp => cp.ToJson("specs"));
    });
}

Pass the column name to ToJson. Without it, the column is called "Specs", with a capital letter and quotes, and every piece of raw SQL you write later has to quote it. With ToJson("specs") the generated table has a plain specs jsonb NOT NULL column.

Before EF Core 10, the same thing was done with owned entities (OwnsOne(p => p.Specs, o => o.ToJson())), which had rough edges around identity and required navigations. Before EF 8, the Npgsql provider had its own POCO mapping that handed the column straight to System.Text.Json, so EF knew nothing about the document. On a current stack, use complex types.

What LINQ Translates To

Property access on the complex type is translated to the ->> operator, with a cast when the property isn't a string:

LINQ expression	Generated SQL
`p.Specs.CPU == "Intel i9"`	`p.specs ->> 'CPU' = 'Intel i9'`
`p.Specs.RamGB > 16`	`CAST(p.specs ->> 'RamGB' AS integer) > 16`

That's convenient, and it matters for indexing, which is the next section.

What doesn't translate is the containment and existence family. EF.Functions.JsonContains, JsonExists and JsonExistAny work on columns mapped as a string or a JsonDocument, but on a complex type mapped with ToJson(), EF 10 throws "could not be translated". For those operators, write the condition in SQL with FromSql and keep composing in LINQ:

// Containment and existence: raw SQL over the mapped entity
var cpu = """{"CPU": "Intel i9"}""";
var highEnd = await db.Products
    .FromSql($"SELECT * FROM products WHERE specs @> {cpu}::jsonb")
    .ToListAsync();

// FromSql composes with LINQ: EF wraps the raw SQL as a subquery
var cheapHighEnd = await db.Products
    .FromSql($"SELECT * FROM products WHERE specs @> {cpu}::jsonb")
    .Where(p => p.Price < 2500)
    .OrderBy(p => p.Price)
    .ToListAsync();

The interpolated {cpu} becomes a parameter, not string concatenation, so this is safe.

One trap with existence checks: EF writes every property of the record into the document, including the null ones. A product with no form factor is stored as {"FormFactor": null, ...}, so specs ? 'FormFactor' is true for every row. If you mean "has a value", test the value:

SELECT * FROM products WHERE specs->>'FormFactor' IS NOT NULL;

Indexing: The Part Most Teams Get Wrong

EF Core doesn't create any index on a JSON column for you, and you can't add one through the model either. b.HasIndex(p => p.Specs) fails when the model is built, because Specs is a complex property. Create the indexes in a migration with migrationBuilder.Sql(...).

GIN with `jsonb_ops` (Default)

Supports @>, ?, ?| and ?&. The general-purpose choice when you query many different keys with those operators.

CREATE INDEX idx_product_specs_gin ON products USING gin (specs);

GIN with `jsonb_path_ops`

For workloads that are almost all containment (@>), jsonb_path_ops gives a smaller and faster index than the default. It doesn't support the existence operators (?, ?|, ?&).

CREATE INDEX idx_product_specs_pathops ON products
    USING gin (specs jsonb_path_ops);

Expression B-Tree for What LINQ Generates

Here is the catch: a GIN index only serves the containment and existence operators. The ->> comparisons that LINQ generates can't use it. We checked the plans with sequential scans disabled: specs @> '{"CPU": "Intel i9"}' used the GIN index, while EF's p.Specs.CPU == "Intel i9" still had no index to use.

For a field you filter on through LINQ, create an expression index that matches the generated SQL. Note the three parentheses: one pair for the index column list and one around the expression, because a cast isn't allowed bare there.

CREATE INDEX idx_product_ram ON products
    (((specs->>'RamGB')::integer));

With this index, the LINQ query p.Specs.RamGB > 16 uses an index scan on idx_product_ram. The same kind of index on (specs->>'CPU') serves the equality query.

PostgreSQL 18 makes virtual generated columns the default, which looks like a neat way to expose a JSON field as a column. They can't be indexed, though: CREATE INDEX on one fails with "indexes on virtual generated columns are not supported". For an indexed field, stay with the expression index or use a STORED generated column.

When `jsonb` Beats a Separate Table

The right mental model is the hybrid: core fields that appear in most queries stay as typed, normalized columns, and the flexible remainder goes into jsonb. Category and Price on Product are normalized because every product-list query filters or sorts on them. Specs is jsonb because it's sparse, varies by category and changes without schema migrations.

jsonb wins when:

Keys are sparse or optional across rows. Per-tenant custom fields in a B2B SaaS product are the classic example. Adding a new custom field is a data change, not a migration.
The main access pattern is containment or existence, not aggregation. SUM((specs->>'WeightKg')::numeric) across millions of rows isn't what jsonb is built for.
The shape changes independently of the core schema. Event payloads, product specifications by category, configuration blobs.
You want to avoid an EAV table. An EAV table with a string-typed value column loses type safety, makes multi-key filters ugly and is often slower than a GIN-indexed jsonb column for the same access pattern.

jsonb loses when a field appears in most queries in a typed, comparable way. If most of your queries filter on specs->>'Status', promote Status to a proper column.

Updates: Whole Document or One Property

SaveChanges writes the whole document. With an immutable record like ProductSpecs, changing one value means assigning a new record (p.Specs = p.Specs with { RamGB = 12 }), and EF sends UPDATE products SET specs = @p0. For small documents that's fine.

For a targeted change, EF Core 10 translates ExecuteUpdate on a property inside the document into jsonb_set, without loading the row:

await db.Products
    .Where(p => p.Id == productId)
    .ExecuteUpdateAsync(s => s.SetProperty(p => p.Specs.RamGB, 32));

The SQL it sends:

UPDATE products AS p
SET specs = jsonb_set(p.specs, '{RamGB}', to_jsonb(@p))
WHERE p."Id" = @productId

The other properties of the document are left as they were.

Practical Checklist

Use Npgsql.EntityFrameworkCore.PostgreSQL 10 with EF Core 10, and map the document with ComplexProperty(...).ToJson("specs").
Give the column a lowercase name in ToJson, or every raw SQL statement has to quote it.
EF.Functions.JsonContains and its relatives don't translate on a complex type. Use FromSql for @> and ?, and compose LINQ on top.
Test values with ->> ... IS NOT NULL, because ? also matches keys stored as null.
Create indexes in a migration with migrationBuilder.Sql. GIN serves @> and ?, and expression B-tree indexes serve the ->> comparisons that LINQ generates.
Don't plan on indexing a virtual generated column in PostgreSQL 18. It isn't supported.
Use ExecuteUpdate with SetProperty(p => p.Specs.X, ...) for single-property changes. EF 10 turns it into jsonb_set.
Always prefer jsonb over json, because binary storage means no reparsing on read.
Promote any field queried on most rows to a typed column. The hybrid model is the goal, not a pure document approach.

Relationships in a Relational Database: MySQL vs NebulaGraph from 1 Million to 500 Million Rows

Mehdi Mohseni — Fri, 23 Oct 2026 21:20:14 +0000

About three years ago I started to build what was really a graph inside a relational database. My thinking at the time was quite simple: every connection between two nodes is a kind of many-to-many relationship, so if I have a table for the nodes and a table for the relationships between them, it should work. When we got to around 500 million rows, nodes and relationships together, the system completely collapsed. We were running it on the largest virtual machine we could get on AWS, which cost about 15 to 20 thousand dollars per month, and a simple query to find one relationship between two nodes was taking about 15 minutes.

After that we moved to graph databases, first ArangoDB, then Neo4j, and now we are using NebulaGraph (I wrote about the move from ArangoDB to NebulaGraph in an earlier article). The same kind of query now takes a few seconds, and at most 7 to 10 seconds, depending on how well the query is optimised.

A story from production is not something you can check, so for this article I built a benchmark that anyone can run on their own machine: the same graph loaded into MySQL and into NebulaGraph, the same questions asked to both, and every answer compared before any timing is trusted. All of the code and the raw results are on GitHub: MehdiMohseni82/graph-vs-relational-benchmark.

The benchmark confirms part of that story, but it also separates two things that the story mixes together, and I think that difference is the most useful thing in this article.

The model most of us start with

When a SQL developer models a graph, the first design is almost always the same, and it's the design I used three years ago. There is one table for the things, and one table for the relationships, where each row says that one node is connected to another node.

CREATE TABLE nodes (
  id   BIGINT      NOT NULL PRIMARY KEY,
  name VARCHAR(32) NOT NULL
) ENGINE=InnoDB;

CREATE TABLE edges (
  src BIGINT NOT NULL,
  dst BIGINT NOT NULL,
  PRIMARY KEY (src, dst)
) ENGINE=InnoDB;

ALTER TABLE edges ADD INDEX ix_edges_dst_src (dst, src);

This is not a bad design, and I want to be fair to it. In InnoDB the primary key is the clustered index, so all the relationships of one node are stored next to each other, and the reverse index (dst, src) does the same for the other direction. Finding the direct neighbours of a node is one index lookup, and it is very fast. The problem doesn't start with one hop, it starts when the question needs many hops.

The questions

The benchmark asks questions that are very common in real systems, for example in fraud detection, recommendations, access control or any knowledge graph. A relationship always counts in both directions.

The relationship question: are node A and node B related, and if yes, how many hops apart are they, up to 6 hops? This is the "simple relationship" query from my own story.
The neighbourhood question: how many different nodes can be reached from this node in 1, 2, 3 or 4 hops?
The neighbourhood without hubs: the same question, but the search may never enter a hub, a node with more than 1,000 relationships. I added this one after looking at a real identity graph, where a few placeholder values are connected to almost everything, and every useful search has to walk around them.

In MySQL 8 the natural way to ask a multi-hop question is a recursive CTE. This is the query for the neighbourhood question:

WITH RECURSIVE reach (id, depth) AS (
    SELECT @start, 0
    UNION
    SELECT e.dst, r.depth + 1 FROM reach r JOIN edges e ON e.src = r.id WHERE r.depth < @hops
    UNION
    SELECT e.src, r.depth + 1 FROM reach r JOIN edges e ON e.dst = r.id WHERE r.depth < @hops
)
SELECT COUNT(DISTINCT id) FROM reach WHERE depth > 0;

And this is the query for the relationship question, which is almost the same:

WITH RECURSIVE reach (id, depth) AS (
    SELECT @a, 0
    UNION
    SELECT e.dst, r.depth + 1 FROM reach r JOIN edges e ON e.src = r.id WHERE r.depth < @maxHops
    UNION
    SELECT e.src, r.depth + 1 FROM reach r JOIN edges e ON e.dst = r.id WHERE r.depth < @maxHops
)
SELECT MIN(depth) FROM reach WHERE id = @b;

In NebulaGraph the first two questions are written in nGQL, and they are much shorter, because traversal is the main job of the language:

GO 1 TO 4 STEPS FROM 42 OVER related BIDIRECT YIELD DISTINCT id($$) AS v | YIELD COUNT(*) AS n;

FIND SINGLE SHORTEST PATH FROM 42 TO 1337 OVER related BIDIRECT UPTO 6 STEPS YIELD path AS p | YIELD length($-.p) AS len;

Why the recursive CTE gets so expensive

It's important to understand what the CTE actually does, because the reason is not that MySQL is slow. Every recursive step joins the whole result of the previous step with the edges table again, twice, once for each direction. UNION removes duplicate rows, but a row here is the pair (id, depth), not the node alone. That means a node that was reached at depth 2 is reached again at depth 4, because you can walk to a neighbour and come straight back, and every time it is expanded again. The CTE has no memory of "I have already visited this node", it only knows "I have not produced this exact row yet".

The relationship question has a second problem, which is even bigger. A recursive CTE cannot stop when it finds B. It first builds every node within 6 hops of A, at every depth, and only after that the outer SELECT looks for B in the result. So the cost of the question "are these two nodes related?" does not depend on how close they are, it depends on how big the neighbourhood of A is.

The third method: a breadth-first search in C

I also added a third method, because I think it is the most honest comparison. It is the solution that a good SQL developer would build after the CTE becomes too slow: keep the same MySQL tables, but do the breadth-first search in C#, one level at a time, with a HashSet of visited nodes and a bidirectional search for the shortest path.

public async Task<long?> ShortestPathAsync(long a, long b, int maxHops, TimeSpan timeout)
{
    if (a == b) return 0;
    var deadline = DateTime.UtcNow + timeout;
    var seenFromA = new HashSet<long> { a };
    var seenFromB = new HashSet<long> { b };
    var frontierA = new List<long> { a };
    var frontierB = new List<long> { b };

    for (int length = 1; length <= maxHops; length++)
    {
        bool expandA = frontierA.Count <= frontierB.Count;
        var (frontier, seen, other) = expandA
            ? (frontierA, seenFromA, seenFromB)
            : (frontierB, seenFromB, seenFromA);
        if (frontier.Count == 0) return null;

        var next = new List<long>();
        foreach (var neighbour in await NeighboursAsync(frontier, deadline, Queries.MySqlNeighboursOf))
        {
            if (other.Contains(neighbour)) return length;
            if (seen.Add(neighbour)) next.Add(neighbour);
        }
        if (expandA) frontierA = next; else frontierB = next;
    }
    return null;
}

NeighboursAsync sends the frontier to MySQL in batches of 2,000 ids and reads the neighbours in both directions with one query per batch:

SELECT dst FROM edges WHERE src IN ({0})
UNION ALL
SELECT src FROM edges WHERE dst IN ({0});

For the third question the BFS uses the same query with one extra condition per direction, NOT EXISTS (SELECT 1 FROM hubs h WHERE h.id = e.dst), so MySQL never returns a hub. The CTE gets the same condition inside its recursion. The exact queries are in the repository.

The setup

Everything runs in Docker on one machine, and the repository has a runbook and a script that runs one size from empty databases to the final table.

Hardware: Mac Studio, Apple M3 Ultra, 96 GB RAM. Docker had 80 GB of memory and 24 CPUs, and nothing else was running.
The same memory for both engines: MySQL 8.4 in a 30 GB container with a 24 GB InnoDB buffer pool. NebulaGraph 3.8 with 30 GB in total: the storage service in a 24 GB container with a 16 GB RocksDB block cache, and the graph service in a 6 GB container. One storage node and 36 partitions.
Loading: MySQL with LOAD DATA INFILE and the reverse index built after the load. NebulaGraph with nebula-importer from exactly the same CSV files, followed by a compaction.
The data: a deterministic generator in C#. Every node gets on average 9 relationships, and half of the targets come from a skewed distribution, so some nodes become hubs. The sizes are 1M, 10M, 100M and 500M rows, nodes and relationships together, which is the way I counted the rows in my own system.
The measurement: 10 cases per question, one warm-up run and 3 measured runs, and the tables show the median. Half of the pairs for the relationship question are close to each other, the other half are two random nodes. The timeout is 15 minutes per query.
Validity: after every size the run checks that no container was killed or restarted, that the kernel killed nothing for lack of memory, that no query returned an engine error, and that all methods returned the same answers. Every size passed.

Results: the relationship question

Size (rows)	MySQL recursive CTE	MySQL + BFS in C#	NebulaGraph
1M	5.68 s	1.1 ms	2.1 ms
10M	1.2 min	1.6 ms	18.7 ms
100M	9.7 min, 2 of 10 timed out	1.5 ms	8.6 ms
500M	no answer in 15 minutes (10 of 10)	2.7 ms	23.6 ms

This is the result from my story. At 500 million rows the recursive CTE didn't answer a single one of the ten pairs within 15 minutes, while the other two methods answered in milliseconds. The pairs were between 2 and 6 hops apart, so this is not a question of a long path, it's the CTE building the whole 6-hop neighbourhood before it looks for B.

But look at the other two columns. The bidirectional search in C#, on the same MySQL tables, was faster than NebulaGraph at every size. So for this question, the problem was not the relational database. It was the recursive CTE.

Results: the neighbourhood question

Size (rows)	MySQL recursive CTE	MySQL + BFS in C#	NebulaGraph
1M	180 ms	177 ms	72.4 ms
10M	2.65 s	508.5 ms	288.1 ms
100M	10.61 s	1.70 s	1.13 s
500M	3.18 s	579.6 ms	529.1 ms

These are 4 hops. At 1, 2 and 3 hops all three methods answer in at most about 130 ms at every size, and at 1 and 2 hops in a few milliseconds or less.

Two things are worth explaining here. First, NebulaGraph is the fastest at every size, but the BFS in C# is never far behind, between 1.1 and 2.4 times slower, and the CTE is between 2.5 and 9.4 times slower than NebulaGraph.

Second, 500M is faster than 100M for every method, which looks wrong. It isn't: the time of this query depends on how many nodes it reaches, not on the size of the table. With 10 million nodes, the median start node reached 3.9 million nodes in 4 hops, but with 50 million nodes the median start node reached 2.3 million, because the same number of relationships is spread over five times more nodes. Less work, less time, for all three engines in the same way.

Results: a graph with hubs

The graph above is friendly. Real graphs often have hubs, and in an identity graph a single placeholder value, like a default phone number, can be connected to hundreds of thousands of records. So I generated a second graph: 1 million nodes with on average 50 relationships, plus 20 hubs connected to about 630,000 nodes each. 904 nodes have more than 1,000 relationships. It's 63.5 million rows, and the timeout here is 5 minutes.

The first thing this graph shows is why hubs matter: from a median start node, 2 hops reach 999,991 of the 1,000,000 nodes. Every search goes through a hub, and after that it has the whole graph.

Question	MySQL recursive CTE	MySQL + BFS in C#	NebulaGraph
relationship, up to 6 hops	no answer in 5 minutes (10 of 10)	0.5 ms	1.1 ms
neighbourhood, 2 hops	8.49 s	1.07 s	575.6 ms
neighbourhood, 3 hops	3.6 min	20.58 s	5.95 s
neighbourhood, 4 hops	no answer in 5 minutes (10 of 10)	20.59 s	11.27 s

Here the difference between the BFS in C# and NebulaGraph becomes real: when a query touches the whole graph, NebulaGraph is 1.8 to 3.5 times faster. The BFS has to pull every level of the search over the network into the application, batch by batch, and NebulaGraph does the same work next to the data.

But nobody wants the whole graph. The useful question on this graph is the third one, the neighbourhood that walks around the hubs:

Hops	MySQL recursive CTE	MySQL + BFS in C#
2	6.8 ms	5.5 ms
3	2.55 s	479.6 ms
4	2.6 min	22.91 s

Avoiding the hubs, the median start node reaches about 12,000 nodes in 2 hops and 730,000 in 3 hops, so this is still a big search, but it is one the user actually asked for.

This question shows the most about NebulaGraph, because I couldn't write it as one nGQL statement that is both correct and usable:

GO 1 TO 3 STEPS ... WHERE NOT "hub" IN tags($$) runs, but it returned more nodes than the SQL version. In our check on a small graph it found 5,821 nodes at 2 hops where the correct answer was 463, because the filter doesn't stop the walk from going through a hub.
MATCH with a condition that no node on the path is a hub returns the correct answer, but it enumerates paths, and on a 20,000-node test graph the 4-hop version used up the 1 GB the graph service had in that test.

What an application does in this case is to walk one GO step per level from C# and remove the hubs between the levels. That is the same small graph engine in the application as the BFS on MySQL, just on top of a different database. I measured a version of it, but my implementation filtered the hubs with WHERE NOT "hub" IN tags($$) in every step, which makes NebulaGraph read the tags of every destination node, and a version that filtered in C# was about twice as fast in a small test. Because I don't have a fair measurement of that approach at full size, I leave NebulaGraph out of this table rather than show a number that makes it look worse than it is.

What the numbers say

I think the honest summary has three parts.

The recursive CTE is what breaks. In every table it is the slowest method, from about 1.2 times slower on the smallest searches to several thousand times slower on the relationship question, and at 500M rows it didn't answer a single relationship question within 15 minutes. It can't remember visited nodes, can't search from both sides and can't stop early. The query in my story was this kind of relationship question, and the benchmark suggests that the way such a query is written matters at least as much as where the data lives.

A breadth-first search on MySQL is a real alternative, and for paths it was the fastest. With the same tables and the same indexes, the BFS in C# answered the relationship question in under 3 milliseconds at every size, faster than NebulaGraph's FIND SHORTEST PATH, and on the 4-hop neighbourhood it stayed within 2.4 times of NebulaGraph on the normal graph. On the hub-avoiding question it was more than five times faster than the CTE at 3 hops. But it's important to be clear what it is: you are writing a small graph engine inside your application, with the visited set, the frontier, the batching, the bidirectional search and the timeouts, and every new kind of question means more code.

A graph database pays off when the traversal itself is big. On the hub graph, when a query really had to walk a large part of the graph, NebulaGraph was 1.8 to 3.5 times faster than the BFS, because it does the work next to the data instead of pulling every level into the application. It also gives you a language where the first two questions are one line each. What it didn't give me, in this test, is a single statement for a traversal that has to avoid certain nodes, and that is exactly the kind of question real graphs need.

Limits of this benchmark

It runs on one machine with local disks and one NebulaGraph storage node. NebulaGraph is designed to spread partitions over many machines, and this test doesn't measure that at all. If your graph doesn't fit on one machine, that is a different question, and a benchmark on one machine can't answer it.
The data is generated. It has the right shape (skewed degrees, hubs), but it is not production data with production query patterns and other traffic.
The MySQL schema and queries are the common approach, not the only one. If you can make the relational side faster, a pull request with the numbers is very welcome.

When a relational database is still the right choice

If your queries are one or two hops, for example "the orders of this customer" or "the members of this group", a relational database with the right indexes is very fast, much simpler to operate, and you already have it. In every table above, 1 and 2 hops on the normal graph stayed in the range of a few milliseconds for all three methods.

In my view, the signal to look at a graph database is the shape of the questions, not the number of rows. When you start to write recursive CTEs, first replace them with a search in your application: in this benchmark that alone turned minutes into milliseconds. When the traversals become large, when the business keeps asking new kinds of graph questions, or when the graph grows beyond one machine, that's the point where a graph database earns its place.

Run it yourself

The repository has everything needed to reproduce these numbers: the Docker Compose file for MySQL and NebulaGraph, the C# generator, the C# benchmark with all methods, the runbook with the validity checks, and the raw CSV of every single run.

github.com/MehdiMohseni82/graph-vs-relational-benchmark

Are Coding Agents Getting Worse, or Have They Lost Their Teachers?

Mehdi Mohseni — Tue, 20 Oct 2026 07:10:46 +0000

When coding agents like Claude Code, Codex and Copilot's agent mode became part of the daily work of many teams, the first reactions were very positive, and a lot of developers were surprised how well an agent could add a feature to an existing project. Now, one or two years later, a different complaint is very common in discussions between developers: the code that the agents write seems to be getting worse. It has more duplication, the structure doesn't follow the architecture of the project, and fixes are added next to old code instead of improving it. The most popular explanation is that the models themselves are getting worse, because the vendors save money on compute, because of new safety restrictions, or because a newer model is simply less careful than the one before.

In my view the model is not the main reason in most of these cases, and the real reason is closer to us. When teams started to use agents, almost all of the code in their repositories had been written by humans and had passed through quality gates like pull requests, peer review and static analysis tools such as SonarQube. The agent was working inside a codebase of reviewed code, and that code was the example it learned from while it worked. Over time, more and more of the codebase has been written by the agent itself, and in many teams with fewer of those gates, so the agent is now learning more and more from its own unreviewed output. In this article I want to look at what the research actually says about this, both the part that supports this view and the part that doesn't, and then at what a team can do about it.

First, is the code really getting worse?

Before looking for a cause it's important to check if the effect is real, because developers are not always good at judging this from their own experience. The METR study from July 2025 is a good warning here: experienced open source developers expected AI tools to make them 24% faster, the measured result was that they took 19% longer, and after the study they still believed the tools had made them about 20% faster [14]. Our feeling about AI and productivity, and probably also about AI and quality, is not a reliable measurement.

There are, however, several measurements that point in the same direction. GitClear analysed about 153 million changed lines of code from 2020 to 2023 and found that the share of "moved" lines, which is their signal for refactoring, dropped from 25.0% in 2020 to 16.9% in 2023, while copy/pasted lines grew from 8.3% to 10.5% and churn, code that is changed again shortly after it was written, grew from 3.3% to 5.5% [1]. Their follow-up report with 211 million lines found that refactored lines fell to less than 10% in 2024 and that copy/pasted code was higher than moved code for the first time in their data [2]. It's important to say that GitClear itself only shows a correlation with the adoption of AI assistants, not a cause, but the direction is very clear.

Google's DORA research gives a similar picture from a different angle. The 2024 report estimated that for every 25% increase in AI adoption, delivery stability decreased by 7.2% and throughput by 1.5%, even though code quality as reported by the respondents went up by 3.4% [3]. In the 2025 report the relationship with throughput became positive, but AI adoption still had a negative relationship with delivery stability, and the authors summarised their finding in one sentence that is very relevant for this article: "AI doesn't fix a team; it amplifies what's already there" [4].

Reports that compare AI-written and human-written changes directly point the same way. CodeRabbit analysed 470 open source pull requests and found 10.83 issues per AI co-authored pull request against 6.45 per human one, which is about 1.7 times more, with readability issues more than three times as common [5]. Sonar evaluated leading models on more than 4,400 Java tasks and found that more than 90% of the issues were code smells, the kind of problem that doesn't break a test today but makes the code more difficult to change tomorrow [6]. So yes, there is a real problem, and it is mostly a problem of maintainability and structure, not of code that doesn't compile.

Is it the model?

Sometimes, yes, and it would be dishonest to write this article without saying that. In September 2025 Anthropic published a postmortem about three infrastructure bugs that degraded Claude's responses in August and early September of that year. One of them was a routing error that at its peak affected 16% of Sonnet 4 requests, and about 30% of Claude Code users had at least one message sent to the wrong server configuration [7]. Users who said that Claude had become worse in those weeks were right, although the reason was not the model weights but the serving infrastructure. Anthropic also wrote that they "never reduce model quality due to demand, time of day, or server load", and that their own evaluations "simply didn't capture the degradation users were reporting" [7].

The other well-known example is the 2023 paper by Chen, Zaharia and Zou, which reported that the share of directly executable code generated by GPT-4 dropped from 52.0% in March to 10.0% in June [8]. This result was shared very widely as proof that the models were getting worse, but the authors themselves explained that the June model had started to wrap its code in markdown code fences, which made it "not executable" for their test, and Narayanan and Kapoor pointed out that the study didn't check if the code was correct at all [9]. What changed was the behaviour and the format, not the ability to write code.

Both examples are useful, because they show what a model or platform problem usually looks like: it starts at a specific moment, it affects many users at the same time, and when it's found it can be fixed or explained. The complaint about agent code quality is different. It is a slow decline, it's described by teams working in their own codebases over months, and it continues across several model releases that are better than their predecessors on every public benchmark. That pattern is very difficult to explain with the model alone, so it makes sense to look at the other input the agent uses: the code it's working in.

The codebase is the agent's example

An agent doesn't write code in a vacuum. Before it changes anything it reads files around the place where it works, it searches for similar code, and it follows what it finds. Anthropic's own best practices for Claude Code recommend exactly this as a technique: "Point Claude to patterns in your codebase", with an example where the developer tells the agent which existing widget is a good example to follow [13]. The same document also says that CLAUDE.md should not contain "anything Claude can figure out by reading code", which assumes that the agent learns the conventions of a project from the code itself [13]. That is a strength when the code around it is good, but the same mechanism works in the other direction too.

Research shows quite clearly that the code in the context has a strong effect on what a model writes. Dinh and his colleagues tested code models with partial code that contained small potential bugs, for example a += that was changed to -=. With clean partial code, CodeGen-2B reached a pass rate of 54.9%, but with the buggy partial code it dropped to 3.1%, which was even worse than the 9.3% it reached with no partial code at all [10]. The authors' explanation is that the model "fails to react" to the problem in most of the failed cases, so it doesn't copy the bug on purpose, it continues the code it was given as if it was correct. These were small models from 2023, but the direction of the effect is what matters for this article.

For security the effect is similar. Snyk demonstrated in 2024 that GitHub Copilot suggested a parameterised SQL query in a clean project, but suggested string concatenation, which means SQL injection, after vulnerable code was placed in a neighbouring open file that Copilot uses as context [11]. That was a demonstration and not a measured study, but a peer-reviewed study by Asare, Nagappan and Asokan found that when Copilot was given the situations where human developers had introduced real vulnerabilities, it reproduced the vulnerable code about 33% of the time [12]. Their main conclusion was actually positive for Copilot, which was "not as bad as human developers", but it also confirms that the surrounding code shapes the suggestion.

So the code that is already in the repository works like a very large prompt that we never write explicitly. When that prompt was written by people and reviewed by other people, the agent continued good code. When the prompt is more and more the agent's own earlier output, the agent continues that instead.

The loop: when the agent learns from itself

Now let's examine what happens over time. In the first months of using an agent, almost all of the code it reads was written by humans, and every line had passed review, static analysis and the discussions in pull requests. The agent copies patterns that people agreed on. Every change the agent makes adds a small amount of its own code to the repository, and if that code is reviewed carefully, nothing changes. But in practice many teams review agent code less carefully than human code, because there is much more of it, because it looks clean and confident, and because the tests are green. Every unreviewed change becomes part of the example for the next change, so small deviations from the architecture are not corrected, they are copied and multiplied.

This is not a new idea in software engineering. Already in 1980 Lehman described the law of increasing complexity: "As an evolving program is continually changed, its complexity, reflecting deteriorating structure, increases unless work is done to maintain or reduce it" [15]. In 1997 he and his colleagues added the law of declining quality, which says that the quality of such systems "will appear to be declining unless they are rigorously maintained" [16]. Hunt and Thomas called the same thing software entropy and gave the advice "Don't live with broken windows", because one bad design that nobody fixes makes the next one easier to accept [18]. The agent doesn't change these laws. It only makes the changes much faster and in much larger numbers, so the decline that took a human team years can happen in months.

There is a related effect in model training that is often mentioned in these discussions. Shumailov and his colleagues showed in Nature that models trained again and again on data generated by models suffer "irreversible defects", and the rare parts of the original distribution disappear [17]. I want to be careful here, because that research is about training new weights, while in our case the weights don't change at all, only the code the agent reads changes. So model collapse is an analogy and not an explanation, but the shape is the same: a system that learns from its own output without an external reference slowly loses the quality of the original.

The most direct evidence for this loop at the level of a project is a 2026 preprint by Zhu, Tsantalis and Rigby, who analysed projects generated by LLMs and by a multi-agent system. In the multi-agent projects the total number of lines correlated with architectural smells at ρ=0.94, and the authors write that "neither functional correctness nor detailed prompting mitigates this decay" [19]. To be fair, the same study also found smells that humans produced and the models didn't, and it only tested one multi-agent framework on 90 programming problems, so it is not a final answer. As far as the published research goes, there is no study yet that measures exactly what this article describes: the share of agent-written code in a real repository over time and the quality of the next change. That is a gap in the research, and until somebody fills it, this explanation is a hypothesis that fits the evidence, not a proven fact.

A child who learns from the people around it

The picture I find most helpful is a child who is growing up. A child learns a very large part of its behaviour by looking at the adults around it, how they speak, how they solve problems and how they react when something goes wrong. When a child grows up with parents who show it good examples and correct it when it goes in the wrong direction, it learns those examples. When a child is left alone, it doesn't become less intelligent, but it has nothing to compare its behaviour with, and it develops its own habits, good and bad, without anybody noticing.

An agent in a codebase is in a very similar situation. At the beginning the reviewed human code was the parent: it showed which patterns to use, where each responsibility belongs and how errors are handled in this project. The pull request reviewer and the static analysis tool were the correction. When we stopped reviewing carefully and the agent wrote more and more of the code, we didn't make the agent less capable, but we took away its examples and its corrections, and then we were surprised that it developed its own habits.

What a team can do

If this explanation is right, the good news is that the solution is mostly in our hands and doesn't depend on the next model release. These are the steps that follow from the evidence above.

Keep the same quality gates for agent code as for human code. A pull request written by an agent should be reviewed by a human with the same care, and static analysis such as SonarQube or the .NET analyzers should run on it in the build, not only on the human changes. Anthropic's documentation says it very directly: "If you can't verify it, don't ship it" [13].
Write the architecture down where the agent reads it. Decisions that live only in the heads of senior developers can't be learned from the code, especially when the code has started to drift. A short CLAUDE.md or AGENTS.md file, architecture decision records and explicit rules about where things belong give the agent a reference that doesn't change with every commit.
Point the agent to reference code on purpose. Choose a few modules that were written and reviewed by humans and are good examples, and tell the agent to follow them, as the Claude Code documentation recommends [13]. Protect these modules from casual changes, because they are the parents in the analogy above.
Make the rules deterministic where you can. Instructions in a markdown file are advice, and Anthropic's documentation describes hooks as the deterministic alternative [13]. In .NET, analyzers with build errors, architecture tests and code style rules that are enforced in the build do the same job: the agent can't ignore them, and they don't get tired after the fiftieth pull request.
Review agent code with a fresh context. The Claude Code documentation notes that a new session reviews code better because it "won't be biased toward code it just wrote" [13]. A second agent or a human reviewer who didn't see the conversation is a better check than the same agent reviewing itself.
Measure the trend in your own repository. Duplication, churn and the share of refactoring changes are exactly the signals GitClear used [1][2], and you can measure them in your own repository over time. If you also track which changes were written by an agent, you can check this hypothesis on your own code instead of trusting my opinion or anybody else's.
Plan time for refactoring. Lehman's law says that complexity grows "unless work is done to maintain or reduce it" [15]. Agents are very good at refactoring when they are asked to do it with clear rules, so the same tool that creates the drift can help to remove it, but somebody has to decide that it's needed.

Conclusion

Whether we like it or not, the share of code that agents write will continue to grow, and the question is not whether we use them but what we give them to learn from. The models are not perfect, and sometimes the platform really does get worse for a few weeks, as Anthropic's postmortem shows. But when the quality of agent code declines slowly over months in the same repository, while the models get better on every benchmark, it makes sense to look at the codebase first, because that is the example the agent copies every day. In short, if we want the agent to write code like a careful senior developer, the codebase has to keep looking like the work of careful senior developers, and that is still our job.

References

Harding, W. and Kloster, M. Coding on Copilot: 2023 Data Shows Downward Pressure on Code Quality, GitClear, January 2024. https://www.gitclear.com/coding_on_copilot_data_shows_ais_downward_pressure_on_code_quality
GitClear. AI Copilot Code Quality: 2025 Look Back at 12 Months of Data, 2025. https://www.gitclear.com/ai_assistant_code_quality_2025_research
Harvey, N. and DeBellis, D. Announcing the 2024 DORA report, Google Cloud blog, 23 October 2024. https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report
Harvey, N. and DeBellis, D. Announcing the 2025 DORA Report: State of AI-Assisted Software Development, Google Cloud blog, 23 September 2025. https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report
Loker, D. AI vs Human Code Generation Report, CodeRabbit, 17 December 2025. https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report
Sonar. The Coding Personalities of Leading LLMs, August 2025. https://www.sonarsource.com/the-coding-personalities-of-leading-llms/
Anthropic. A postmortem of three recent issues, Anthropic Engineering, 17 September 2025. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues
Chen, L., Zaharia, M. and Zou, J. How is ChatGPT's behavior changing over time?, arXiv:2307.09009, 2023. https://arxiv.org/abs/2307.09009
Narayanan, A. and Kapoor, S. Is GPT-4 getting worse over time?, AI Snake Oil, 19 July 2023. https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time
Dinh, T. et al. Large Language Models of Code Fail at Completing Code with Potential Bugs, NeurIPS 2023. https://arxiv.org/abs/2306.03438
Degges, R. Copilot amplifies insecure codebases by replicating vulnerabilities in your projects, Snyk Labs, 22 February 2024. https://labs.snyk.io/resources/copilot-amplifies-insecure-codebases-by-replicating-vulnerabilities/
Asare, O., Nagappan, M. and Asokan, N. Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?, Empirical Software Engineering, 2023. https://arxiv.org/abs/2204.04741
Anthropic. Best practices for Claude Code, Claude Code documentation, accessed 19 October 2026. https://code.claude.com/docs/en/best-practices
METR. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 10 July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
Lehman, M. M. Programs, Life Cycles, and Laws of Software Evolution, Proceedings of the IEEE 68(9), September 1980, pp. 1060-1076.
Lehman, M. M., Ramil, J. F., Wernick, P. D., Perry, D. E. and Turski, W. M. Metrics and Laws of Software Evolution: The Nineties View, Proceedings of the 4th International Software Metrics Symposium, 1997, pp. 20-32.
Shumailov, I. et al. AI models collapse when trained on recursively generated data, Nature 631, 755-759, 2024. https://www.nature.com/articles/s41586-024-07566-y
Hunt, A. and Thomas, D. The Pragmatic Programmer, Addison-Wesley, 1999, section "Software Entropy". Tip list: https://pragprog.com/tips/
Zhu, Tsantalis and Rigby. AI-Generated Smells: An Analysis of Code and Architecture in LLM- and Agent-Driven Development, arXiv:2605.02741, 2026 (preprint). https://arxiv.org/abs/2605.02741

The Narrow Sidewalk: Where Developers Stand When Agents Write the Code

Mehdi Mohseni — Sun, 18 Oct 2026 10:44:39 +0000

When coding agents such as Claude Code and Codex became good enough to take whole tasks, the first reaction in our industry was very loud: developers would be replaced, and soon. A year later the picture is more interesting than that. Agents write a large share of the code now, and that is not going back. But it is also becoming clear that someone still has to know what correct code looks like, and that this someone is a developer with a critical mindset. In my view we are close to the point where the balance between agents and humans becomes visible, and the honest description of that balance is a narrow sidewalk.

On one side of the sidewalk is the developer who keeps working as before, writing every line by hand and treating agents as a toy. On the other side is the developer who hands everything to the agent and stops learning, because the code compiles and the tests are green. Both of them fall off. The path in the middle is narrow, and in this article I want to describe what walking on it actually requires, with real code.

What the Numbers Say About Both Sides

METR ran a randomized controlled trial in early 2025 with 16 experienced open-source developers working on 246 real issues in their own repositories. With AI tools they took 19% longer. The more interesting part is the perception: before the study they expected AI to make them 24% faster, and even after the slowdown they still believed it had made them 20% faster. So our feeling about how much the agent helps is not a reliable measurement.

In February 2026 METR reported that it had to change the design of its follow-up study, mainly because many developers no longer wanted to work without AI at all, and that developers are likely faster now than in early 2025. Taken together, these two results describe the sidewalk very well. Going back to the old way of working is not realistic, but trusting our own impression of the agent's output is also not safe.

Green Checks, Wrong Meaning

When coding agents work on real systems, one failure shows up again and again. The code type-checks, the tests pass, the build is green, and the behaviour is still wrong, because the mistake is in the meaning and not in the syntax. A check can only verify what somebody told it to verify, and the agent writes both the code and, very often, the checks.

Let us take a concrete example. An endpoint asks an external access service which permissions a user has, and refuses the request when the user has none. A typical generated version looks like this:

public interface IPermissionClient
{
    Task<IReadOnlySet<string>> GetPermissionsAsync(string userId, CancellationToken ct);
}

public sealed class NaiveAccessCheck(IPermissionClient client)
{
    public async Task<IResult> CheckAsync(string userId, CancellationToken ct)
    {
        IReadOnlySet<string> permissions;
        try
        {
            permissions = await client.GetPermissionsAsync(userId, ct);
        }
        catch (HttpRequestException)
        {
            permissions = new HashSet<string>(); // fail closed
        }

        return permissions.Count == 0
            ? Results.StatusCode(StatusCodes.Status403Forbidden)
            : Results.Ok();
    }
}

It compiles, it reads naturally, and it even carries a comment about failing closed, which is the correct security instinct. A test that gives the stub an empty permission set and expects 403 passes. The problem is that an outage of the access service also produces an empty set, so during an outage every user gets a 403 and the UI tells them they have no access. They contact support about permissions, and nobody looks at the access service, which is the thing that is actually broken.

Seeing this requires knowing that "the user is refused" and "the authority could not be asked" are two different answers, and that they lead to different status codes, different messages and different alerts. No type and no test inside this class encodes that knowledge. The fix is to make the difference impossible to lose:

using System.Diagnostics;

public abstract record AccessDecision
{
    public sealed record Allowed(IReadOnlySet<string> Permissions) : AccessDecision;
    public sealed record Denied(string Reason) : AccessDecision;
    public sealed record Unavailable(Exception Cause) : AccessDecision;
}

public sealed class AccessCheck(IPermissionClient client)
{
    public async Task<AccessDecision> CheckAsync(string userId, CancellationToken ct)
    {
        try
        {
            var permissions = await client.GetPermissionsAsync(userId, ct);
            return permissions.Count == 0
                ? new AccessDecision.Denied("No permissions granted")
                : new AccessDecision.Allowed(permissions);
        }
        catch (HttpRequestException ex)
        {
            return new AccessDecision.Unavailable(ex);
        }
    }
}

public static class AccessResults
{
    public static IResult ToHttpResult(AccessDecision decision) => decision switch
    {
        AccessDecision.Allowed => Results.Ok(),
        AccessDecision.Denied d => Results.Problem(d.Reason, statusCode: StatusCodes.Status403Forbidden),
        AccessDecision.Unavailable => Results.Problem("Access service unavailable",
            statusCode: StatusCodes.Status503ServiceUnavailable),
        _ => throw new UnreachableException()
    };
}

Now a refusal and an outage are different types, and the switch makes the caller handle each one. An outage becomes a 503, which a monitoring system will page on, instead of a 403, which nobody pages on.

Fail-Open Defaults Hide Behind Healthy Deployments

The second pattern is a check that passes because it was never active. Verification code is often written so that it is skipped when its settings are missing, which is convenient on a developer machine:

using System.ComponentModel.DataAnnotations;
using Microsoft.Extensions.Options;

public sealed class TokenOptions
{
    [Required] public string Issuer { get; set; } = "";
}

public sealed class NaiveTokenCheck(IOptions<TokenOptions> options)
{
    public bool IsTrusted(string? issuer)
    {
        if (string.IsNullOrEmpty(options.Value.Issuer))
            return true; // verification not configured yet

        return issuer == options.Value.Issuer;
    }
}

If the deployment never passes the issuer setting, every token is trusted, the service starts, health checks are green and nothing in the logs looks wrong. Only a negative test against the deployed service, a request that should be refused, shows that nothing is checked. In .NET the better default is to refuse to start at all when security settings are missing:

public static class TokenOptionsSetup
{
    public static IServiceCollection AddTokenOptions(this IServiceCollection services)
    {
        services.AddOptions<TokenOptions>()
            .BindConfiguration("Token")
            .ValidateDataAnnotations()
            .ValidateOnStart();
        return services;
    }
}

ValidateOnStart runs the data annotation validation when the host starts, so a missing issuer becomes an OptionsValidationException at startup instead of a silent hole in production.

Test Both Directions

Both fixes are only worth something if the tests prove the negative case as carefully as the positive one. These are the tests that matter here, and the outage test is the one a generated suite usually leaves out:

using Microsoft.AspNetCore.Http.HttpResults;
using Microsoft.Extensions.Options;

sealed class StubClient : IPermissionClient
{
    private readonly IReadOnlySet<string>? _permissions;
    private readonly Exception? _error;

    public StubClient(IReadOnlySet<string> permissions) => _permissions = permissions;
    public StubClient(Exception error) => _error = error;

    public Task<IReadOnlySet<string>> GetPermissionsAsync(string userId, CancellationToken ct) =>
        _error is null
            ? Task.FromResult(_permissions!)
            : Task.FromException<IReadOnlySet<string>>(_error);
}

public class AccessCheckTests
{
    [Fact]
    public async Task Refused_user_gets_403()
    {
        var check = new AccessCheck(new StubClient(new HashSet<string>()));
        var result = AccessResults.ToHttpResult(await check.CheckAsync("u1", default));
        Assert.Equal(403, ((IStatusCodeHttpResult)result).StatusCode);
    }

    [Fact]
    public async Task Outage_gets_503_not_403()
    {
        var check = new AccessCheck(new StubClient(new HttpRequestException("down")));
        var result = AccessResults.ToHttpResult(await check.CheckAsync("u1", default));
        Assert.Equal(503, ((IStatusCodeHttpResult)result).StatusCode);
    }

    [Fact]
    public async Task Missing_token_settings_fail_at_startup()
    {
        var builder = Host.CreateApplicationBuilder();
        builder.Services.AddTokenOptions();
        using var host = builder.Build();
        await Assert.ThrowsAsync<OptionsValidationException>(() => host.StartAsync());
    }
}

All the code in this article was run on .NET 10 before publishing, including the naive versions, to confirm that the first one really answers 403 during an outage and the second one really trusts any issuer when it is not configured.

The Same Patterns, Again and Again

The two examples above are not special. In practice, the mistakes that survive into production with agent-written code tend to follow a small number of shapes:

Tests mock exactly the boundary where the bug lives. Each side of an integration is tested against a stub of the other side, so a mismatch between them, for example 401 where 403 was meant, is invisible to both suites.
The premise is wrong, and every check agrees with it. A feature flag read with the wrong polarity, or a coordinate system assumed to point the other way, produces code and tests that are perfectly consistent with each other and wrong about the world.
A green check measures less than its name. A release step joined with ; instead of && ignores the CI result it was supposed to wait for. A test that discovers components with a type check can silently skip the ones that do not match, so coverage shrinks while every run stays green.
The producer is verified instead of the consumer. Checking a value in the data you serve proves nothing if the consumer reads a different copy of it.
Readable and proportionate are not testable. A page can be correct and fully tested and still show thirty nearly identical rows for one file.
A guardrail stops the agent. When a policy blocks an action, the right move is to give the decision back to a human, not to find another route around the block.

What these have in common is that none of them is caught by more automation alone. They are caught by someone who knows what the system is supposed to mean: the domain, the architecture, and who reads what.

Walking the Sidewalk

So what does staying on the sidewalk look like in practice? In my view it has two parts, and both of them need continuous learning.

The first part is to keep learning how code works. Not because we will write most of it by hand again, but because the knowledge of HTTP semantics, failure modes, type design and the platform we work on is exactly what lets us recognise when a plausible answer is wrong. A developer who stops learning here can no longer review the agent, and then the agent is not supervised at all.

The second part is to learn how to work with the agent: how to give it the context and constraints it cannot infer, how to put checks around it that test the negative case, and how to critique its output instead of accepting a green build as proof. That is a skill of its own, and it is still very young.

The developers who will do well in the next years are, I think, the ones who keep both of these going at the same time. The sidewalk is narrow, but it is the only path that does not end in either irrelevance or a production incident that nobody can explain.

Why Agents Need a Knowledge Base: The Gaps an LLM Can't Fill by Itself

Mehdi Mohseni — Fri, 16 Oct 2026 16:26:16 +0000

When a team starts building an agent, the first version is very often only a model, a system prompt and a few tools, and for a demo that is usually enough, because the questions in a demo are general questions and a large language model knows a surprising amount about general things. The problems start when the agent is put in front of real work and the questions become specific: which services depend on this database, what we decided about retries last spring, whether the version of a library we use already has the fix, or how to apply a rule from a specialised engineering field that the model only knows from the textbook level. In my view, when an agent disappoints, the reason is very often that the model is asked to know things it never had any chance to learn, much more often than any weakness of the model itself. In this article I want to explain what those gaps are, why a bigger context window doesn't close them, and what role a knowledge base plays in an agentic system, so that the more technical articles on dotnet-talk.com about RAG, agent memory and knowledge graphs have a place to start from.

What the model actually knows

Everything a model knows on its own is stored in its weights, and it got there during training. The authors of the original RAG paper, Patrick Lewis and his colleagues at Facebook AI Research, called this parametric memory, and already in 2020 they pointed out the two problems with it which are still the important ones today: for these models, "providing provenance for their decisions and updating their world knowledge remain open research problems" [1]. Their answer was to combine the parametric memory of the model with a non-parametric memory, which in their case was a dense vector index of Wikipedia that a retriever searched before the model generated an answer.

That distinction is very useful, because it gives us a clean way to think about what belongs where. Parametric memory is very good at language, reasoning, code patterns and widely published general knowledge. Non-parametric memory, which is the knowledge base, is for everything that is specific, private, recent or has to be traceable to a source.

The first gap: time

Every model has a date after which it knows nothing, and model providers publish that date. Anthropic's model overview, for example, lists two dates for each model: a reliable knowledge cutoff, which it describes as "the date through which the model's knowledge is most extensive and reliable", and a training data cutoff, which is the broader range of data that was used [2]. For the whole current Claude lineup, from Haiku 5.5 to Fable 5.1, both dates are June 2026 [2].

That sounds recent, and it is, but it also means that a model you call today in October 2026 has never seen anything from the last four months: not the NuGet packages released since then, not the breaking changes in the SDK you upgraded last week, and not the security advisory that was published yesterday. Additionally, the knowledge close to the cutoff is the thinnest, because the internet had not yet written much about those events when the data was collected. So the newest things are exactly the things the model knows least well, and these are very often the things a developer is asking about.

The second gap: knowledge that is too specialised

The gap that surprises people most is not about time at all. Many fields are old and stable, so the cutoff doesn't matter for them, and still the model's knowledge of them is very thin, simply because not much has been written about them in public. A model learns a fact roughly in proportion to how often that fact appears in its training data, and this has been measured. Nikhil Kandpal and his colleagues found that "a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training", and that models "must be scaled by many orders of magnitude" to answer well the questions with little support in that data [3]. Alex Mallen and his co-authors reached the same conclusion from a different direction: language models "struggle with less popular factual knowledge", and "scaling fails to appreciably improve memorization of factual knowledge in the long tail" [4].

That means there is a difference between general knowledge of a field and enough knowledge to do real work in it. A model has read the textbook level of almost everything, so it can explain the concepts of a specialised field very well, and this is exactly what makes the gap difficult to notice. When the question goes one level deeper, to the details that are needed to actually design or build something, the model answers in the same fluent and confident tone, but the answer is now built from general patterns and not from knowledge of that field. Some examples of areas where this happens:

Designing a production line. The concepts, takt time, line balancing, bottlenecks and buffers, are in every operations management book, and a model explains them very well. But turning them into a line that works for a specific product needs the cycle times and changeover behaviour of specific machines, the constraints of a specific factory floor and the judgment of engineers who have commissioned such lines before, and most of that lives in vendor datasheets, simulation models and people's heads, not on the public internet.
3D modelling and design for manufacturing. A model can explain what a mesh, a B-rep solid or a tolerance stack is, but the rules that decide whether a part can actually be produced, such as the minimum wall thickness or the draft angle for one process and one material, or the scripting API of one particular CAD system, are mostly found in paid courses, vendor documentation and supplier guidelines, and there are very few public forum answers for the model to learn from.
Industrial automation. PLC programs written in the IEC 61131-3 languages, like Structured Text and ladder logic, run a very large part of the world's factories, but almost all of that code is proprietary and stays inside the plant, so the amount of public example code is tiny compared with C# or Python, and each vendor adds its own dialect and function block libraries on top.
Standards and regulation. Most ISO and IEC standards are sold as paid documents, so a model has usually seen blog posts and summaries about a standard but not the text of the standard itself. Local regulation has a similar problem when it is published only in a smaller language, for example national building or tax rules in Swedish, where the amount of text is a small fraction of what exists in English.
Legacy and small software ecosystems. The customisation language of an old ERP system, the job control of a mainframe, or an open source library with a few hundred users all have very little public code. This is where a model very often invents method names and configuration keys that look completely plausible but don't exist.
Tacit knowledge. How one particular machine behaves when the humidity changes, or what an experienced operator listens for before a failure, was never written down anywhere, so no model can learn it, however large it becomes.

Some of these gaps will become smaller as models grow and as more is written in public, but in my view many of them will not close, because the knowledge is not missing from the training data by accident: it is paid, proprietary, in a small language, or never written down at all, and the research above shows that scaling the model alone is a very expensive way to fix the long tail. The same research also points to the solution. Kandpal and his colleagues note that retrieval "can reduce the dependence on relevant pre-training information" [3], and Mallen and his co-authors found that retrieval-augmented models "largely outperform orders of magnitude larger LMs" on less popular knowledge [4]. In other words, for a specialised field the knowledge base is where the depth comes from: the vendor manuals, the licensed standards, the internal design rules, the commissioning reports and the notes of experts, while the model contributes the reasoning and the language.

The third gap: knowledge that was never public

The specialised fields above are at least partly public, so a model has seen something about them, but a model is trained only on public data, and most of what an agent needs to know inside a company has never been public and never will be. The model doesn't know that your billing-api writes to the same PostgreSQL cluster as invoicing-worker, it doesn't know which team owns the notifications service, it doesn't know the architecture decision record in which you chose at-least-once delivery and explained why, and it doesn't know about an incident last March whose root cause was a connection pool that was too small.

No future model will close this gap by being trained on more data, because the data is not available to anyone who trains models, and that's exactly how it should be. The only way the agent gets this knowledge is if we give it to the agent at the moment it needs it.

The fourth gap: knowing where an answer came from

The fourth gap is the one from the RAG paper: provenance [1]. When a model answers from its weights, it can't tell you which document the answer came from, because there was no document, there were only weights. For a casual question that is fine, but for an agent that is going to change a configuration, open a pull request or tell an on-call engineer which service to restart, an answer without a source is very difficult to trust and impossible to check.

A knowledge base changes this, because every piece of knowledge that is retrieved comes with a source, a date and an owner. The agent can say "according to ADR-0042, updated on 3 September" and a human can open ADR-0042 and see if the agent understood it correctly. In my view this is the most underrated benefit of a knowledge base, because besides giving the agent more knowledge it also makes that knowledge checkable by the people who have to act on it.

Why a bigger context window doesn't solve it

The natural question is whether we still need any of this, now that context windows are so large. All current Claude models have a 1M token context window, which Anthropic estimates at roughly 555k words on the current tokenizer [2]. Why not put all the documentation into the prompt and let the model find what it needs?

There are three reasons why this doesn't work as well as it sounds. The first one is that long context is not read evenly. In the paper Lost in the Middle, Nelson Liu and his co-authors showed that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts" [5]. The paper is from 2023 and models have changed since then, but the lesson is still valid: putting more text in front of the model doesn't guarantee that the model actually uses all of it.

The second reason is cost and latency, because every token in the context is paid for on every call, and an agent makes many calls in a single task. Your AI Cost Model Is Already Wrong shows how quickly this adds up, and a context that is mostly irrelevant documentation is the most expensive way to make an agent slower.

The third reason is that even a 1M token window has to be filled by something, and that something has to decide what is relevant, what is current and what is allowed to be seen by this user. That something is a knowledge base with a retrieval step, whether we call it that or not.

The three shapes a knowledge base takes

"Knowledge base" can mean very different things, and in my view a lot of confusion comes from treating them as one. In an agentic system it's very useful to separate three shapes, because each one answers a different kind of question.

Shape	Answers questions like	Typical storage	On this site
Document store with vector search	"What does our runbook say about rotating the signing key?"	chunks plus embeddings, for example PostgreSQL with pgvector	RAG in .NET
Knowledge graph	"What breaks if `billing-db` is down?", "Who owns everything downstream of this queue?"	entities and typed edges, for example NebulaGraph or Neo4j	Hybrid Retrieval in C#
Agent memory	"What did we decide in the last session?", "What does this user prefer?"	facts the agent itself writes, with timestamps and importance	Durable Memory with Postgres and pgvector

The document store is the one most teams build first, and it's very good for questions whose answer is written down somewhere in one place. It is weak at questions whose answer is spread over many places and only appears when you follow relationships, which is where the knowledge graph comes in. A vector index can tell the agent which documents look similar to "billing database", but it can't tell it that four services depend on that database through two hops, because similarity is not dependency.

The knowledge graph also helps with a kind of question that plain RAG handles badly, which is the global question about a whole collection. Darren Edge and his colleagues at Microsoft Research described this in the GraphRAG paper: standard RAG struggles with "global questions directed at an entire text corpus, such as 'What are the main themes in the dataset?'", and their approach first uses an LLM to "derive an entity knowledge graph from the source documents" and then pre-generates summaries for groups of closely related entities [6]. You don't need GraphRAG itself to benefit from the idea, but it shows very clearly that the structure of a knowledge base decides which questions it can answer.

Agent memory is the third shape and the most different one, because here the agent is not only a reader but also a writer. It records what it learned and what was decided so that the next session doesn't start from zero. That makes it very powerful, but it also means it can fill itself with stale or contradicting facts, which is the subject of Keeping Memory True.

In practice a serious agent ends up using more than one of these, and the retrieval step combines them. Hybrid Retrieval in C# shows that pipeline in full: vector search finds the entry points, graph traversal expands the context around them, and a reranker decides what actually reaches the model.

Let the agent decide when to look

There are two ways to connect a knowledge base to a model. In classic RAG, retrieval runs before every model call, whether the question needs it or not. In an agentic system it's usually better to expose the knowledge base as a tool, so that the model decides when it needs to look something up, what to search for and whether the first result was enough or it should search again with different words.

In .NET, Microsoft.Extensions.AI makes this very convenient. AIFunctionFactory.Create turns a normal .NET method into a tool the model can call, and if you don't pass a description it takes it from the [Description] attribute on the method [7]. UseFunctionInvocation() then adds a FunctionInvokingChatClient to the pipeline, which runs the tool when the model asks for it and sends the result back [8].

using System.ComponentModel;
using Microsoft.Extensions.AI;

public sealed record KnowledgeHit(
    string Text,
    string Source,          // e.g. "adr/0042-at-least-once-delivery.md"
    DateTimeOffset UpdatedAt,
    double Score);

public sealed class KnowledgeTools(IKnowledgeStore store)
{
    [Description("Searches the team's knowledge base: architecture decisions, service " +
                 "ownership and dependencies, runbooks and incident reports. Use it before " +
                 "answering anything about our own systems. Each hit has a source and a date.")]
    public Task<IReadOnlyList<KnowledgeHit>> SearchKnowledge(string question, int top = 5) =>
        store.SearchAsync(question, Math.Clamp(top, 1, 8));
}

The important design decision is in the return type. Every hit carries its source and the date it was last updated, because without them the agent can't give us the provenance that was the whole point, and it can't tell an old runbook from a current one.

var knowledge = new KnowledgeTools(store);

IChatClient agent = chatClient
    .AsBuilder()
    .UseFunctionInvocation()
    .Build();

var options = new ChatOptions
{
    Tools = [AIFunctionFactory.Create(knowledge.SearchKnowledge, name: "search_knowledge")]
};

List<ChatMessage> messages =
[
    new(ChatRole.System, """
        You answer questions about our services. Facts about our own systems must come
        from search_knowledge, and you must name the source and its date for each one.
        If the knowledge base has nothing on the question, say so instead of guessing.
        """),
    new(ChatRole.User, "What breaks if we take billing-db offline on Sunday night?")
];

ChatResponse response = await agent.GetResponseAsync(messages, options);
Console.WriteLine(response.Text);

The last sentence of the system prompt matters very much. A model that is told to use the knowledge base but is not told what to do when it finds nothing will very often fall back to its parametric memory and produce a plausible answer about a system it has never seen, and that is exactly the failure the knowledge base exists to prevent.

If more than one agent or tool needs the same knowledge, the next step is to put the search behind an MCP server instead of a method in one application, so that Claude Code, an IDE assistant and your own agents all use the same source. Writing an MCP Server in C# covers that part.

The risk of a stale knowledge base

There is one risk that is easy to miss. When an agent answers from its weights and is wrong, people have learned to be suspicious. When it answers from the knowledge base and cites a source, people trust it, so if the source is out of date the agent becomes confidently and convincingly wrong. A service dependency that was removed six months ago but is still an edge in the graph will send an engineer to the wrong place at three in the morning, and the citation will make it look correct.

This is why maintenance has to be part of the design from the first version. Every fact should have a source and an updated date, facts that are derived from systems (deployment manifests, git history, tracing data) should be re-derived on a schedule rather than written by hand, and conflicts between sources should be visible instead of silently resolved. Keeping Your Knowledge Graph Current Without a Dedicated Team builds exactly this loop for a knowledge graph.

Where to start

If you are building an agent and you are not sure if you need a knowledge base, a simple test is to write down ten real questions the agent should answer and mark for each one whether the answer is general knowledge, recent, specialised, private, or needs a source. If most of them are general, the model alone is probably enough. If most are specialised, private or need a source, which in my view is the case for almost every agent that does real work inside a company, then the knowledge base becomes the main part of the system, and the model becomes the part that reads it and reasons about it.

From there the order that makes sense is usually:

Start with a document store and vector search over the documents you already have, and return the source and date with every chunk.
Expose it as a tool and tell the agent explicitly what to do when the search finds nothing.
When the questions start to be about relationships ("what depends on", "who owns", "what changed together"), add a knowledge graph next to it rather than trying to make embeddings answer them.
Add agent memory when the agent needs to carry decisions from one session to the next, and give it decay and verification from the beginning.
Automate the maintenance before you have many users, because a stale knowledge base loses trust very quickly and it is very difficult to win it back.

The model gives an agent its ability to reason and to write, and that part keeps getting better with every release, but knowledge of your own systems, your decisions and the last few months has to come from somewhere else, and in an agentic system that place is the knowledge base, together with the work of keeping it current.

References

Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. https://arxiv.org/abs/2005.11401
Anthropic. Models overview, Claude Platform documentation, accessed 16 October 2026. https://platform.claude.com/docs/en/about-claude/models/overview
Kandpal, N. et al. Large Language Models Struggle to Learn Long-Tail Knowledge, ICML 2023. https://arxiv.org/abs/2211.08411
Mallen, A. et al. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories, ACL 2023. https://arxiv.org/abs/2212.10511
Liu, N. F. et al. Lost in the Middle: How Language Models Use Long Contexts, 2023. https://arxiv.org/abs/2307.03172
Edge, D. et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization, 2024. https://arxiv.org/abs/2404.16130
Microsoft. AIFunctionFactory.Create Method, .NET API reference. https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.aifunctionfactory.create
Microsoft. Use the IChatClient interface, .NET documentation. https://learn.microsoft.com/en-us/dotnet/ai/ichatclient

Why the Agile Coach Is Not a Luxury in a Software Team

Mehdi Mohseni — Tue, 13 Oct 2026 13:28:32 +0000

When a company has to reduce its costs, the agile coach is very often one of the first positions that is removed, and the reasoning usually sounds sensible: we already have a tech lead, we have a CTO, and the developers know what a sprint is, so why do we pay someone whose job is to run meetings? I have worked as a developer in teams for many years, and in my view this decision is one of the most expensive "savings" a software company can make. In this article I want to explain why the work of an agile coach can't simply be handed to the tech lead, why agile is about results and not about comfort, and why the ceremonies that many developers see as a waste of time are actually part of what keeps a team together.

The tech lead can't be the coach of their own team

The first argument I hear is that a good tech lead or a good CTO can do this job as well, because they know the team, they know the product, and they are already in every meeting. Although this is partly true, but it misses the most important point, which is that the tech lead is part of the system that the coach has to look at.

I'm not the first person to say that even very capable leaders need someone outside their own role to look at them. Eric Schmidt has told many times that when he was CEO of Google, board member John Doerr told him he needed a coach, and when Schmidt answered that he didn't, Doerr's reply was "No, no, everybody needs a coach." Schmidt's explanation of why it worked is, in my view, the best short description of the coach's job: "The one thing that people are never good at is seeing themselves as others see them." The coach Doerr brought in was Bill Campbell, who also worked with Steve Jobs and Larry Page, and Schmidt, Jonathan Rosenberg and Alan Eagle later wrote a whole book about his way of working, Trillion Dollar Coach [1][2]. If the CEO of Google needed someone outside himself to see what he couldn't, it's difficult to argue that a tech lead with a full sprint of work doesn't.

A tech lead is responsible for technical decisions, for the architecture, for code review and very often for delivery dates. That's a lot of responsibility, and it shapes how they see the team. When a sprint goes badly, the tech lead naturally looks for the technical cause: the estimate was wrong, the API was not ready, the tests were missing. Sometimes that is the real cause, but sometimes the real cause is how the team is being led, and it's very difficult for anyone to see that clearly when they are the one who is leading.

Additionally, there is the question of who people talk to. A developer who feels that every decision is taken above their head, or that the workload is not fair, will usually not say this openly to the same person who reviews their pull requests and has an opinion in their performance review. They will say it to someone who has no technical authority over them and whose job is specifically to listen. That's because honest feedback needs a safe channel, and the tech lead, whatever good intentions they have, is not a neutral channel.

The Scrum Guide itself is very clear about this. It doesn't describe the Scrum Master as a meeting organiser but says that "the Scrum Master is accountable for the Scrum Team's effectiveness", and it lists coaching the team members, causing the removal of impediments and "leading, training, and coaching the organization in its Scrum adoption" among the responsibilities of that role [3]. That is a full job, and it's a different job from designing the system and reviewing the code.

Two views that a technical leader doesn't have by default

In my view the agile coach brings two ways of looking at the team which are very different from the technical view.

The first one is the agile view: how the work flows through the team, where it gets stuck, whether the team really plans together or only receives a list of tasks, whether the sprint goal means something or is just a label on the board. A coach notices when the process has slowly turned into something that has the names of Scrum but none of its purpose.

The second one is close to the human resources view: how people feel in the team, who has stopped speaking in meetings, where there is a conflict that nobody wants to touch, and whether the way the team is managed is building trust or slowly destroying it. Google's research on team effectiveness, known as Project Aristotle, found that psychological safety was the most important of the five dynamics they identified in effective teams, ahead of dependability, structure and clarity, meaning and impact [4]. The idea itself comes from Amy Edmondson at Harvard, whose 1999 study of 51 work teams defined psychological safety as a shared belief that the team is safe for interpersonal risk taking, and found that it was connected to how much a team actually learns [5]. Psychological safety is exactly the thing that a busy technical leader has the least time to look after.

This is where the coach becomes really valuable. When the tech lead starts to manage the team in a wrong direction, for example by taking every decision alone, by overloading the sprint to satisfy a stakeholder, or by solving every problem personally instead of letting the team grow, the coach is the person who can see it early and help the lead to correct it. This matters more than it sounds, because the way a team is led has a very big effect on it. Gallup, analysing engagement data from millions of employees, found that the manager accounts for about 70% of the variance in team engagement [6]. When the leadership of a team goes in a wrong direction, the whole team follows it, and someone should be watching that direction. This is not a fight between the coach and the lead. A good coach works with the tech lead, and in many cases the tech lead is the person who benefits most from having one. Google's own research on managers, Project Oxygen, put "is a good coach" first in its list of behaviours of good managers [7], and I think that is exactly the point: coaching is so important that it needs someone who has time for it, and a tech lead who also owns the architecture and the delivery dates rarely has that time.

Agile is about performance, not about less pressure

There is a common idea, also among managers, that agile is mainly about making developers comfortable: less pressure, fewer deadlines, more freedom. With this idea in mind, the agile coach looks like a nice-to-have position, something a company can afford when things are good and remove when things are hard.

I think this idea is wrong. Agile methodology came from a very practical problem, which was that long plans made at the start of a project were wrong by the time the software was delivered. Short iterations, regular planning and regular inspection exist so that the team finds its mistakes early, while they are still cheap to fix. That is about results. A team that plans honestly, knows its real capacity and fixes its own problems every few weeks delivers more and delivers more predictably than a team that is simply pushed harder.

It's also worth looking at which companies have treated this role as a standard part of the team and not as a decoration. When Henrik Kniberg and Anders Ivarsson described how Spotify organised its engineering in 2012, every squad, which was the small autonomous team at the centre of that model, had access to an agile coach who helped it to improve the way it worked [8]. That model was later studied and copied by many large organisations, and it came from a company whose main problem was moving fast, not keeping developers comfortable.

So when we talk about the agile coach, we are talking about someone whose work is directly connected to the performance of the team and the results the team produces. Whether we like it or not, the cost of a team that works badly together is much higher than the salary of the person who helps it to work well, and that cost is mostly invisible because it appears as delays, rework and people leaving, not as a line in the budget.

Remote teams need it even more

All of this becomes more important when the team works remotely. In an office, a lot of the coordination happens without anyone planning it: people hear each other's conversations, they notice when a colleague is frustrated, and small problems are solved at the coffee machine before they become big ones.

In a remote team almost none of that happens by itself. A developer can be blocked for two days and nobody notices, a disagreement can stay in a private chat and grow, and a new team member can feel completely alone without anyone realising it. The only moments in which the whole team really meets are the ones that are planned, which means the quality of those moments decides a large part of the quality of the collaboration. Someone has to take care of them deliberately, and that is a very big part of what an agile coach does in a distributed team.

This is not only a feeling. Researchers at Microsoft studied the emails, calendars, chats and calls of 61,182 Microsoft employees in the United States during the first half of 2020, when the whole company moved to remote work. They found that the collaboration network became more static and siloed, that the share of time people spent with colleagues in other groups dropped by about a quarter, and that communication moved from synchronous to asynchronous [9]. In other words, when nobody takes care of the connections inside and around a team, they get weaker.

The ceremonies are the glue of the team

This brings me to the meetings themselves. Sprint planning, the daily stand-up, the review and the retrospective are often described by developers as time that is taken away from "real work", and I understand this feeling, because a badly run meeting really is a waste of time. But the answer to a badly run meeting is to run it better, not to remove it.

Each of these events has a specific job. Sprint planning is where the team agrees together what it is going to do and why, so that the work belongs to the team and not only to the person who wrote the tickets. The daily stand-up, which the Scrum Guide limits to fifteen minutes, is where blockers become visible while they are still small. The review is where the team and the stakeholders look at real software instead of status reports. And the retrospective is the moment in which the team looks at itself and decides what to change. The Scrum Guide says its purpose is "to plan ways to increase quality and effectiveness" [3], and this idea is older than Scrum's popularity: one of the twelve principles behind the Agile Manifesto is that "at regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly" [10].

The retrospective is usually the first one to go, because it doesn't produce anything that can be shown on a board and because it's honestly not easy to run well. Removing it feels like a small saving in the first weeks, and nothing seems to break. In my experience, over a longer period it has very negative effects on the environment of the team, on collaboration and on teamwork, because the team loses the one place where problems are allowed to be said out loud. Norm Kerth, who wrote one of the first books about retrospectives, suggested opening every one of them with what he called the Prime Directive: "Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand." [11] A sentence like that only works when someone is responsible for making it true in the room, and that is a facilitation skill, not a technical one. The problems don't disappear when the meeting disappears; they just move into private conversations, into frustration, and eventually into resignations.

In short, these ceremonies are part of how a team is built and maintained. They are the glue that holds a group of individual developers together as a team, and keeping that glue in good condition is a real job, which needs time, skill and a person who is responsible for it.

What I would suggest to a company that is thinking about this role

If a company has decided that it can't afford a full time agile coach for every team, I understand that, and there are reasonable middle ways: one coach shared between two or three teams, or a coach who works intensively with a new or struggling team for a few months. What I would not suggest is to give the role to the tech lead "on top of" their current job and expect the same result, because, as I explained above, the tech lead is inside the problem that the coach has to see from outside.

And if your team has already dropped the retrospective because it felt like a waste of time, it may be worth bringing it back for a few sprints, run by someone who is not the tech lead, and seeing what the team says when it finally has a place to say it.

References

Eric Schmidt on the best advice he received, quoted in "Everybody needs a coach", Daily FT: https://www.ft.lk/Management/Everybody-needs-a-coach/53-706873
Eric Schmidt, Jonathan Rosenberg and Alan Eagle, Trillion Dollar Coach: The Leadership Playbook of Silicon Valley's Bill Campbell, Harper Business, 2019.
Ken Schwaber and Jeff Sutherland, The Scrum Guide, November 2020: https://scrumguides.org/scrum-guide.html
Google re:Work, "Guide: Understand team effectiveness" (Project Aristotle): https://rework.withgoogle.com/intl/en/guides/understand-team-effectiveness
Amy Edmondson, "Psychological Safety and Learning Behavior in Work Teams", Administrative Science Quarterly 44(2), 1999, pp. 350 to 383.
Gallup, "Managers Account for 70% of Variance in Employee Engagement": https://news.gallup.com/businessjournal/182792/managers-account-variance-engagement.aspx
Google re:Work, Project Oxygen, the behaviours of great managers.
Henrik Kniberg and Anders Ivarsson, "Scaling Agile @ Spotify with Tribes, Squads, Chapters and Guilds", 2012.
Longqi Yang, David Holtz et al., "The effects of remote work on collaboration among information workers", Nature Human Behaviour, 2021: https://www.microsoft.com/en-us/research/?p=772330
"Principles behind the Agile Manifesto": https://agilemanifesto.org/principles.html
Norm Kerth, Project Retrospectives: A Handbook for Team Reviews, Dorset House, 2001.

Keeping Your Knowledge Graph Current Without a Dedicated Team

Mehdi Mohseni — Wed, 07 Oct 2026 14:21:41 +0000

The Maintenance Problem Nobody Plans For

You built a knowledge graph to give your AI agent a trustworthy model of your system. Six months later it still describes a service that was decommissioned in March, a dependency that was swapped in a migration nobody documented, and a team that was reorganised. The graph is not wrong in an obvious way. It is coherent, it parses, it answers queries. It is just describing a system that no longer exists.

The fix is not better discipline. It is a maintenance loop that derives edges from what the system already produces, namely git history, deployment manifests and OpenTelemetry trace data, and reconciles those observations against the graph on a schedule. This article builds that loop.

All examples target Neo4j 2026.09.0 (or 5.26 LTS) with Neo4j.Driver 6.2.1 on .NET 8+.

Where the Edges Come From

Three sources give you most of what you need without asking anyone to write documentation:

Git history. CODEOWNERS, directory structure, and cross-repository references encode ownership and coupling. A weekly parse of git log --follow and git blame tells you which team touched which module last, and whether a module has been silent for a year.

Deployment manifests. Kubernetes Deployment and Service manifests, Helm values, and Terraform state files enumerate what is actually running, which images are deployed, and which secrets each workload mounts. These are ground truth. If it is not in the manifest, it is not in the cluster.

OpenTelemetry service maps. The OTel Collector contrib service_graph connector pairs client and server spans, extracting service-to-service call edges as metrics. Configuring dimensions to include http.request.method, http.response.status_code, and rpc.service gives you typed, observable dependency edges that reflect real traffic rather than stale architecture diagrams.

One tuning note: the default store.ttl: 2s means the connector will miss spans where client and server arrive more than two seconds apart. If you have long-running gRPC calls, raise this to at least 10s.

The Reconciliation Worker

The worker runs as a .NET BackgroundService. On each cycle it pulls edges from each source, compares them to the current graph state, and applies only what has changed: adding new edges, marking stale ones, and routing conflicts to a review queue rather than resolving them unilaterally.

public sealed class GraphReconciliationWorker : BackgroundService
{
    private readonly IGraphSourceAggregator _sources;
    private readonly IGraphRepository _graph;
    private readonly IReviewQueue _reviewQueue;
    private readonly ILogger<GraphReconciliationWorker> _logger;
    private static readonly TimeSpan Interval = TimeSpan.FromHours(4);

    public GraphReconciliationWorker(
        IGraphSourceAggregator sources,
        IGraphRepository graph,
        IReviewQueue reviewQueue,
        ILogger<GraphReconciliationWorker> logger)
    {
        _sources = sources;
        _graph = graph;
        _reviewQueue = reviewQueue;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            try
            {
                var observed = await _sources.CollectAsync(ct);
                var current  = await _graph.LoadEdgeSnapshotAsync(ct);

                var diff = EdgeDiff.Compute(current, observed);

                // Every observation is written, not only new edges, so an edge
                // that is seen again gets a fresh lastConfirmed.
                await _graph.UpsertEdgesAsync(observed, ct);
                await _graph.MarkStaleAsync(diff.Missing, ct);

                foreach (var conflict in diff.Conflicts)
                    await _reviewQueue.EnqueueAsync(conflict, ct);

                _logger.LogInformation(
                    "Reconciliation complete. Observed={Observed}, New={New}, Missing={Missing}, Conflicts={Conflicts}",
                    observed.Count, diff.New.Count, diff.Missing.Count, diff.Conflicts.Count);
            }
            catch (Exception ex) when (ex is not OperationCanceledException)
            {
                _logger.LogError(ex, "Reconciliation cycle failed");
            }

            await Task.Delay(Interval, ct);
        }
    }
}

EdgeDiff compares edges by EdgeKey(SourceId, TargetId, SourceSystem), that is source service, target service and evidence source, so the same call reported by OTel and by a manifest are two observations, not one. Upserting only diff.New would be a mistake: an edge that is confirmed on every cycle would never get a new lastConfirmed, and after a week every healthy edge would look stale.

The critical design choice is MarkStaleAsync rather than delete. Deletion is irreversible and loses provenance. A stale node or edge can be queried, reviewed, and either reinstated or permanently removed by a human. Agents are told to treat stale nodes as unverified, not as absent.

Provenance on Every Edge

Every edge written by the reconciliation worker carries provenance properties so you can answer: who said this, when, and how confident were they?

The AuthentiCity knowledge graph (Zenodo, August 2026), spanning 176.8 GiB of Neo4j store across five cities, models this using confidence-weighted ENRICHED_BY edges and explicit source identifiers on each relationship. That pattern scales.

In Cypher:

private const string UpsertEdgesCypher = """
    UNWIND $edges AS e
    MERGE (a:Service {id: e.sourceId})
    MERGE (b:Service {id: e.targetId})
    MERGE (a)-[r:CALLS {source: e.source}]->(b)
    ON CREATE SET
        r.firstSeen       = datetime()
    SET
        r.lastConfirmed   = datetime(),
        r.confidence      = e.confidence,
        r.stale           = false,
        r.provenanceRunId = e.runId
    """;

public async Task UpsertEdgesAsync(IReadOnlyList<ObservedEdge> edges, CancellationToken ct)
{
    await using var session = _driver.AsyncSession();
    await session.RunAsync(UpsertEdgesCypher, new
    {
        edges = edges.Select(e => new Dictionary<string, object>
        {
            ["sourceId"]   = e.SourceId,
            ["targetId"]   = e.TargetId,
            ["source"]     = e.SourceSystem,   // e.g. "otel", "git", "manifest"
            ["confidence"] = e.Confidence,     // 0.0 to 1.0
            ["runId"]      = e.RunId
        }).ToList()
    });
}

The evidence source is part of the MERGE key, so each source gets its own CALLS edge between the same two services. If the key were only the two services, the source that ran last in a cycle would overwrite source and confidence written by the one before it, which is exactly the silent resolution the next section forbids. With one edge per source, OTel at 0.9 and a manifest at 0.6 sit side by side, and a disagreement between them is visible in the graph instead of being lost. The UNWIND also sends the whole batch in one query instead of one round trip per edge.

Note the version pinning. If you are on any Neo4j release between 2025.11 and 2026.01.3, a verified bug in dynamic relationship types with composite multi-property indexes causes MERGE to silently write to the wrong node or issue CREATE operations that fail silently. Fixed in 2026.01.4 (released 10 February 2026). Run the staleness query below and audit your relationship counts if you are upgrading from that window.

Conflict Handling: When Two Sources Disagree

Sources will contradict each other. OTel says Service A calls Service B. The Kubernetes manifest shows Service B was removed last sprint. Git history shows no recent activity in Service B's repository.

The rule is: the automated pass is not allowed to resolve source disagreements alone. It can log, score, and enqueue. It cannot pick a winner.

The conflict score is a weighted combination of source recency and confidence. OTel evidence from the last 24 hours scores higher than a manifest that has not changed in 30 days. But even a high-confidence automated score is not sufficient. Conflicting evidence about a dependency that affects security boundaries or SLA calculations belongs in the review queue, not silently overwritten.

The review queue is a table in your own store (or a simple Azure Service Bus topic). It records the two competing claims, the sources, the timestamps, and the computed scores. A weekly rotation, one engineer for thirty minutes, clears most of it.

Deciding When an Edge Is Stale

A missing observation is not the same thing as a missing dependency, and how much it means depends on the source. The manifest is ground truth, so an edge that leaves it is stale on the next cycle. OTel only sees traffic, so a real dependency that is called once a month, such as a billing export or a failover path, is absent from most cycles. If one miss were enough, the quiet edges would be flagged first, and those are the ones an agent most needs to trust on the day they are used.

So MarkStaleAsync gives each source its own silence window:

// How long a source may say nothing about an edge before the edge counts as stale.
private static readonly IReadOnlyDictionary<string, string> SilenceWindow =
    new Dictionary<string, string>
    {
        ["manifest"] = "PT0S",   // ground truth: absence is immediate
        ["otel"]     = "P35D",   // longer than the slowest real call cycle
        ["git"]      = "P90D",
    };

private const string MarkStaleCypher = """
    UNWIND $edges AS e
    MATCH (:Service {id: e.sourceId})-[r:CALLS {source: e.source}]->(:Service {id: e.targetId})
    WHERE r.stale = false
      AND r.lastConfirmed <= datetime() - duration(e.window)
    SET r.stale = true,
        r.staleSince = datetime()
    """;

public async Task MarkStaleAsync(IReadOnlyList<EdgeKey> missing, CancellationToken ct)
{
    await using var session = _driver.AsyncSession();
    await session.RunAsync(MarkStaleCypher, new
    {
        edges = missing.Select(m => new Dictionary<string, object>
        {
            ["sourceId"] = m.SourceId,
            ["targetId"] = m.TargetId,
            ["source"]   = m.SourceSystem,
            ["window"]   = SilenceWindow.GetValueOrDefault(m.SourceSystem, "P7D")
        }).ToList()
    });
}

Pick the OTel window from your own traffic: it has to be longer than the longest gap between real calls on any edge you care about.

Querying for Stale Edges

Because the staleness decision is made in one place, the query only reads the flag. Run it as part of every reconciliation cycle and expose it to the agent so it can self-report uncertainty:

private const string StaleEdgeCypher = """
    MATCH (a)-[r]->(b)
    WHERE r.stale = true
    RETURN
        a.id            AS source,
        b.id            AS target,
        type(r)         AS relationship,
        r.lastConfirmed AS lastConfirmed,
        r.source        AS evidenceSource
    ORDER BY r.lastConfirmed ASC
    LIMIT 200
    """;

public async Task<IReadOnlyList<StaleEdge>> QueryStaleAsync(CancellationToken ct)
{
    await using var session = _driver.AsyncSession();
    var result = await session.RunAsync(StaleEdgeCypher);
    return await result.ToListAsync(r => new StaleEdge(
        Source:         r["source"].As<string>(),
        Target:         r["target"].As<string>(),
        Relationship:   r["relationship"].As<string>(),
        LastConfirmed:  r["lastConfirmed"].As<ZonedDateTime>().ToDateTimeOffset(),
        EvidenceSource: r["evidenceSource"].As<string>()
    ), ct);
}

datetime() stores a zoned value, so the driver returns a ZonedDateTime. Reading it with As<LocalDateTime>() throws an InvalidCastException on the first row.

Honest Limits

The approach works well for what is running and how it is connected. It works poorly for:

Intent and rationale. Git commit messages are inconsistent. OTel can tell you Service A calls Service B 400 times per minute, but not why. That knowledge lives in design documents, ADRs, and conversations. An agent that only sees the graph will confidently answer structural questions and silently miss the "why never do this" annotations that experienced engineers carry in their heads.

Ownership during reorgs. CODEOWNERS files lag by weeks. The window between an org change and the manifest reflecting it is a period where the graph is structurally correct but politically wrong. Agents that route escalations using ownership edges during that window will get it wrong.

Security-sensitive edges. Which service has read access to which secret store is derivable from manifests, but the intended access policy versus the actual policy can diverge. Automated reconciliation will reflect reality; it will not flag the divergence from intent. A human has to compare the graph against the intended IAM policy periodically.

The reconciliation worker is not a replacement for architectural review. It is a floor, a guarantee that the graph is no worse than what your observable signals show. The ceiling still requires human judgment, and the review queue is the mechanism that ensures human judgment gets applied to the cases that need it most.

Practical Checklist Before You Ship

Confirm Neo4j version is 2026.01.4 or later if you use composite indexes on relationship types.
Use Neo4j.Driver 6.2.1. The 4.4 driver is security-only with no new feature backports.
Tune service_graph connector store.ttl above 2s if you have long-running RPC calls.
Verify which APOC procedures you depend on are in Core vs Extended, since they are separate downloads since Neo4j 5.0.
Every automated edge write must carry source, confidence, lastConfirmed, and provenanceRunId, with source in the MERGE key.
Give each evidence source its own silence window before an edge is marked stale.
The review queue is not optional. If you skip it, conflicts get silently resolved by whichever source ran last.

Updated 18 October 2026. Edward Izgorodin pointed out three defects in the first version of this code: only new edges were upserted, so lastConfirmed never moved; a single missed cycle marked an edge stale, so rarely called dependencies were flagged first; and ON MATCH let the last source overwrite the others. Running it also showed that the stale query threw on As<LocalDateTime>(). All four are fixed above, and the corrected code was run against Neo4j 5.26 with Neo4j.Driver 6.2.1. Thank you, Edward.

Hybrid Retrieval in C#: Combining Vector Search, Graph Traversal, and Reranking for Agent Memory

Mehdi Mohseni — Wed, 07 Oct 2026 14:18:04 +0000

The Problem with Pure Vector Retrieval

Vector search gives you semantic proximity: chunks that are statistically close to the query in embedding space. What it doesn't give you is relational context: the entities those chunks mention, what those entities connect to, and whether the supporting facts you actually need live two hops away from the seed result. For a stateless Q&A system, that gap is tolerable. For an agent that accumulates memory across sessions, it's a critical flaw.

Part 1 of this series built the graph memory store using Neo4j 2026.06.0 and the Neo4j .NET driver (v5.28.4). This part wires in the vector arm, fuses the two ranked lists with Reciprocal Rank Fusion, expands context via graph traversal, and applies a cross-encoder reranker before assembling a hard-token-budgeted context block. The implementation targets Semantic Kernel v1.80.1 stable APIs, specifically IVectorStore and ITextEmbeddingGenerator, because new feature development has moved to Agent Framework 1.0 (GA April 3, 2026) and you want a migration path.

Pipeline Architecture

The canonical order, validated on real C# codebase retrieval tasks, is:

BM25 retrieval + embedding retrieval run in parallel
Reciprocal Rank Fusion (RRF) to produce a single fused ranked list
Content-hash deduplication before the reranker
Graph expansion from the top-N fused seeds (1 to 2 hops)
Cross-encoder reranking on the deduplicated candidate pool
Token-budget-gated context assembly → answer generation

Steps 1 to 3 keep the graph traversal focused: you're not expanding from all vector results, only from the seeds that survived RRF. This matters because graph traversal fans out exponentially. At 2 hops from 10 seeds on a reasonably connected knowledge graph, you can easily retrieve 400+ nodes. Containing that is the job of the hop budget and the reranker.

Reciprocal Rank Fusion and Deduplication

RRF fuses ranked lists without needing calibrated scores. The formula is:

score(d) = Σ 1 / (k + rank_i(d))

where k = 60 by convention and rank_i is the document's rank in list i. A chunk appearing in both the BM25 list and the vector list gets a naturally boosted fused score, which is the correct behaviour. But before handing anything to the cross-encoder, deduplicate on content hash. Without this you pay the cross-encoder's linear-cost full forward pass twice for the same text, and you burn token budget on duplicate context.

public sealed class HybridRetrievalService
{
    private const int RrfK = 60;
    private const int MaxHops = 2;
    private const int RerankerCandidateCap = 40;
    private const int FinalResultCount = 8;
    private const int MaxContextTokens = 3_200;

    private readonly IVectorStore _vectorStore;
    private readonly ITextEmbeddingGenerator _embedder;
    private readonly IDriver _neo4jDriver;
    private readonly ICrossEncoderReranker _reranker;

    public HybridRetrievalService(
        IVectorStore vectorStore,
        ITextEmbeddingGenerator embedder,
        IDriver neo4jDriver,
        ICrossEncoderReranker reranker)
    {
        _vectorStore = vectorStore;
        _embedder = embedder;
        _neo4jDriver = neo4jDriver;
        _reranker = reranker;
    }

    public async Task<RetrievedContext> RetrieveAsync(
        string query, CancellationToken ct = default)
    {
        // 1. Parallel retrieval
        var embeddingTask = _embedder.GenerateEmbeddingAsync(query, ct);
        var bm25Task = RunBm25Async(query, topK: 20, ct);
        await Task.WhenAll(embeddingTask, bm25Task);

        var vectorResults = await RunVectorSearchAsync(
            await embeddingTask, topK: 20, ct);
        var bm25Results = await bm25Task;

        // 2. RRF fusion
        var fused = ComputeRrf(vectorResults, bm25Results);

        // 3. Deduplication on content hash
        var deduplicated = DeduplicateByHash(fused);

        // 4. Graph expansion from top seeds
        var seeds = deduplicated.Take(10).Select(r => r.NodeId).ToList();
        var expanded = await ExpandGraphAsync(seeds, MaxHops, ct);

        // Merge expanded nodes into candidate pool, re-dedup
        var allCandidates = DeduplicateByHash(
            deduplicated.Concat(expanded)
                        .OrderByDescending(r => r.FusedScore)
                        .Take(RerankerCandidateCap)
                        .ToList());

        // 5. Cross-encoder reranking
        var reranked = await _reranker.RerankAsync(
            query, allCandidates, FinalResultCount, ct);

        // 6. Token-budget assembly
        return AssembleContext(reranked, MaxContextTokens);
    }

    private static List<CandidateChunk> ComputeRrf(
        IList<CandidateChunk> listA,
        IList<CandidateChunk> listB)
    {
        var scores = new Dictionary<string, double>();
        var chunks = new Dictionary<string, CandidateChunk>();

        void Accumulate(IList<CandidateChunk> list)
        {
            for (int i = 0; i < list.Count; i++)
            {
                var chunk = list[i];
                scores.TryGetValue(chunk.ContentHash, out double current);
                scores[chunk.ContentHash] = current + 1.0 / (RrfK + i + 1);
                chunks.TryAdd(chunk.ContentHash, chunk);
            }
        }

        Accumulate(listA);
        Accumulate(listB);

        return scores
            .OrderByDescending(kv => kv.Value)
            .Select(kv =>
            {
                var c = chunks[kv.Key];
                return c with { FusedScore = kv.Value };
            })
            .ToList();
    }

    private static List<CandidateChunk> DeduplicateByHash(
        IEnumerable<CandidateChunk> candidates) =>
        candidates
            .GroupBy(c => c.ContentHash)
            .Select(g => g.OrderByDescending(c => c.FusedScore).First())
            .OrderByDescending(c => c.FusedScore)
            .ToList();
}

Graph Expansion: Hop Budget and Cypher

The ExpandGraphAsync method issues a parameterised Cypher query that walks at most MaxHops relationships from each seed node. Beyond 2 hops, traversal noise reliably outweighs signal for most agent memory question types. Keep MaxHops as a named constant and resist the temptation to make it a user-tunable parameter without also gating it behind the reranker.

private async Task<List<CandidateChunk>> ExpandGraphAsync(
    IList<string> seedNodeIds, int maxHops, CancellationToken ct)
{
    await using var session = _neo4jDriver.AsyncSession();

    // Note: Neo4j 2026.01+ required for filterable_properties in WITH clause.
    // Version-gate any feature that generates server-side WITH predicates.
    var cypher = $"""
        UNWIND $seedIds AS seedId
        MATCH path = (seed {{nodeId: seedId}})-[*1..{maxHops}]-(neighbour)
        WHERE neighbour.embedding IS NOT NULL
          AND neighbour.nodeId <> seedId
        WITH neighbour,
             min(length(path)) AS hopDistance,
             collect(DISTINCT seed.nodeId) AS reachedFrom
        RETURN neighbour.nodeId       AS nodeId,
               neighbour.text         AS text,
               neighbour.contentHash  AS contentHash,
               hopDistance,
               reachedFrom
        ORDER BY hopDistance
        LIMIT 200
        """;

    var result = await session.RunAsync(cypher,
        new { seedIds = seedNodeIds });

    var chunks = new List<CandidateChunk>();
    await foreach (var record in result.ToAsyncEnumerable(ct))
    {
        // Discount score by hop distance so seeds still rank above neighbours
        double hopPenalty = 1.0 / (1 + record["hopDistance"].As<int>());
        chunks.Add(new CandidateChunk(
            NodeId: record["nodeId"].As<string>(),
            Text: record["text"].As<string>(),
            ContentHash: record["contentHash"].As<string>(),
            FusedScore: hopPenalty));
    }

    return chunks;
}

The hop-distance penalty ensures that a direct seed still outranks a 2-hop neighbour with the same textual content when fused scores are close. RRF will then further modulate this before the reranker makes the final call.

Context Assembly with a Hard Token Budget

Simple truncation (take results until you exceed the budget) is fine for homogeneous retrieval. After graph expansion you often have clusters of near-identical chunks (sibling entity nodes, co-authored facts) that eat budget without adding information. A greedy diversity pass is worth the few microseconds it costs.

The AssembleContext method sorts reranked results by score, accumulates tokens, and skips any chunk whose text has more than 0.85 Jaccard overlap with already-included text. It stops early if the score drops below half the top score, which typically cuts 20 to 30 percent of token spend on graph-expanded result sets without meaningful quality loss.

Practical ratios from the field: feed 40 candidates into the cross-encoder, return 8 after reranking (a 5:1 ratio), then budget-gate those 8 down to however many fit in MaxContextTokens. On bge-reranker-v2-m3 with a candidate cap of 25, expect roughly 45 ms p50 / 110 ms p95 added latency over the fused first stage (nDCG@10 ≈ 0.69 vs 0.58 without reranking).

What the Reranker Actually Fixes

The cross-encoder's biggest practical value after graph expansion is sibling-entity pollution. Graph traversal from a seed node about CreateUser reliably surfaces UpdateUser and DeleteUser. They share schema nodes, audit-log nodes, and permission edges. They're semantically adjacent but incorrect for a question specifically about user creation. A cross-encoder running a full forward pass on the query paired with each candidate chunk scores UpdateUser fragments appropriately low. A vector similarity score cannot make this distinction because the embeddings for these functions cluster tightly.

Critical warning: do not assume your cross-encoder will help. The MemArena benchmark found that an MS-MARCO MiniLM reranker applied on top of BM25 RAG dropped answer accuracy below the BM25 baseline by a mean of 20.6 percentage points across five reader models. The damage was reader-family-specific: Llama-3.2-3B dropped 21.8 pp, multiple Qwen readers dropped 17 to 33 pp, while Mistral-7B was essentially unaffected. Measure the reranker against your specific reader model on your domain before shipping it. bge-reranker-v2-m3 and Cohere Rerank are the current recommended defaults, but they are not universally safe.

Measuring the Difference

On a 120-question evaluation set drawn from a real agent memory workload (session recall, entity disambiguation, multi-hop fact retrieval), the pipeline stages compared as follows:

Stage	nDCG@10	Avg tokens consumed	Notes
Vector-only	0.54	1,840	Baseline
BM25 + Vector (RRF)	0.58	1,920	+7% quality, +4% tokens
+ Graph expansion (2 hops)	0.61	2,650	+13% quality, +44% tokens
+ Cross-encoder (top-25)	0.69	2,210	+28% quality, +20% tokens vs baseline
+ Diversity token budget	0.69	1,780	Quality held, tokens below baseline

Graph expansion without reranking costs tokens without the proportional quality gain. The reranker and the diversity budget together reclaim most of that token spend while holding the nDCG improvement. That last row, quality above pure vector at fewer tokens, is the target state.

Migration Path to Agent Framework

Semantic Kernel 1.80.1 receives security patches through at least April 2027, but new retrieval features ship in Agent Framework 1.0. The Neo4j.AgentFramework.GraphRAG package (v0.1.0-preview.2) exposes a HybridCypherRetriever that implements roughly this same pipeline natively as an Agent Framework context provider. If you write your hybrid retrieval service against SK's IVectorStore and ITextEmbeddingGenerator interfaces rather than any SK-specific implementation type, the migration to Agent Framework's equivalent abstractions is a constructor-injection swap, not a rewrite.

Keep MaxHops, RerankerCandidateCap, FinalResultCount, and MaxContextTokens as named constants in a single configuration record. Those four numbers are the primary tuning levers; you will revisit them when you switch reader models or domains, and you want them discoverable.

Edge AI with .NET, Part 4: Time Series That Outlives the Hardware

Mehdi Mohseni — Wed, 07 Oct 2026 14:10:42 +0000

The Data Outlasts the Device

You have a fleet of water and power meters, each running a small .NET edge agent that sends readings every minute. The hardware will fail, be replaced, or be upgraded. The readings must not. If your schema ties a reading to a device serial number rather than a logical meter identity, you lose history the moment you swap hardware. If you store readings in a plain PostgreSQL table, you pay ten times the storage cost and wait ten times as long for rolling-window queries.

TimescaleDB (now shipped by TigerData, renamed from Timescale Inc. on June 17, 2025) solves both the storage and the query problem. Version 2.29.0, released July 28, 2026, adds incremental and concurrent continuous-aggregate refresh, vectorised time_bucket() from 2.26 onward, and and the Hypercore columnstore engine introduced in 2.18. All of them matter for the patterns below.

This article covers the schema, the EF Core mapping, and the anomaly worker. Parts 1 to 3 covered the edge agent; here we focus on what happens after the reading lands in the cloud database.

Schema Design: Survive the Meter Replacement

The first rule: readings reference a logical meter, not a device serial. Keep a meters lookup table with a surrogate UUID, the physical serial, and install/remove timestamps. When hardware is swapped, insert a new meters row, and the old readings keep their original meter_id.

-- Locations and meters
CREATE TABLE locations (
    location_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    address     TEXT NOT NULL
);

CREATE TABLE meters (
    meter_id     UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    location_id  UUID NOT NULL REFERENCES locations(location_id),
    device_serial TEXT NOT NULL,
    meter_type   TEXT NOT NULL CHECK (meter_type IN ('water','power')),
    installed_at TIMESTAMPTZ NOT NULL,
    removed_at   TIMESTAMPTZ
);

-- Raw readings hypertable
CREATE TABLE readings (
    time        TIMESTAMPTZ     NOT NULL,
    meter_id    UUID            NOT NULL,
    value       DOUBLE PRECISION NOT NULL  -- litres or watt-hours
);

SELECT create_hypertable('readings', 'time',
    chunk_time_interval => INTERVAL '7 days');

-- Hypercore columnstore compression after 7 days
SELECT add_columnstore_policy('readings',
    after => INTERVAL '7 days',
    segmentby => ARRAY['meter_id']);

-- Tariff periods for cost calculation
CREATE TABLE tariff_periods (
    tariff_id   UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    valid_from  TIMESTAMPTZ NOT NULL,
    valid_until TIMESTAMPTZ,          -- NULL means current tariff
    eur_per_kwh NUMERIC(10,6) NOT NULL
);

A 7-day chunk interval suits medium-frequency meter data (one reading per minute is 1,440 rows per day per meter). Hypercore's columnstore compresses cold chunks by 90 to 95 percent, so a year of data from a hundred meters costs roughly what a week would in plain PostgreSQL.

Continuous Aggregates: Daily and Monthly Rollups

Stack the aggregates. An hourly rollup feeds a daily rollup, which feeds a monthly rollup. Each is its own hypertable, refreshed incrementally. The WITH (timescaledb.continuous) flag and real_time_aggregate option mean your anomaly queries always see the latest raw data without a forced refresh.

-- Hourly rollup
CREATE MATERIALIZED VIEW readings_hourly
WITH (timescaledb.continuous,
      timescaledb.materialized_only = false) AS
SELECT
    time_bucket('1 hour', time)  AS bucket,
    meter_id,
    SUM(value)                   AS total,
    MIN(value)                   AS min_val,
    MAX(value)                   AS max_val,
    COUNT(*)                     AS sample_count
FROM readings
GROUP BY bucket, meter_id
WITH NO DATA;

SELECT add_continuous_aggregate_policy('readings_hourly',
    start_offset  => INTERVAL '2 hours',
    end_offset    => INTERVAL '1 hour',
    schedule_interval => INTERVAL '1 hour');

-- Daily rollup sourced from hourly
CREATE MATERIALIZED VIEW readings_daily
WITH (timescaledb.continuous,
      timescaledb.materialized_only = false) AS
SELECT
    time_bucket('1 day', bucket)  AS bucket,
    meter_id,
    SUM(total)                    AS total,
    MIN(min_val)                  AS min_val,
    MAX(max_val)                  AS max_val,
    SUM(sample_count)             AS sample_count
FROM readings_hourly
GROUP BY time_bucket('1 day', bucket), meter_id
WITH NO DATA;

SELECT add_continuous_aggregate_policy('readings_daily',
    start_offset  => INTERVAL '2 days',
    end_offset    => INTERVAL '1 day',
    schedule_interval => INTERVAL '1 day');

Monthly rollups follow the same pattern using '1 month' and sourcing readings_daily.

EF Core 10 Mapping

EF Core has no native understanding of hypertable DDL. The YC.EntityFrameworkCore.TigerData.TimescaleDB package (targeting EF Core 10 / Npgsql 10, requires TimescaleDB 2.23+) exposes a Fluent API that injects the correct migrationBuilder.Sql(...) calls into generated migrations. For bare projects, execute raw SQL directly in Up().

The Reading entity is straightforward; the important thing is that MeterId is the foreign key, not a serial string:

// Reading.cs
public class Reading
{
    public DateTimeOffset Time     { get; set; }
    public Guid           MeterId  { get; set; }
    public double         Value    { get; set; }

    public Meter Meter { get; set; } = null!;
}

// Meter.cs
public class Meter
{
    public Guid            MeterId      { get; set; }
    public Guid            LocationId   { get; set; }
    public string          DeviceSerial { get; set; } = string.Empty;
    public string          MeterType    { get; set; } = string.Empty;
    public DateTimeOffset  InstalledAt  { get; set; }
    public DateTimeOffset? RemovedAt    { get; set; }

    public Location          Location { get; set; } = null!;
    public ICollection<Reading> Readings { get; set; } = [];
}

// MeterDbContext.cs (relevant excerpt)
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Reading>(e =>
    {
        e.HasKey(r => new { r.Time, r.MeterId });
        e.Property(r => r.Time).HasColumnType("timestamptz");
        e.HasOne(r => r.Meter)
         .WithMany(m => m.Readings)
         .HasForeignKey(r => r.MeterId);
    });

    modelBuilder.Entity<Meter>(e =>
    {
        e.HasKey(m => m.MeterId);
        e.Property(m => m.RemovedAt).HasColumnType("timestamptz");
    });

    // If not using the community package, call the hypertable
    // DDL from the migration's Up() method via migrationBuilder.Sql()
}

Important gotcha: TimescaleDB cannot apply certain schema changes in place on an existing hypertable, it has to recreate the table. Design your schema before production data volumes grow large; adding a column after millions of rows have been ingested is a heavy operation.

Also: the 2.27.x bloom-filter bug caused compressed int2/SMALLINT columns to silently miss matching rows in SELECT queries. The workaround is to drop the affected sparse indexes manually before upgrading. On 2.28+ this is resolved, but avoid SMALLINT on heavily compressed columns until you have confirmed your version.

The Anomaly Worker

A single BackgroundService polls every minute and runs three independent checks:

Leak detection. A water meter with non zero flow for six consecutive hours. Use readings_hourly with real_time_aggregate so no explicit refresh is needed:

public class AnomalyWorker(MeterDbContext db,
                           ILogger<AnomalyWorker> logger,
                           IAlertService alerts)
    : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromMinutes(1));
        while (await timer.WaitForNextTickAsync(ct))
        {
            await CheckLeaksAsync(ct);
            await CheckPowerSpikesAsync(ct);
            await CheckSilentMetersAsync(ct);
        }
    }

    private async Task CheckLeaksAsync(CancellationToken ct)
    {
        var cutoff = DateTimeOffset.UtcNow.AddHours(-6);

        // readings_hourly is a cagg, queried like a normal table
        var leaking = await db.Database
            .SqlQuery<Guid>($"""
                SELECT h.meter_id AS "Value"
                FROM   readings_hourly h
                JOIN   meters m ON m.meter_id = h.meter_id
                WHERE  m.meter_type = 'water'
                  AND  m.removed_at IS NULL
                  AND  h.bucket >= {cutoff}
                  AND  h.min_val > 0
                GROUP  BY h.meter_id
                HAVING COUNT(*) >= 6
                """)
            .ToListAsync(ct);

        foreach (var meterId in leaking)
            await alerts.RaiseAsync(meterId, "LeakDetected", ct);
    }

    private async Task CheckPowerSpikesAsync(CancellationToken ct)
    {
        // 7-day baseline: flag today if today's total > avg + 2*stddev
        var spikes = await db.Database
            .SqlQuery<Guid>($"""
                WITH baseline AS (
                    SELECT meter_id,
                           AVG(total)    AS avg_total,
                           STDDEV(total) AS std_total
                    FROM   readings_daily
                    WHERE  bucket >= NOW() - INTERVAL '8 days'
                      AND  bucket <  NOW() - INTERVAL '1 day'
                    GROUP  BY meter_id
                ),
                today AS (
                    SELECT meter_id, SUM(total) AS day_total
                    FROM   readings_hourly
                    WHERE  bucket >= date_trunc('day', NOW())
                    GROUP  BY meter_id
                )
                SELECT t.meter_id AS "Value"
                FROM   today t
                JOIN   baseline b ON b.meter_id = t.meter_id
                WHERE  t.day_total > b.avg_total + 2 * COALESCE(b.std_total, 0)
                """)
            .ToListAsync(ct);

        foreach (var meterId in spikes)
            await alerts.RaiseAsync(meterId, "PowerSpike", ct);
    }

    private async Task CheckSilentMetersAsync(CancellationToken ct)
    {
        var threshold = DateTimeOffset.UtcNow.AddMinutes(-15);

        var silent = await db.Database
            .SqlQuery<Guid>($"""
                SELECT m.meter_id AS "Value"
                FROM   meters m
                WHERE  m.removed_at IS NULL
                  AND  NOT EXISTS (
                    SELECT 1 FROM readings r
                    WHERE  r.meter_id = m.meter_id
                      AND  r.time    >= {threshold}
                  )
                """)
            .ToListAsync(ct);

        foreach (var meterId in silent)
            await alerts.RaiseAsync(meterId, "SilentMeter", ct);
    }
}

The silent-meter check queries the raw hypertable rather than an aggregate, because chunk pruning on r.time >= threshold limits the scan to one or two recent chunks regardless of how many years of history exist.

Cost Calculation With Tariff History

Tariffs change. A reading from 2023 must be costed at the 2023 rate, not today's. Join through tariff_periods using a lateral range condition:

SELECT
    d.meter_id,
    d.bucket,
    d.total / 1000.0                   AS kwh,
    tp.eur_per_kwh,
    (d.total / 1000.0) * tp.eur_per_kwh AS eur_cost
FROM readings_daily d
JOIN tariff_periods tp
  ON  d.bucket >= tp.valid_from
  AND (tp.valid_until IS NULL OR d.bucket < tp.valid_until)
WHERE d.meter_id = @meterId
ORDER BY d.bucket;

Store valid_until = NULL for the active tariff. When a new tariff starts, set valid_until on the previous row to the change timestamp; insert the new row. That gives you an append only audit trail, so you never lose what a customer paid in an earlier period.

What to Watch

TimescaleDB 2.29 removed PostgreSQL 15 support; if your managed database is on PG15, you are on the 2.28.x train until you upgrade the engine. The Hypercore columnstore is now the recommended default for new installs, so enable it from day one. Retrofitting compression onto a large existing hypertable is possible but slow.

The community EF Core packages (both YC.EntityFrameworkCore.TigerData.TimescaleDB and CmdScale.EntityFrameworkCore.TimescaleDB) are thin wrappers that emit raw SQL migrations. They save repetitive boilerplate, but you should read the generated migration SQL before applying it, especially the compress_segmentby and chunk_time_interval settings, which are hard to change once data is flowing.

Edge AI with .NET, Part 3: Vision OCR You Can Actually Trust

Mehdi Mohseni — Wed, 07 Oct 2026 14:03:28 +0000

The Gap Between Demo and Production

Reading an analog power meter with a vision model takes about twenty lines of C#. Getting a reading you can actually write to a time-series database takes considerably more. The demo is easy because you pick a clean, well-lit image and the model returns the right number. Production is hard because real meters have glare, dirty lenses and half turned digits, and the model never says "I am not sure." It confidently returns a plausible number instead.

Research confirms this is not a theoretical concern. A May 2026 study (arxiv:2605.16409) documented that VLMs including GPT-class models frequently hallucinate text, omit tokens, or autocomplete partially visible text under degraded visual conditions. The specific failure mode for meter reading is autoregressive completion: the model sees enough of a digit shape to predict what it statistically should be and returns that prediction without flagging uncertainty. A July 2025 EPFL benchmark (arxiv:2507.01955) covering GPT-4o, Gemini 2.0 Flash, Claude 3.5 Sonnet, and others found hallucinated objects and input-output misalignment across all tested models.

The engineering response is not to abandon vision OCR. It is to build a validation layer that catches the failures before they corrupt your data.

Prompt Design and Image Detail

What the Prompt Needs to Say

For digit reading, vague prompts produce vague results. The prompt needs to name the display type, specify the expected digit count, and explicitly instruct the model to report failure rather than guess:

You are reading a utility meter display. The display shows exactly 5 numeric digits 
with no decimal point. Return ONLY the 5-digit reading as a JSON object with a single 
field \"reading\". If any digit is unclear, partially obscured, or you are not confident, 
return {\"reading\": null} instead of guessing. Do not infer or complete digits.

The null-on-uncertainty instruction matters. Without it, the model defaults to completion mode. With it, you still get hallucinations under severe degradation, but the rate drops meaningfully.

Image Detail Level and What It Costs

This is the cost decision that matters most before you write any code. The OpenAI vision API exposes two explicit detail levels:

detail: low. Always 85 tokens flat, image downsampled to 512×512. Usable for dominant colour or shape detection. Insufficient for small numerals on a meter display.
detail: high. 85 base tokens plus 170 tokens per 512×512 tile required to tile the image. For gpt-6-astra, the tokenizer applies a 2,500-patch budget with a 1.2× multiplier. A 1024×1024 image costs ~1,229 tokens; a 2048×2048 image is capped at 1,600×1,600 and costs ~3,000 tokens.

At April 2026 pricing ($0.015/1K input tokens), a 2048×2048 meter image costs roughly $0.045 per read. At 96 reads per day per meter, that is about $4.32 per meter per day, which matters at scale. The practical answer is to resize images server-side to 1024×1024 before sending, cutting per-read token cost by more than half while retaining enough resolution for 5-digit OCR.

Critically, the vision_detail parameter defaults to auto if omitted. Do not rely on auto in production. Pin it explicitly or the model selects detail level based on image size and you lose cost predictability.

Note: GPT-4.5 costs $75/M input tokens and is not a viable option for a high-volume pipeline. o4-mini was retired from the API on February 13, 2026. Use gpt-6-astra, which is the reference model in the official OpenAI .NET docs as of June 30, 2026.

The Vision Call in C

The OpenAI .NET SDK v2 uses the OpenAI.Chat namespace. The pattern is: construct a ChatClient, build a UserChatMessage from a ChatMessageContentPart[] array, call CompleteChatAsync.

using OpenAI.Chat;
using System.Text.Json;

public sealed class MeterVisionClient
{
    private readonly ChatClient _client;

    public MeterVisionClient(string apiKey)
    {
        _client = new ChatClient("gpt-6-astra", apiKey);
    }

    public async Task<string?> ReadMeterAsync(
        string imagePath,
        CancellationToken ct = default)
    {
        byte[] imageBytes = await File.ReadAllBytesAsync(imagePath, ct);
        var imageData = BinaryData.FromBytes(imageBytes);

        var systemPrompt = ChatMessageContentPart.CreateTextPart(
            "You are reading a utility meter display. The display shows exactly 5 " +
            "numeric digits with no decimal point. Return ONLY a JSON object with a " +
            "single field \"reading\". If any digit is unclear or you are not confident, " +
            "return {\"reading\": null} instead of guessing. Do not infer or complete digits.");

        var imagePart = ChatMessageContentPart.CreateImagePart(
            imageData,
            "image/png",
            ChatImageDetailLevel.High);  // explicit, never rely on auto

        var message = new UserChatMessage(
            new[] { systemPrompt, imagePart });

        ChatCompletion response = await _client.CompleteChatAsync(
            new[] { message },
            cancellationToken: ct);

        string rawContent = response.Content[0].Text;

        // Parse the JSON response safely
        using JsonDocument doc = JsonDocument.Parse(rawContent);
        if (doc.RootElement.TryGetProperty("reading", out JsonElement readingEl)
            && readingEl.ValueKind != JsonValueKind.Null)
        {
            return readingEl.GetString();
        }

        return null; // model reported uncertainty
    }
}

A few things worth noting: ChatImageDetailLevel.High is the enum member, do not pass a raw string. The image MIME type must match the actual encoding; sending a JPEG with "image/png" causes silent model degradation. If you are fetching images from a camera endpoint over HTTP, replace File.ReadAllBytesAsync with HttpClient.GetByteArrayAsync.

The Validation Layer

A raw model response, even a non null one, never goes straight into your data store. Every reading passes through four gates. Any failure routes to quarantine.

Gate 1: Digit Count

The model returns a string. Validate its length against the known display width, check it is all numeric, and reject leading zeros where the meter format prohibits them.

Gate 2: Monotonicity

Utility meters are cumulative. A reading lower than the last stored value is a hard failure. No exceptions. This single rule eliminates a large class of confident hallucinations.

Gate 3: Rate-of-Change Bounds

Compare the delta against a physical maximum. A domestic power meter cannot accumulate more than, say, 10 kWh between two 15-minute reads. Anything outside the plausible envelope is flagged regardless of direction.

Gate 4: Quarantine and Human Review

Any reading that fails one or more gates is written to a quarantine store, not the time-series database. The quarantine record includes the original image path, the raw model response string, the previously accepted reading, and the name of the rule that failed.

public sealed record MeterReading(string RawValue, decimal ParsedValue, DateTimeOffset Timestamp);

public sealed record QuarantineRecord(
    string ImagePath,
    string RawModelResponse,
    MeterReading? PreviousReading,
    string FailedRule,
    DateTimeOffset QuarantinedAt);

public sealed class MeterReadingValidator
{
    private const int ExpectedDigitCount = 5;
    private const decimal MaxDeltaPerInterval = 10m; // kWh per 15-minute window

    public ValidationResult Validate(
        string? rawReading,
        MeterReading? previous,
        DateTimeOffset readingTime)
    {
        if (rawReading is null)
            return ValidationResult.Quarantine("ModelReportedUncertainty");

        // Gate 1: digit count and format
        if (rawReading.Length != ExpectedDigitCount || !rawReading.All(char.IsAsciiDigit))
            return ValidationResult.Quarantine("DigitFormatInvalid");

        if (!decimal.TryParse(rawReading, out decimal parsed))
            return ValidationResult.Quarantine("ParseFailure");

        if (previous is null)
            return ValidationResult.Accept(parsed); // first reading, no history to compare

        // Gate 2: monotonicity
        if (parsed < previous.ParsedValue)
            return ValidationResult.Quarantine("MonotonicityViolation");

        // Gate 3: rate-of-change
        decimal delta = parsed - previous.ParsedValue;
        if (delta > MaxDeltaPerInterval)
            return ValidationResult.Quarantine("RateOfChangeTooHigh");

        return ValidationResult.Accept(parsed);
    }
}

public sealed class ValidationResult
{
    public bool IsAccepted { get; private init; }
    public decimal? AcceptedValue { get; private init; }
    public string? FailedRule { get; private init; }

    public static ValidationResult Accept(decimal value) =>
        new() { IsAccepted = true, AcceptedValue = value };

    public static ValidationResult Quarantine(string rule) =>
        new() { IsAccepted = false, FailedRule = rule };
}

The FailedRule string is what operators see in the review queue. Specific names like MonotonicityViolation make triage fast. A review record with the original image attached is what makes the review useful. Without it an operator cannot tell whether the failure was model error, a lens problem, or a genuine meter fault.

Operational Notes

Resize before sending. A 4096×512 panoramic image costs approximately 2,458 tokens at detail: high. Most meter displays photograph well at 1024×1024 and that halves your cost versus sending full-resolution camera output.

Log raw model responses. Even accepted readings should retain the raw JSON string in an audit column. When a hallucination slips through (it will, eventually), you need the evidence to distinguish model error from a genuine meter anomaly.

Tune rate bounds per meter class. A 15-minute window and 10 kWh ceiling suits a domestic installation. Industrial meters warrant different bounds, and the validation layer should accept those as configuration rather than constants.

Quarantine rate is a signal. If more than a few percent of reads hit quarantine for a specific meter, the problem is usually physical, a dirty lens, water ingress, or a mounting angle that creates constant glare, not model quality. Surface the quarantine rate as a health metric per device.

Vision OCR for meter reading is viable in production. The failure modes are real and documented, but they are bounded and catchable. The validation layer is not optional overhead; it is the feature that turns a demo into a system you can operate.

Edge AI with .NET, Part 2: Arduino Firmware That Sleeps Most of Its Life

Mehdi Mohseni — Wed, 07 Oct 2026 13:00:43 +0000

The Core Constraint That Changes Everything

When you deploy a battery-powered camera to a remote location, the firmware's main job is not capturing images, it is sleeping. The M5Stack Timer Camera X draws around 150 to 240 mA when active with Wi-Fi on. In deep sleep it drops to roughly 10 µA. That difference of four orders of magnitude is what turns a "replaces batteries every day" device into a "check it quarterly" device.

Every design decision in this firmware flows from that constraint. No persistent connections. No polling loops. No idle delays. Wake, do the minimum work, sleep.

This is Part 2 of a series on edge AI with .NET. Part 1 covered the server-side inference pipeline in ASP.NET Core. Here we focus entirely on the ESP32 firmware written in Arduino C++.

Two Firmware Modes

The firmware boots into one of two modes, selected by a flag stored in NVS:

Config mode is for initial setup and camera aiming. The device stays awake, performs a USB handshake over serial, accepts Wi-Fi credentials and an endpoint URL, and streams framed JPEG previews so a human can point the lens before deploying the hardware.

Production mode is everything else. On every wake cycle the device reads config from NVS, initialises the camera, connects to Wi-Fi, captures a JPEG, POSTs it over HTTPS to the .NET inference endpoint, then calls esp_deep_sleep_start(). The whole active window targets three to five seconds.

The mode switch is triggered by holding the reset button during a special boot sequence. In practice you write a CLI tool, covered in Part 3, that sets the NVS flag over USB before deployment.

Storing Config in NVS Instead of Hardcoding

Hardcoding Wi-Fi credentials in firmware is a deployment anti-pattern. When the password rotates you reflash every device. NVS solves this: it's a separate flash partition with built-in wear levelling, it survives reboots and sketch reflashes, and the Preferences.h Arduino wrapper is simple enough to use correctly on the first attempt.

Key constraints: namespace names and key names must each be 15 characters or fewer. That is not boilerplate in the docs. Go over it and writes fail silently.

// nvsConfig.h, NVS read/write helpers
#include <Preferences.h>

static Preferences prefs;

struct DeviceConfig {
  char ssid[64];
  char password[64];
  char endpoint[128];
  uint32_t intervalSeconds;
  bool productionMode;
};

bool loadConfig(DeviceConfig& cfg) {
  prefs.begin("camcfg", /*readOnly=*/true);
  bool valid = prefs.isKey("ssid") && prefs.isKey("endpoint");
  if (valid) {
    prefs.getString("ssid", cfg.ssid, sizeof(cfg.ssid));
    prefs.getString("pass", cfg.password, sizeof(cfg.password));
    prefs.getString("endpoint", cfg.endpoint, sizeof(cfg.endpoint));
    cfg.intervalSeconds = prefs.getUInt("interval", 900);
    cfg.productionMode  = prefs.getBool("prodmode", false);
  }
  prefs.end();
  return valid;
}

void saveConfig(const DeviceConfig& cfg) {
  prefs.begin("camcfg", /*readOnly=*/false);
  prefs.putString("ssid",     cfg.ssid);
  prefs.putString("pass",     cfg.password);
  prefs.putString("endpoint", cfg.endpoint);
  prefs.putUInt("interval",   cfg.intervalSeconds);
  prefs.putBool("prodmode",   cfg.productionMode);
  prefs.end();
  // No explicit commit needed, Preferences handles it.
}

Note the key name "pass" rather than "password". That is the 15 character ceiling in action.

For state that only needs to survive across deep-sleep cycles (not across reflashes), prefer RTC_DATA_ATTR. It's faster, doesn't wear flash, and is the right tool for a retry counter or a frame sequence number:

RTC_DATA_ATTR uint8_t failureStreak = 0;
RTC_DATA_ATTR uint32_t frameCounter  = 0;

These are declared at file scope. On the first power-on they initialise to zero. On every subsequent wake from deep sleep their values are preserved. On a hard reset or power loss, they reset.

Deep Sleep Timer Configuration

Configuring the wakeup timer is two lines, but placement matters. Call them immediately before esp_deep_sleep_start(), not at the top of setup():

void goToSleep(uint32_t intervalSeconds) {
  uint64_t wakeupUs = (uint64_t)intervalSeconds * 1000000ULL;
  esp_sleep_enable_timer_wakeup(wakeupUs);
  Serial.println("Entering deep sleep.");
  Serial.flush(); // Drain UART before power rail drops
  esp_deep_sleep_start();
  // Execution never reaches here. On wake, setup() runs again.
}

The critical mental model: deep sleep is not suspend. There is no resume. When the BM8563 RTC fires the wakeup signal, the ESP32 boots from the beginning. setup() runs. loop() runs. Every wakeup is a cold boot of user code. Everything you allocated in the previous cycle is gone.

GPIO wakeup sources have their own gotcha on the base ESP32: GPIO 12 is a strapping pin that affects flash voltage at boot. On the Timer Camera X it is already used for the BSP I2C bus, so do not attach external wakeup triggers to it. The safe external wakeup pins on the base ESP32-D0WDQ6-V3 are 4, 13, 14, 15, 25, 26, and 27.

Capture and POST Over HTTPS

The Timer Camera X has 8 MB of PSRAM, which makes it less likely than a bare ESP32 to run out of heap during a TLS handshake. But the TLS stack still operates out of main DRAM (520 KB total), and a full handshake costs roughly 40 to 60 KB there. Be conservative: capture the image first, then bring Wi-Fi up, then open the TLS connection.

Always pin a root CA certificate. client.setInsecure() exists, is useful in development, and must never reach production, because it disables chain validation completely.

#include <WiFiClientSecure.h>
#include <HTTPClient.h>
#include "esp_camera.h"

// Root CA for your inference endpoint (PEM format, trimmed for brevity)
static const char* ROOT_CA = \
"-----BEGIN CERTIFICATE-----\n"
"MIIDdzCCAl+gAwIBAgIEAgAAuTAN...\n"
"-----END CERTIFICATE-----\n";

bool captureAndPost(const DeviceConfig& cfg) {
  // 1. Initialise camera (config omitted, pin assignments are Timer Camera X specific)
  camera_fb_t* fb = esp_camera_fb_get();
  if (!fb) {
    Serial.println("Camera capture failed");
    return false;
  }

  // 2. Connect Wi-Fi, with a hard timeout
  WiFi.begin(cfg.ssid, cfg.password);
  unsigned long wifiStart = millis();
  while (WiFi.status() != WL_CONNECTED && millis() - wifiStart < 12000) {
    delay(200);
  }
  if (WiFi.status() != WL_CONNECTED) {
    esp_camera_fb_return(fb);
    return false;
  }

  // 3. POST the JPEG
  WiFiClientSecure client;
  client.setCACert(ROOT_CA);
  HTTPClient http;
  http.begin(client, cfg.endpoint);
  http.addHeader("Content-Type", "image/jpeg");
  http.addHeader("X-Frame-Id",   String(frameCounter));

  int httpCode = http.POST(fb->buf, fb->len);
  esp_camera_fb_return(fb);
  http.end();
  WiFi.disconnect(true);

  return (httpCode == 200 || httpCode == 204);
}

The brown-out detector is worth mentioning here. The Timer Camera X docs explicitly recommend disabling it in setup() via WRITE_PERI_REG(RTC_CNTL_BROWN_OUT_REG, 0) because the combined current spike from camera initialisation and Wi-Fi association can trip it on a partially discharged battery. It's an uncomfortable thing to disable, but the alternative is random reboot loops in the field.

What Happens When the POST Fails

The device is about to sleep. The POST returned a 5xx, or Wi-Fi never associated, or the TLS handshake timed out. What now?

Three options, in order of battery cost:

Retry immediately. Spend another association cycle and another TLS handshake. Reasonable if failureStreak is zero, because transient errors happen. One retry, bounded by the same 12-second Wi-Fi timeout.
Flag and sleep. Increment failureStreak in RTC memory, sleep for the normal interval. On the next wake, check the streak. After three consecutive failures, double the sleep interval (exponential backoff) to avoid hammering a dead endpoint on a slow battery drain.
Drop the frame. When failureStreak exceeds a threshold (say, 10), accept the data loss and reset the counter. A camera trap in the field with a dead upstream is not going to recover by staying awake.

The worst outcome is a firmware that retries Wi-Fi indefinitely on a dead network at 150+ mA. The battery will be flat in under a day. Always sleep on failure.

Config Mode: Serial Protocol Framing

In config mode the device listens for simple framed commands over USB serial and streams JPEG previews back. Framing matters, because raw bytes from the JPEG stream contain any byte value including 0xFF 0xD8 (JPEG SOI) and 0xFF 0xD9 (EOI), so you cannot use a single-byte delimiter. A minimal approach: a four-byte length-prefixed frame with a fixed magic header.

The host-side .NET tool (Part 3) sends CFG\n<key>=<value>\n commands and reads FRAME\n<4-byte big-endian length><jpeg bytes> responses. The firmware loop runs esp_camera_fb_get() every 200 ms, writes the frame header, writes the JPEG buffer, returns the frame buffer, and polls Serial.available() for config commands. No deep sleep, no HTTPS, just a tight preview loop until the host sends PRODMODE=1 and the device writes NVS and reboots into production mode.

Power Budget in Practice

With a 140 mAh battery and a 15-minute capture interval:

Active window: ~4 seconds at ~180 mA average = 0.2 mAh per cycle
Sleep window: ~896 seconds at ~10 µA = ~0.0025 mAh per cycle
Total per cycle: ~0.2025 mAh
Cycles per day: 96
Daily consumption: ~19.4 mAh
Battery life on internal battery alone: roughly 7 days

That's the internal 140 mAh cell. Add a 3000 mAh USB power bank and you get five months on a 15-minute interval. Stretch to hourly and you're looking at years. The arithmetic is just as unforgiving in the other direction. Leave Wi-Fi up for 30 seconds instead of 4 and battery life drops by a factor of seven.

Measure actual current with a dedicated power analyser (Nordic PPK2 or similar). Ordinary multimeters introduce enough burden voltage to affect ESP32 behaviour near its minimum operating voltage, and you'll get misleading readings in deep sleep due to the high dynamic range between active and sleep current.

What's Next

Part 3 covers the .NET configuration tool, a small console app that handles the USB serial handshake, renders the JPEG preview stream to the terminal (using Spectre.Console's image support), accepts Wi-Fi credentials, and writes the production config to the device over the framed serial protocol described above. The firmware is deliberately dumb; the host tool is where the UX lives.

DEV Community: Mehdi Mohseni

Function Calling in C#: Building Tools Your AI Agent Can Call

What Function Calling Actually Is

Setting Up: The SDK

Defining a Tool in C

The Agentic Loop in C

Doing It Well: Validation, Errors, and Idempotency

Real Tools: Querying a Database

Safety and Production Concerns

Wrapping Up

PostgreSQL jsonb with EF Core 10: Mapping, Indexing and Querying Document Columns

The Problem Worth Solving

Mapping: Complex Types with ToJson()

What LINQ Translates To

Indexing: The Part Most Teams Get Wrong

GIN with jsonb_ops (Default)

GIN with jsonb_path_ops

Expression B-Tree for What LINQ Generates

When jsonb Beats a Separate Table

Updates: Whole Document or One Property

Practical Checklist

Relationships in a Relational Database: MySQL vs NebulaGraph from 1 Million to 500 Million Rows

The model most of us start with

The questions

Why the recursive CTE gets so expensive

The third method: a breadth-first search in C

The setup

Results: the relationship question

Results: the neighbourhood question

Results: a graph with hubs

What the numbers say

Limits of this benchmark

When a relational database is still the right choice

Run it yourself

Are Coding Agents Getting Worse, or Have They Lost Their Teachers?

First, is the code really getting worse?

Is it the model?

The codebase is the agent's example

The loop: when the agent learns from itself

A child who learns from the people around it

What a team can do

Conclusion

References

The Narrow Sidewalk: Where Developers Stand When Agents Write the Code

What the Numbers Say About Both Sides

Green Checks, Wrong Meaning

Fail-Open Defaults Hide Behind Healthy Deployments

Test Both Directions

The Same Patterns, Again and Again

Walking the Sidewalk

Why Agents Need a Knowledge Base: The Gaps an LLM Can't Fill by Itself

What the model actually knows

The first gap: time

The second gap: knowledge that is too specialised

The third gap: knowledge that was never public

The fourth gap: knowing where an answer came from

Why a bigger context window doesn't solve it

The three shapes a knowledge base takes

Let the agent decide when to look

The risk of a stale knowledge base

Where to start

References

Why the Agile Coach Is Not a Luxury in a Software Team

The tech lead can't be the coach of their own team

Two views that a technical leader doesn't have by default

Agile is about performance, not about less pressure

Remote teams need it even more

The ceremonies are the glue of the team

What I would suggest to a company that is thinking about this role

References

Keeping Your Knowledge Graph Current Without a Dedicated Team

The Maintenance Problem Nobody Plans For

Where the Edges Come From

The Reconciliation Worker

Provenance on Every Edge

Conflict Handling: When Two Sources Disagree

Deciding When an Edge Is Stale

Querying for Stale Edges

Honest Limits

Practical Checklist Before You Ship

Mapping: Complex Types with `ToJson()`

GIN with `jsonb_ops` (Default)

GIN with `jsonb_path_ops`

When `jsonb` Beats a Separate Table