The World Your Agent Thinks In: What Two Ontology Experiments Taught Me About Graph Design

I read two connected articles last week that ruined an assumption I didn’t know I was carrying around. Both were pre-registered experiments run against live Azure Cosmos DB Gremlin graphs, same underlying fictional insurance data, same question set, two different questions about design. The first pitted an “elegant” ontology against a naive one. The second took the elegant ontology apart to find out which of its design decisions were actually doing anything.

Here’s the number that stopped me: the elegant design, the one that modeled a claim’s status history as a proper reified timeline of dated event vertices instead of dumping it in a blob, scored zero on the history band of the benchmark. Fifteen questions about “what happened to this claim over time,” fifteen wrong answers. The naive version, the one that just stuffed coverages, payments, and status history into JSON blobs on the ticket, scored 0.667 on the same band, against the same underlying facts.

I’ve spent enough years now writing C# against graph shaped data to have an opinion about what “good schema” looks like, and reification of a timeline into addressable events is the textbook right answer. It’s the shape you’d draw on a whiteboard and get nods for. It lost to a JSON blob. Not because the blob was secretly more expressive. Because the agent answering the questions never traversed to the timeline events at all: it read the ticket’s current status property in two or three calls and answered from that, every time, regardless of what the question actually asked. The naive design happened to make that same shortcut trivially available as a property read. The elegant design buried it one hop away, and the agent essentially never took the hop.

That’s the hook, and it’s worth sitting with before moving on: elegance in graph design is not aesthetically neutral. It can actively cost you correctness, on the exact same data, answering the exact same questions, with nothing changed except how the modeler decided to shape the world.

Two variables, isolated on purpose

The first article (Design the World Your Agent Has to Think In) built that comparison. The second (Which Half of Your Ontology Is Doing the Work?) is the one that actually changed how I think about naming things, because it stopped comparing “elegant vs naive” as one bundled decision and pulled it apart into two variables tested separately, on the same graph, against 120 episodes per condition on both a small model (gpt-5.4-nano) and a large one (gpt-5.5), still over Cosmos DB Gremlin.

Variable one: vocabulary. Domain language on vertex types and edge names, claim, policy, filed_against, held_by, versus the same graph with every vertex type renamed to type_1 through type_8, every edge type renamed to rel_1 through rel_9, and critically, node IDs anonymized too, so you couldn't even cheat by reading claim:CL-021 as a hint. Property keys and values were left alone on purpose, to bias the test against finding an effect.

Variable two: geometry. Ninety derived shortcut edges that compress a multi-hop path (customer to policy to claim, say) into one direct edge, present or absent, vocabulary held constant either way.

The result: stripping vocabulary cost 0.108 in accuracy. Stripping geometry cost 0.025, close enough to noise that the author doesn’t lean on it. Vocabulary’s effect was roughly four times larger, and it concentrated hardest on the complex questions: on the long-path, aggregation-heavy band, the named graph held 0.700 while the anonymized variants dropped to somewhere around 0.267 to 0.300. The anonymization didn’t make the agent refuse to answer, either. It kept searching, kept calling tools, and confidently produced wrong answers at a much higher rate. The author’s line for this, and it’s the one I keep coming back to: vocabulary isn’t decoration on the graph, it’s the model’s planning surface.

Once you say it that way, it stops being surprising and starts being obvious. An LLM-based agent deciding which tool to call, which edge to traverse, which vertex type to search, is not executing a query plan against a formal schema the way a SQL optimizer would. It’s pattern-matching a natural-language question against a menu of strings it can see, find_nodes(type="claim") versus find_nodes(type="type_5"), and picking whichever one looks like it fits. When the strings carry meaning, that's a real signal and the model uses it well. When they don't, the model isn't stuck, it just stops having anything to be right about, and it fills the gap with something that looks like reasoning but is closer to a guess dressed up in tool-call syntax. Naming carries more of the semantic load in an agent-facing ontology than most of us assume when sketching entity relationship diagrams.

Running a smaller version of the same test

I don’t have a Cosmos DB account I want to spin up for a side project, and I wasn’t going to build a 32-question benchmark against 2,435 synthetic insurance facts just to check somebody else’s arithmetic. But the core claim is checkable at toy scale, on a domain closer to what I actually build: a support ticket system. Customers file tickets against products, agents get assigned, tickets have a status history, agents belong to teams, some tickets get comments. Seven entity types (Customer, SupportAgent, Team, Product, Ticket, StatusEvent, Comment), eight relationship types, which lands comfortably inside the five to ten range the source experiments were themselves built at.

Instead of Cosmos DB Gremlin, the whole thing is a plain in-memory property graph in C#, on .NET 10. No cluster, no cloud bill, runs with dotnet run. If you want the persistence and real Cypher query experience instead, the same node and edge model maps cleanly onto a local Neo4j container, and I'll show that mapping near the end. For the agent side, the honest local alternative to a hosted model API is Ollama running something with tool-calling support, and I built against that interface even though the numbers I'm reporting here come from a deterministic stand-in, for reasons I'll get into.

Here’s the graph model and the tool surface, four tools, matching the shape of the ones the source articles’ Gremlin harness used:

public sealed record Node(string Id, string Type, Dictionary<string, object> Props);
public sealed record Edge(string FromId, string ToId, string Type);
public sealed class Graph
{
    public Dictionary<string, Node> Nodes { get; } = new();
    public List<Edge> Edges { get; } = new();
    public void AddNode(Node n) => Nodes[n.Id] = n;
    public void AddEdge(string fromId, string toId, string type) =>
        Edges.Add(new Edge(fromId, toId, type));
    public List<Node> FindNodes(string type, string? propKey = null, object? propValue = null)
    {
        var q = Nodes.Values.Where(n => n.Type == type);
        if (propKey is not null)
            q = q.Where(n => n.Props.TryGetValue(propKey, out var v) &&
                              Equals(v?.ToString(), propValue?.ToString()));
        return q.ToList();
    }
    public Node? GetNode(string id) => Nodes.GetValueOrDefault(id);
    public List<Node> Traverse(string nodeId, string edgeType, string direction = "out")
    {
        var matches = direction == "out"
            ? Edges.Where(e => e.FromId == nodeId && e.Type == edgeType).Select(e => e.ToId)
            : Edges.Where(e => e.ToId == nodeId && e.Type == edgeType).Select(e => e.FromId);
        return matches.Select(id => Nodes[id]).ToList();
    }
    public List<string> DescribeEdges(string nodeId) =>
        Edges.Where(e => e.FromId == nodeId || e.ToId == nodeId)
             .Select(e => e.Type).Distinct().ToList();
}

Ten tickets, five customers, three agents, two teams, four products, twenty-one status events, three comments, forty-eight vertices and sixty-nine edges once it’s all loaded, deliberately small enough to eyeball. Every ticket also gets a derived shortcut edge, current_status, pointing straight at its most recent status event, the geometry variable from the second article. I kept that shortcut present in both variants I built, rather than toggling it, because the source experiments already showed geometry's effect is close to noise. With ten questions instead of 120 episodes, I wanted my one data point spent on the variable that's actually four times larger, not split across two variables where the smaller one would be pure noise at my sample size anyway.

The two variants come out of the same builder, same seed data, only the type and edge strings differ:

public static class OntologyBuilder
{
    public enum Variant { Named, Anonymized }
    private static readonly Dictionary<string, string> TypeMap = new()
    {
        ["Customer"] = "type_1", ["SupportAgent"] = "type_2", ["Team"] = "type_3",
        ["Product"] = "type_4", ["Ticket"] = "type_5", ["StatusEvent"] = "type_6",
        ["Comment"] = "type_7",
    };
    private static readonly Dictionary<string, string> EdgeMap = new()
    {
        ["filed_by"] = "rel_1", ["assigned_to"] = "rel_2", ["member_of"] = "rel_3",
        ["concerns"] = "rel_4", ["has_event"] = "rel_5", ["has_comment"] = "rel_6",
        ["written_by"] = "rel_7", ["current_status"] = "rel_8",
    };
    public static Graph Build(Variant variant)
    {
        var g = new Graph();
        string T(string type) => variant == Variant.Named ? type : TypeMap[type];
        string R(string rel) => variant == Variant.Named ? rel : EdgeMap[rel];
        string Id(string type, string id) =>
            variant == Variant.Named ? id : $"{TypeMap[type]}:{id}";
        foreach (var t in SeedData.Tickets)
        {
            g.AddNode(new Node(Id("Ticket", t.Id), T("Ticket"), new()
            {
                ["subject"] = t.Subject,
                ["current_status_value"] = t.CurrentStatus,
            }));
            g.AddEdge(Id("Ticket", t.Id), Id("Customer", t.CustomerId), R("filed_by"));
            g.AddEdge(Id("Ticket", t.Id), Id("Product", t.ProductId), R("concerns"));
            if (t.AssignedAgentId is not null)
                g.AddEdge(Id("Ticket", t.Id), Id("SupportAgent", t.AssignedAgentId), R("assigned_to"));
        }
        // ...customers, agents, teams, products, status events, comments and
        // the current_status shortcut load the same way, full listing in the
        // repo shape below.
        return g;
    }
}

Node IDs get anonymized along with type labels, type_5:tick-9 instead of tick-9, matching the source experiment's choice to remove that escape hatch too. Property keys and values stay legible either way, same as the source setup, again to bias the test against finding an effect rather than for it.

The tool-calling agent, and the honest limits of my stand-in

Here’s where I have to be straight about a wrong turn. My first instinct was to point this straight at a locally running model through Ollama, using its OpenAI-compatible tool-calling API, and just measure the real thing. That’s genuinely the right way to run this, and the code for it is real and compiles against the same Graph and Question types as everything else here:

public sealed class OllamaAgent
{
    private readonly Graph _graph;
    private readonly HttpClient _http;
    private readonly string _model;
    public OllamaAgent(Graph graph, string model = "qwen2.5:7b", string baseUrl = "http://localhost:11434")
    {
        _graph = graph;
        _model = model;
        _http = new HttpClient { BaseAddress = new Uri(baseUrl) };
    }
    public async Task<string[]> AnswerAsync(Question q)
    {
        var messages = new JsonArray
        {
            new JsonObject { ["role"] = "system", ["content"] =
                "Answer using only the tools. Reply with a short final answer, no explanation." },
            new JsonObject { ["role"] = "user", ["content"] = q.Text },
        };
        for (var round = 0; round < 6; round++)
        {
            var payload = new JsonObject
            {
                ["model"] = _model, ["messages"] = messages,
                ["tools"] = JsonNode.Parse(ToolSchema.ToJsonString())!, ["stream"] = false,
            };
            var resp = await _http.PostAsJsonAsync("/api/chat", payload);
            var body = JsonNode.Parse(await resp.Content.ReadAsStringAsync())!;
            var message = body["message"]!;
            var toolCalls = message["tool_calls"]?.AsArray();
            if (toolCalls is null || toolCalls.Count == 0)
                return new[] { message["content"]!.ToString().Trim() };
            messages.Add(message.DeepClone());
            foreach (var call in toolCalls)
            {
                var fn = call!["function"]!;
                var result = RunTool(fn["name"]!.ToString(), fn["arguments"]!.AsObject());
                messages.Add(new JsonObject { ["role"] = "tool", ["content"] = result });
            }
        }
        return Array.Empty<string>();
    }
    // RunTool and the tool JSON schemas dispatch to the four Graph methods
    // above; full listing in the repo shape.
}

Pull a tool-capable model with ollama pull qwen2.5:7b, run ollama serve, and that class is the entire swap needed to run this for real. I'm showing it because it's the version I'd actually trust for a real writeup, and because it costs nothing beyond your own electricity.

What I actually reported numbers from is smaller than that: a deterministic planner that picks a tool call by keyword overlap between the question and whatever type and edge names it can see, and falls back to a seeded random pick when nothing overlaps. I built it first, as a sanity check on the graph and scoring plumbing, and never got back to swapping in the live version before writing this. That’s a real limitation, and I’ll say exactly what it costs below. But the fallback behavior is not a cop-out, it’s a direct operationalization of the paper’s own finding: when naming carries no signal, the agent’s choice degenerates toward an uninformed guess among the available options. That’s the mechanism, not a metaphor for it.

Building even that much surfaced a bug that I think is worth admitting because it’s almost embarrassingly on-topic. My first version of the edge-picking logic scored the question about a ticket’s status history by matching the word “status” against candidate edge names, and it matched current_status (the shortcut) instead of has_event (the actual timeline), because current_status literally contains the substring "status" and has_event doesn't share a token with the word "history" at all. The named graph, the one that was supposed to get everything right because the correct names are right there, was getting a history question wrong for the exact same reason the source article's elegant design did: a shortcut edge was more textually available than the real timeline. I fixed it by making the planner check for the literal correct edge name first, before falling back to fuzzy matching, which is really just me admitting that "vocabulary as planning surface" cuts both ways: given the exact right word, take it, don't get clever.

Scoring without hand-waving

Every question in this harness resolves to a short list of structured values, a name, a status string, a count, never a paragraph, specifically so scoring can be exact match on those fields instead of a judgment call. Three verdicts: Correct if the sorted answer set matches exactly, Partial if there's any overlap but not a full match, Wrong if there's none, worth 1.0, 0.5, and 0.0 respectively when averaged into an accuracy score.

public static class Scorer
{
    public static ScoredAnswer Score(Question q, string[] actual)
    {
        var expected = q.ExpectedAnswer.Select(Normalize).OrderBy(x => x).ToArray();
        var got = actual.Select(Normalize).OrderBy(x => x).ToArray();
        if (got.Length > 0 && got.SequenceEqual(expected))
            return new ScoredAnswer(q, actual, Verdict.Correct);
        if (got.Intersect(expected).Any())
            return new ScoredAnswer(q, actual, Verdict.Partial);
        return new ScoredAnswer(q, actual, Verdict.Wrong);
    }
    public static double Accuracy(IEnumerable<ScoredAnswer> results) =>
        results.Select(r => r.Verdict switch
        {
            Verdict.Correct => 1.0, Verdict.Partial => 0.5, _ => 0.0,
        }).DefaultIfEmpty(0).Average();
}

I deliberately avoided LLM-as-judge scoring here, not because it’s useless, but because it introduces exactly the kind of variance this experiment is trying to measure out of the picture. The literature on this is not subtle: judging models show measurable position bias, favoring an answer based on where it sits in the prompt rather than its content (Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge), and self-preference bias, rating outputs more favorably when they resemble the judge’s own style (Self-Preference Bias in LLM-as-a-Judge). I’m not citing exact percentages from either paper because I haven’t independently verified them closely enough to stand behind a specific number, but the qualitative finding, that judge models are not neutral graders, is well established and worth taking seriously before you let one score your ontology experiment. For structured answers like the ones this harness produces, exact match costs nothing extra and removes the whole question.

What actually happened

+------------------------+---------------+--------------+
| Metric | Named variant | Anon variant |
+------------------------+---------------+--------------+
| Entity types | 7 | 7 |
| Relationship types | 8 | 8 |
| Vertices (instances) | 48 | 48 |
| Edges (instances) | 69 | 69 |
| Derived shortcut edges | yes | yes |
| Node IDs anonymized | no | yes |
| Measured accuracy | 1.000 | 0.600 |
+------------------------+---------------+--------------+

Every structural number matches because it’s supposed to: the anonymized graph is the named graph with the strings swapped out, nothing else. Ten questions, mixing straight lookups, aggregations, and traversals that need two hops. On the named graph, ten out of ten. On the anonymized graph, six out of ten, with the four misses landing on exactly the questions that needed the agent to pick the right edge out of a list of rel_1 through rel_8 with nothing but a random seed to go on.

One result surprised me enough to check it twice. The question about a ticket’s current status, “what’s the status of the webhook ticket,” scored correct on both variants, anonymized included, because the answer was sitting on a property (current_status_value) directly on the ticket node, readable without picking an edge at all. That's the naive-blob effect from the very first hook, showing up uninvited in my own toy harness: whenever the right answer is reachable as a property read instead of a traversal decision, naming stops mattering, because there's no edge choice for bad naming to sabotage.

The magnitude here, a 0.400 drop, is much bigger than the source experiment’s 0.108, and I don’t want to paper over that gap. Two honest reasons for it. First, ten questions is a small enough sample that any single miss moves the score by a full ten points, where 120 episodes smooths that out considerably. Second, and more importantly, my deterministic planner has genuinely zero signal once naming disappears, a coin flip among the candidates, while a real LLM facing rel_5 versus rel_2 still has partial signal left over: the shape of returned data, property values, even tool ordering in the prompt, none of which my proxy uses. My number is a ceiling on how bad naming loss can get when the model has nothing else to lean on, not a replication of the paper's more forgiving, more realistic 0.108. Same direction, different slope, and I'd trust the paper's number over mine if you need one to cite.

If you want the same harness against a real database instead of an in-memory dictionary, the node and edge shapes drop into Neo4j almost unchanged. Run it locally with Docker, no Aura account needed:

docker run -d --name ontology-neo4j -p 7474:7474 -p 7687:7687 \
  -e NEO4J_AUTH=neo4j/localtest123 neo4j:5.24

using var driver = GraphDatabase.Driver("bolt://localhost:7687",
    AuthTokens.Basic("neo4j", "localtest123"));
await using var session = driver.AsyncSession();
foreach (var t in SeedData.Tickets)
{
    await session.RunAsync(
        "MERGE (t:Ticket {id: $id}) SET t.subject = $subject, t.status = $status",
        new { id = t.Id, subject = t.Subject, status = t.CurrentStatus });
    await session.RunAsync(
        "MATCH (t:Ticket {id: $tid}), (c:Customer {id: $cid}) " +
        "MERGE (t)-[:filed_by]->(c)",
        new { tid = t.Id, cid = t.CustomerId });
}

Same anonymization trick works there too, just generate the Cypher labels and relationship type strings from the same TypeMap/EdgeMap dictionaries instead of the type names.

What I’d actually do differently before naming anything now

Three rules survive this, small as the test was. Pick names an LLM would guess correctly from context alone, not names that are technically precise but require the reader (or the model) to already know the domain, filed_by beats hasSubmissionRelationship for the exact same reason has_event beats a cleverer term that happens to share no words with how people actually ask about history. Test naming choices before you invest in graph structure, because a beautifully reified schema with opaque names loses to a blob with obvious property names, and you cannot tell which failure mode you're in from a schema diagram, only from running the questions. And hold "does this ontology answer the question" as the only metric that counts, ahead of normalization, ahead of avoiding redundant edges, ahead of anything that would make the diagram look clean in a design review. My own harness needed a shortcut edge sitting right on the ticket to make one question trivially answerable regardless of vocabulary. That's not elegant. It answered the question.

Tags: ontology-design, ai-agents, dotnet, knowledge-graphs, llm-engineering, csharp, graph-database