I rebuilt a tiny version of Miodrag Cekikj’s drift study in .NET with SQLite and a 7B model running on my laptop. Staleness turned out to be the easy part. Entity names, merge rules and letting recency decide were the hard ones.
Every agent demo I have built in the last year had the same quiet assumption: the knowledge it reads is current. The vector index was built yesterday, the graph was extracted last week, and nobody asks what happens when the company renames a product on Tuesday.
Then I read three posts by Miodrag Cekikj in a row, and the third one, about what happens when the world moves and the ontology does not, made me uncomfortable enough to build something. Not a benchmark. A miniature, small enough that I could read every wrong answer myself.
This post has three parts. First, what Cekikj measured, with his numbers labeled as his. Second, why I think the missing piece is a runtime question as much as a modeling question, with Netflix and Thilo Hermann as the two reference points. Third, my own small experiment, the code, and what I would version if I were doing this for real.
What Cekikj measured (first-party numbers, small samples, synthetic data)
All three studies run on Naissus Pay, a fintech company that Cekikj invented and generated from one world model. Every number below comes from his posts. I did not rerun any of it, the sample sizes are small, and he himself treats results on generated data as a ceiling rather than a forecast. Please read them that way.
Study 1: Microsoft’s knowledge layer, measured. He compared classic RAG, Foundry IQ at three retrieval effort settings, and his own hybrid on 32 questions.
Study 2: Extract or load. He built the same knowledge graph twice. Once by loading it from structured source data, once by having an LLM extract it from the pages.
Study 3: What breaks when the world moves. 203 runs in which either the pages or the graph lag behind reality, plus a rename.
Study Setup Reported result (Cekikj, first-party)
------------------------ ---------------------------------------- ------------------------------------------
1. Knowledge layer classic RAG 58%
(n = 32 questions) Foundry IQ, effort minimal / low / medium 62% / 72% / 78%
his hybrid (graph + retrieval) 91%
cost about 1.5 to 2.0 cents per question
2. Extract vs load gpt-4.1 extraction 224 of 227 nodes, 477 of 487 edges, $0.59
gpt-4o-mini extraction lost about 22% of edges
GraphRAG on the gpt-4o-mini graph 52.5%, below the no-graph baseline of 60%
3. Drift (203 runs) stale pages, current graph RAG 8%, graph agent 75%, hybrid 92%
stale graph, current pages RAG 50%, graph agent 25%, hybrid 25%
product rename graph agent dropped to 30%
after three find-and-replace graph edits graph agent back to 95%
hedging 1 of 24 wrong answers carried a hedge
Three things in that table changed how I think about this.
The gpt-4o-mini row says a cheap extractor can produce a graph that makes the agent worse than having no graph at all. That is not an argument against graphs. It is an argument that the extraction step is part of the system and needs its own test.
The drift rows are symmetrical in a way that bothered me. Whichever store is stale drags down the design that trusts it, and the hybrid did not escape that. It scored 92% when the graph was right and 25% when the graph was wrong, which tells me it was mostly following the graph.
The last row is the one I keep coming back to. Twenty-four wrong answers, and only one said anything like “this might be out of date”. The agent was not just wrong. It was wrong in a confident voice, which is the version that ends up in a customer email.
Two reference points: Netflix and Hermann
The most serious production example of “model the domain once” that I know of is Netflix’s post on UDA, their Unified Data Architecture, titled Model Once, Represent Everywhere. The idea, in my own words: Netflix defines domain concepts once in an ontology framework called Upper, built on RDF and SHACL, and projects those definitions into the places data actually lives, such as GraphQL schemas, Avro records, Iceberg tables and SQL. Mappings between the conceptual model and the physical containers are kept explicitly, so a change to a concept has a defined path to every representation that depends on it.
I am not claiming Netflix built this for agents, or that they have a drift problem. My takeaway is narrower. When the ontology is the source and every representation is a projection with a recorded mapping, “is this copy stale?” becomes a question you can actually answer. When your graph is a one-off LLM extraction of a wiki, there is no such mapping, and nobody can answer it.
The second reference point is Thilo Hermann’s post on the agentic runtime flow nobody draws. His point, as I read it, is that architecture diagrams show boxes and skip the sequence: who asks what, in which order, under whose permissions, and what is left behind afterwards. Two parts of that sequence map directly onto drift. Retrieval should be filtered by the caller’s permissions before anything reaches the model. And anything an agent writes back into memory or a knowledge store should carry provenance: where it came from, and when.
That second part is the runtime layer I think Cekikj’s drift study is missing. His graph knows facts. It does not seem to know when it learned them or from which version of which page. Without that, the agent has nothing to hedge with.
So my question for the miniature was simple. If I store source version, extraction date and valid_from next to every fact, does the agent get more accurate, does it hedge when it should, and what does it cost to keep the graph patched instead of rebuilding it?
The miniature
I kept it deliberately small, so I could open results.csv and read every single wrong answer.
The company. Tessaly Freight is a made-up logistics SaaS: three products, three pricing plans, three teams, three services, a refund policy and a retention policy. The knowledge lives in 40 short pages: a pricing page, policy pages, team pages, a service catalog, FAQs, support macros, a runbook, a postmortem, release notes, meeting notes, and about a dozen pages of pure noise such as the travel policy and the office guide. Every fact sentence comes from one helper class, so the pages are templated. That makes extraction much easier than on a real wiki, and I will come back to that.
The graph. SQLite, three tables: entities, aliases, facts. Every fact row carries source_doc, source_version, extracted_at and valid_from. A fourth table, extractions, records which version of each page the graph last read. That one turned out to matter most.
Four change events , one week apart:
Event Day Kind What changes
----- ---- ------- ----------------------------------------------
1 +7 rename "Tessaly Track" becomes "Tessaly Beacon"
2 +14 policy refund window goes from 30 to 14 days
3 +21 price Pro plan goes from 49 to 59 EUR per user/month
4 +28 owner Payments service moves from team Atlas to Orion
Twenty questions. Eight controls whose answers never change, and twelve that flip: three per event. Two of them need two hops, for example “Who should I page about a Payments service outage?” needs the owning team and then that team’s lead. I ask all twenty after the initial load and after each event, so every design answers 100 questions per scenario, 30 of which have drifted at that point.
Four drift scenarios.
Scenario Pages Graph
------------- --------------------------------------- -------------------------------------------
Fresh every page that mentions a fact updated patched from the changed pages
StalePages never updated patched from the system of record
StaleGraph every page updated never touched after day 0
PartialPages only the canonical page updated patched from the changed pages
(FAQs, macros, runbooks lag behind)
StalePages and StaleGraph are Cekikj’s two drift directions. PartialPages is the one I see in real companies: someone updates the pricing page, and the sales onboarding guide, two support macros and the partner guide keep the old price for months.
Four designs.
Design What the reader gets
---------------- --------------------------------------------------------------------------
Rag top 3 pages from BM25
Graph one graph fact per question: the value most sources agree on
Hybrid the graph fact first, then the top 3 pages; the first usable answer wins
(enforced in code by the deterministic reader, only an ordering for the model)
HybridVersioned every distinct graph value with valid_from and extraction date, plus the
top 3 pages with their updated_at and a flag if the page changed since the
graph last read it; the newest claim wins, and the answer is marked as
possibly stale if sources disagree, the graph has no entry, or a page
changed after extraction
The “value most sources agree on” rule for Graph and Hybrid is not a straw man. Majority vote is a common way to merge duplicate extractions, and it looked perfectly reasonable to me when I wrote it.
Two readers. I ran everything twice.
- A deterministic reader with no model at all. It parses candidate values out of the evidence with the same patterns the extractor uses and applies the design’s rule. Every wrong answer in this run is the knowledge layer’s fault, nothing else. Like Cekikj’s numbers, this is a ceiling.
- qwen2.5–7b-instruct-1m in LM Studio on my MacBook, doing both jobs: extracting facts from pages into the graph, and answering from the evidence. The prompt asks for JSON with an answer and a possibly_stale flag, and I also count hedge phrases in the answer text.
Retrieval is BM25 in both runs. The code has an embedding path (Ollama’s nomic-embed-text, or the nomic model LM Studio ships with), but I did not run it for this post, so I have no numbers for it.
Cost. For the patch-versus-rebuild comparison I count pages read, input and output tokens, and graph writes. In the deterministic run nothing calls a model, so tokens are an estimate at four characters per token and are marked with a star. In the LM Studio run the tokens come from the server’s usage field.
Results
First, the label I asked of Cekikj, applied to myself. This is synthetic data from one world model I wrote, 40 templated pages, 20 questions, one 7B model, one run each, on one laptop. Treat every number below as a ceiling for this tiny world, not a prediction for yours.
Both runs used the same code. The deterministic run took a few seconds. The LM Studio run with qwen2.5-7b-instruct-1m took about 35 minutes on my MacBook, mostly in the extraction passes.
Accuracy by design and drift scenario
“All” is 100 answers (20 questions at 5 checkpoints). “Drifted” is the 30 of those whose answer or name had already changed at that checkpoint. The last column counts the qwen run’s wrong answers, split into stated plainly versus marked as possibly stale.
Design Scenario Deterministic qwen2.5-7b (LM Studio) qwen wrong answers
All Drifted All Drifted no hedge / hedged
--------------- ------------ ----- ------- ----- ------- -----------------
Rag Fresh 100% 100% 100% 100% 0 / 0
Rag StalePages 70% 0% 70% 0% 18 / 0 (+12 abstain)
Rag StaleGraph 100% 100% 100% 100% 0 / 0
Rag PartialPages 94% 80% 89% 63% 11 / 0
Graph Fresh 100% 100% 80% 100% 15 / 0 (+5 abstain)
Graph StalePages 100% 100% 80% 100% 15 / 0 (+5 abstain)
Graph StaleGraph 70% 0% 50% 0% 33 / 0 (+17 abstain)
Graph PartialPages 82% 40% 58% 27% 37 / 0 (+5 abstain)
Hybrid Fresh 100% 100% 100% 100% 0 / 0
Hybrid StalePages 100% 100% 86% 53% 14 / 0
Hybrid StaleGraph 82% 40% 100% 100% 0 / 0
Hybrid PartialPages 82% 40% 83% 43% 17 / 0
HybridVersioned Fresh 99% 100% 94% 100% 0 / 6
HybridVersioned StalePages 100% 100% 86% 73% 2 / 12
HybridVersioned StaleGraph 100% 100% 94% 100% 0 / 6
HybridVersioned PartialPages 99% 100% 94% 100% 0 / 6
The shape of Cekikj’s result shows up in the deterministic column almost exactly. Rag collapses when the pages are stale (0% on drifted questions), Graph collapses when the graph is stale (0%), and a hybrid that trusts the graph first inherits the graph’s failure (40%). Twelve of Rag’s StalePages misses are the rename, where the pages only know “Tessaly Track”, the question says “Tessaly Beacon”, and nothing connects the two. The other 18 are old prices, old refund windows and old owners, read off old pages without a flicker of doubt.
Then the model run did some things I did not predict.
What actually went wrong with the 7B model
The graph’s biggest problem was not drift. It was names. When qwen extracted the graph, it wrote “Starter plan” on one page and “Pro” on another, “Helix team” here and “Atlas” there. My questions ask about “Starter”, “Pro” and “Helix”. The lookup is an exact match on name or alias, so half of the control questions never found their node. That is why Graph sits at 80% even in the Fresh scenario, while scoring 100% on the drifted questions. Cekikj’s gpt-4o-mini graph losing 22% of its edges felt abstract when I read it. Watching “Pro” and “Pro plan” become two nodes with two different prices made it concrete.
When the graph handed the model nothing, the model answered anyway. For the Starter price, with zero evidence in the prompt, qwen answered “9”. For Enterprise it answered “12”, for the Pro uptime SLA “99”. The prompt says to answer “unknown” if the evidence does not contain the answer. It said “unknown” for the Helix lead and made up numbers for the other three, every single time. No hedge on any of them.
The model did not follow my “graph first” rule, and that made the plain hybrid look great. Hybrid scored 100% in StaleGraph with the model, better than its own deterministic ceiling of 82%. In the deterministic reader, “graph first” is enforced in code. With the model, it is only the order of the evidence: one graph line on top, three pages below. qwen mostly sided with the pages. In StaleGraph the pages are current, so that looks brilliant. In StalePages the pages are old, and the same habit drops the hybrid to 53% on drifted questions. The naming mess helped too, because several graph lookups returned nothing at all. I would not ship any of this as a feature. The model is quietly running its own merge policy, and I did not choose it.
The system-of-record patch missed half the graph. In StalePages, my patch writes “Pro costs 59” from the billing system. The deterministic graph had one “Pro” node. The qwen graph had “Pro” and “Pro plan”, and the patch only replaced the first, so “Pro plan costs 49” stayed and kept winning the majority vote. That is the drift study’s “three find-and-replace edits restored 95%” result seen from the other side: a find-and-replace only works if you know every spelling of the thing you are replacing.
Newest-wins amplified two extraction mistakes. This is the one that changed my recommendation. The postmortem process page says to write a postmortem “within five working days”. The pricing page says annual billing gets “two months free”. qwen extracted both as refund windows: 5 days and 2 days. In the versioned design, the pricing page changed on day 7 because of the rename, got re-extracted, and its bogus “2 days” fact now had the newest date in the graph. So the versioned hybrid answered “2” for all three refund questions right after the rename, with the correct 30 sitting right next to it. At day 0, when the real 30 and the bogus 5 had the same date, it picked 5. It did mark all six answers as possibly stale, which is the only reason I am not more upset.
The trap mostly did not fire, for an accidental reason. I planted a draft in the meeting notes on day 28: “draft numbers for 2027, nothing approved yet. The Starter plan costs €24”. In the deterministic run, newest-wins took the bait in Fresh and PartialPages (the one wrong answer in each of those rows), and it was hedged both times. In the qwen run, the model did extract the 24 into the graph, but nobody could find it because it sat on a node called “Starter plan”.
The model ignored the versioning instructions at least once. In StalePages, the evidence held a graph fact “Pro costs 59” dated after the billing change and three older pages saying 49. The prompt says the newest claim wins and to flag disagreement. qwen answered “€49” with possibly_stale: false, twice. Those are the 2 unhedged wrong answers in that row.
Hedging
This is the part I most wanted to measure, so here it is with the caveat up front: my naive designs use a prompt that has the possibly_stale field but no instructions about dates or disagreement. So the comparison below is "versioned prompt with dates" versus "plain prompt", not a clean ablation of the dates alone.
qwen2.5-7b run, 16 design-scenario cells Wrong answers Hedged Share hedged
------------------------------------------ ------------- ------ ------------
Rag, Graph, Hybrid (no dates in prompt) 160 0 0%
HybridVersioned (dates + staleness flags) 32 30 94%
HybridVersioned, hedged when the answer Fresh 9 of 94, StalePages 6 of 86,
was actually right StaleGraph 58 of 94, PartialPages 17 of 94
Cekikj reported 1 hedge in 24 wrong answers. My plain designs did worse: 0 in 160. With dates and a “page changed since the graph last read it” flag in the prompt, 30 of 32 wrong answers came with a warning.
The cost is false alarms. In StaleGraph, 58 of 94 correct answers were also flagged, mostly control questions. The pricing page changed for the Pro price, so a question about the Starter price, which never changed, still sees “this page is newer than what the graph last read” and hedges. Page-level change tracking is coarse. Users will learn to ignore a warning that fires on 62% of correct answers, so I do not think this is good enough to ship as is. It is good enough to tell me where the signal comes from.
In the deterministic run, where the hedge rules are computed in code rather than left to the model, the versioned design hedged both of its wrong answers and had the same false alarm pattern (61 of 100 correct answers in StaleGraph).
Patching versus re-extracting
All numbers from the qwen run, tokens from LM Studio’s usage field, time is wall clock on my laptop.
After event Approach Pages read Input tok Output tok Graph writes Time
------------------- --------------------------- ---------- --------- ---------- ------------ ------
1 rename full re-extract 40 9,797 1,166 113 36.7 s
patch changed pages 8 2,041 432 37 13.0 s
patch from system of record 0 0 0 2 < 0.1 s
2 refund policy full re-extract 40 9,797 1,166 113 36.6 s
patch changed pages 4 1,011 127 14 4.1 s
patch from system of record 0 0 0 6 < 0.1 s
3 Pro price full re-extract 40 9,797 1,166 113 35.9 s
patch changed pages 5 1,292 268 27 7.9 s
patch from system of record 0 0 0 2 < 0.1 s
4 Payments owner full re-extract 40 9,825 1,186 114 36.2 s
patch changed pages 7 1,749 221 24 6.9 s
patch from system of record 0 0 0 5 < 0.1 s
Repair after four full re-extract 40 9,825 1,186 114 36.3 s
missed events patch pages changed since 21 5,286 847 78 25.1 s
last extraction
On my 40 pages, patching read 4 to 8 pages per event instead of 40, about a fifth of the tokens, and the graph-only accuracy check came out the same for both (80% after every event, the same naming problem in both graphs). That ratio is the whole argument for patching, and it only gets better as the corpus grows while the change stays small.
The repair row is the one I care about most. A graph that missed four weeks of changes could be brought back by re-reading the 21 pages whose version was newer than the version in the extractions table. Without that table, the only safe option is the full rebuild. The patch is not clever. It is just a LEFT JOIN that is impossible to write if you never stored which version you read.
The system-of-record patch is almost free, and in the deterministic run it was perfect. In the qwen run it was only as good as the entity resolution behind it, which, as above, was not good.
What I would version so the agent can say “I might be stale”
I started this thinking the answer was three columns: source version, extraction date, valid_from. After reading the qwen run’s wrong answers one by one, I think those three are necessary and not nearly enough. Here is the list I would actually build with now.
Where Field Why, in terms of what broke above
-------------- ---------------------------- ----------------------------------------------------------
every page version, updated_at without these there is no "changed since" question to ask
extraction log doc_id, version_read, the repair patch (21 pages instead of 40) is a LEFT JOIN
extracted_at on this table; it also powers the "page changed since the
graph last read it" hedge
every fact source_doc, source_version lets you delete exactly what one page contributed and
re-extract only that page
every fact extracted_at "the graph learned this on day 0" is the honest version of
"the graph says"
every fact valid_from, valid_to when the world says it became true, and when it stopped;
do not overwrite, supersede
every fact extractor id model, prompt version; the "2 day refund window" came from
one re-extraction and I could not tell which run made it
every fact authority rank record > canonical page > mention > meeting notes; newest
wins only inside the same rank, or a draft beats the price
list and a pricing footnote beats the refund policy
every entity canonical id + aliases, "Pro" and "Pro plan" must be one node before anything else
normalised at write time in this list helps; a system-of-record patch is keyed by it
And three rules that are code, not schema:
- Compute the staleness signal in code and hand it to the model as a fact. The deterministic hedges were consistent. The model ignored its instructions at least twice. “Sources disagree” and “page changed after extraction” are cheap to compute and should not depend on a 7B model reading a system prompt carefully.
- Treat an empty graph lookup as an answer. “Not in the graph” is information. If the evidence list is empty, return “I don’t know” without calling the model, because given nothing, qwen invented 9, 12 and 99.
- Never let recency alone decide. Newest-wins fixed every drift case and then amplified two extraction errors and a draft. Recency is a tie-breaker inside an authority level, not a truth rule.
The schema is cheap. The hard part is the third row from the bottom of that table: deciding who is allowed to be the source of truth for which fact. That is an ontology question, which is probably why Netflix put so much effort into having one place where a concept is defined and every other representation is a recorded projection of it. In Hermann’s terms, it is also a runtime question: whatever writes back into the graph needs to leave its name and its source on the row, or the next reader cannot tell a billing record from a meeting note.
What I don’t know yet
- Whether this survives a real wiki. My pages are templated, which makes extraction far easier than real prose. The naming problem would be worse, not better, on real pages.
- Whether a bigger model fixes the naming. I only ran qwen2.5–7b. I have a 32B model in LM Studio and did not run it for this post, so I can’t tell you whether “Pro” and “Pro plan” merge by themselves at that size. My bet is that normalisation needs to be code either way.
- Embedding retrieval. The code supports it; I ran BM25 only.
- Variance. One run per configuration at temperature 0. I have not measured how much these numbers move between runs, and with 30 drifted questions per cell, a few answers is several percentage points.
- Fact-level change tracking. My “page changed” hedge fires on 62% of correct answers in StaleGraph. A fact-level diff (re-extract the page, compare only the facts the question touches) should cut that a lot. I have not built it.
The code
Six C# files and a project file, about 1,150 lines. It targets .NET 10, uses Microsoft.Data.Sqlite, and talks to either LM Studio (OpenAI-compatible API) or Ollama (native API). No cloud calls.
When I restored packages, NuGet printed a NU1903 warning about the bundled SQLitePCLRaw.lib.e_sqlite3 2.1.11. For a local experiment I left it. For anything else, check the advisory and pin a patched SQLitePCLRaw bundle before you copy this anywhere.
Run it
mkdir DriftLab && cd DriftLab
# put the files below in this folder, then:
# 1. no model: deterministic reader, regex extraction, BM25 (a few seconds)
dotnet run -c Release
With LM Studio (what I used). Start the server in the Developer tab, or with lms server start, then:
curl -s http://localhost:1234/v1/models # copy the exact model id from here
dotnet run -c Release -- --api openai --reader ollama --extractor ollama --retrieval bm25 \
--model qwen2.5-7b-instruct-1m
With Ollama in Docker instead. I wired this path and did not use it for the numbers above:
docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama
docker exec ollama ollama pull qwen2.5:7b
docker exec ollama ollama pull nomic-embed-text
dotnet run -c Release -- --reader ollama --extractor ollama --retrieval embed
On a Mac, Ollama inside Docker runs on the CPU. brew install ollama && ollama serve uses the GPU and is much faster. The run writes results.csv with every answer, the expected value, whether it was hedged, and why. That file is where all the stories above came from.
The --reader ollama and --extractor ollama flags select the model-backed reader and extractor for both servers. --api decides which API shape is spoken.
DriftLab.csproj
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.Sqlite" Version="10.0.0" />
</ItemGroup>
</Project>
World.cs: the company, the pages, the events, the questions
namespace DriftLab;
public enum Scenario { Fresh, StalePages, StaleGraph, PartialPages }
public sealed record WorldState(string TrackName, int RefundDays, int ProPrice, string PaymentsOwner)
{
public static readonly WorldState Baseline = new("Tessaly Track", 30, 49, "Atlas");
}
public sealed record ChangeEvent(int Id, string Kind, DateTime At, Func<WorldState, WorldState> Apply);
public sealed record Doc(string Id, string Title, string[] CanonicalFor, Func<WorldState, string> Body);
public sealed record Question(string Id, int Event, string Text, string Entity, string[] Hops,
Func<WorldState, string> Expected)
{
// {P} is replaced with whatever the tracking product is called at that point in time.
public string TextAt(WorldState s) => Text.Replace("{P}", s.TrackName);
public string EntityAt(WorldState s) => Entity.Replace("{P}", s.TrackName);
}
// Every fact sentence in the corpus comes from here, so docs and graph renderings use the same wording.
public static class Say
{
public static string Price(string plan, string eur) => $"The {plan} plan costs €{eur} per user per month.";
public static string Refund(string days) => $"Customers can request a refund within {days} days of purchase.";
public static string Retention(string days) => $"We keep shipment data for {days} days.";
public static string Owner(string thing, string team) => $"{thing} is owned by the {team} team.";
public static string Lead(string team, string person) => $"The {team} team is led by {person}.";
public static string Included(string product, string plan) => $"{product} is included in the {plan} plan.";
public static string Uptime(string plan, string pct) => $"The {plan} plan comes with a {pct}% uptime SLA.";
public static string Does(string product, string what) => $"{product} is our {what}.";
public static string Formerly(string now, string old) => $"{now} was formerly called {old}.";
public static string Render(string predicate, string subject, string obj) => predicate switch
{
"price" => Price(subject, obj),
"refund_days" => Refund(obj),
"retention_days" => Retention(obj),
"owned_by" => Owner(subject, obj),
"led_by" => Lead(subject, obj),
"included_in" => Included(subject, obj),
"uptime" => Uptime(subject, obj),
"does" => Does(subject, obj),
"formerly" => Formerly(subject, obj),
_ => throw new ArgumentOutOfRangeException(nameof(predicate), predicate, null)
};
}
public static class World
{
public static readonly DateTime T0 = new(2026, 9, 1, 9, 0, 0, DateTimeKind.Utc);
public static readonly ChangeEvent[] Events =
[
new(1, "rename", T0.AddDays(7), s => s with { TrackName = "Tessaly Beacon" }),
new(2, "policy", T0.AddDays(14), s => s with { RefundDays = 14 }),
new(3, "price", T0.AddDays(21), s => s with { ProPrice = 59 }),
new(4, "owner", T0.AddDays(28), s => s with { PaymentsOwner = "Orion" }),
];
public static WorldState StateAt(int k)
{
var s = WorldState.Baseline;
for (var i = 0; i < k; i++) s = Events[i].Apply(s);
return s;
}
// What a given page "knows" at step k. Fresh and StaleGraph pages follow the world,
// StalePages never change, PartialPages only update the canonical page for each change.
public static WorldState PageView(Scenario scenario, int k, Doc doc)
{
var s = WorldState.Baseline;
for (var i = 0; i < k; i++)
{
var ev = Events[i];
var apply = scenario switch
{
Scenario.Fresh or Scenario.StaleGraph => true,
Scenario.PartialPages => doc.CanonicalFor.Contains(ev.Kind),
_ => false
};
if (apply) s = ev.Apply(s);
}
return s;
}
static string TrackIntro(WorldState v) =>
(v.TrackName != WorldState.Baseline.TrackName ? Say.Formerly(v.TrackName, WorldState.Baseline.TrackName) + " " : "")
+ Say.Does(v.TrackName, "real-time shipment tracking app");
static string PaymentsLine(WorldState v) => Say.Owner("Payments service", v.PaymentsOwner);
public static readonly Doc[] Docs =
[
new("d01", "Company overview", [], v =>
$"Tessaly Freight builds software for regional freight forwarders. {Say.Does(v.TrackName, "real-time shipment tracking app")} " +
$"{Say.Does("Tessaly Dock", "warehouse slotting tool")} {Say.Does("Tessaly Route", "route planning engine")} " +
"About 140 people work across offices in Izmir, Thessaloniki and Sofia."),
new("d02", "Pricing", ["price"], v =>
$"{Say.Price("Starter", "19")} {Say.Price("Pro", v.ProPrice.ToString())} {Say.Price("Enterprise", "199")} " +
$"{Say.Included(v.TrackName, "Starter")} {Say.Included("Tessaly Dock", "Pro")} {Say.Included("Tessaly Route", "Enterprise")} " +
"Annual billing gets two months free."),
new("d03", "Refund policy", ["policy"], v =>
$"{Say.Refund(v.RefundDays.ToString())} Refunds go back to the original payment method. " +
"Partial months are not refunded after the refund window closes."),
new("d04", "Data retention policy", [], v =>
$"{Say.Retention("365")} After that, shipment records are anonymised. Invoices are kept for ten years for tax reasons."),
new("d05", "Service level agreement", [], v =>
$"{Say.Uptime("Starter", "99.5")} {Say.Uptime("Pro", "99.9")} {Say.Uptime("Enterprise", "99.95")} " +
"Credits are issued automatically when we miss the target."),
new("d06", "Product page: shipment tracking", ["rename"], v =>
$"{TrackIntro(v)} It shows live ETAs for every pallet and sends alerts when a truck is late. " +
$"{Say.Included(v.TrackName, "Starter")} {Say.Owner(v.TrackName, "Helix")}"),
new("d07", "Product page: Tessaly Dock", [], v =>
$"{Say.Does("Tessaly Dock", "warehouse slotting tool")} It suggests bin locations from order history. " +
$"{Say.Included("Tessaly Dock", "Pro")} {Say.Owner("Tessaly Dock", "Atlas")}"),
new("d08", "Product page: Tessaly Route", [], v =>
$"{Say.Does("Tessaly Route", "route planning engine")} It plans multi-drop routes with driver hour limits. " +
$"{Say.Included("Tessaly Route", "Enterprise")} {Say.Owner("Tessaly Route", "Atlas")}"),
new("d09", "Team page: Atlas", [], v =>
$"{Say.Lead("Atlas", "Derya Kaya")} {Say.Owner("Routing engine", "Atlas")} " +
(v.PaymentsOwner == "Atlas" ? PaymentsLine(v) + " " : "") + "Atlas sits on the third floor in Izmir."),
new("d10", "Team page: Orion", [], v =>
$"{Say.Lead("Orion", "Emre Aslan")} Orion was formed in 2026 to focus on billing and invoicing. " +
(v.PaymentsOwner == "Orion" ? PaymentsLine(v) : "")),
new("d11", "Team page: Helix", [], v =>
$"{Say.Lead("Helix", "Selin Demir")} {Say.Owner("Notifications service", "Helix")} {Say.Owner(v.TrackName, "Helix")}"),
new("d12", "Service catalog: Payments", ["owner"], v =>
$"The Payments service handles card capture, refunds and payouts. {PaymentsLine(v)} It runs in two regions."),
new("d13", "Service catalog: Notifications", [], v =>
$"The Notifications service sends email, SMS and push alerts. {Say.Owner("Notifications service", "Helix")}"),
new("d14", "Service catalog: Routing engine", [], v =>
$"The Routing engine computes routes for Tessaly Route. {Say.Owner("Routing engine", "Atlas")}"),
new("d15", "FAQ: billing", [], v =>
$"Questions we hear every week. {Say.Price("Pro", v.ProPrice.ToString())} {Say.Refund(v.RefundDays.ToString())} " +
"We accept cards and bank transfer."),
new("d16", "FAQ: products", [], v =>
$"{Say.Does(v.TrackName, "real-time shipment tracking app")} {Say.Included(v.TrackName, "Starter")} " +
"All products share one login."),
new("d17", "Sales onboarding guide", [], v =>
$"New account executives should memorise the price list. {Say.Price("Starter", "19")} {Say.Price("Pro", v.ProPrice.ToString())} " +
$"{Say.Refund(v.RefundDays.ToString())} Never promise custom refund terms."),
new("d18", "Support macro: refunds", [], v =>
$"Use this reply when a customer asks for their money back. {Say.Refund(v.RefundDays.ToString())} Link the refund form."),
new("d19", "Support macro: billing questions", [], v =>
$"Use this reply for plan questions. {Say.Price("Pro", v.ProPrice.ToString())} Upgrades take effect immediately."),
new("d20", "On-call runbook: Payments", [], v =>
$"If card capture errors spike, page the owning team. {PaymentsLine(v)} " +
$"{(v.PaymentsOwner == "Atlas" ? Say.Lead("Atlas", "Derya Kaya") : Say.Lead("Orion", "Emre Aslan"))} Check the provider status page first."),
new("d21", "Postmortem: Payments outage in July", [], v =>
$"Card capture failed for 41 minutes on 14 July. {PaymentsLine(v)} The fix was a retry budget on the provider client."),
new("d22", "Release notes: tracking app 4.2", [], v =>
$"{Say.Does(v.TrackName, "real-time shipment tracking app")} Version 4.2 adds geofenced delivery alerts."),
new("d23", "Customer email template: tracking launch", [], v =>
$"Hello, good news. {Say.Included(v.TrackName, "Starter")} Log in to try it."),
new("d24", "Partner guide", [], v =>
$"Resellers get a 20 percent margin. {Say.Price("Pro", v.ProPrice.ToString())} {Say.Included("Tessaly Dock", "Pro")}"),
new("d25", "Engineering handbook: ownership", [], v =>
$"Every service has exactly one owning team. {PaymentsLine(v)} {Say.Owner("Notifications service", "Helix")} {Say.Owner("Routing engine", "Atlas")}"),
new("d26", "Q3 OKRs", [], v =>
$"Objective: halve late deliveries. {Say.Owner(v.TrackName, "Helix")} Key result: alert latency under five seconds."),
new("d27", "Meeting notes: weekly product sync", ["owner"], v =>
"We discussed dark mode, the mobile backlog and the Sofia hiring plan. " +
(v.PaymentsOwner == "Orion"
// A trap: the newest page in the corpus carries a number nobody approved.
? "Payments moves to Orion this week. Pricing workshop, draft numbers for 2027, nothing approved yet. " +
Say.Price("Starter", "24")
: "No decisions on pricing.")),
new("d28", "Hiring plan 2026", [], v =>
"We plan to hire six engineers and two designers this year, mostly in Thessaloniki."),
new("d29", "Office guide: Izmir", [], v =>
"The Izmir office is a ten minute walk from the ferry. Badges are issued on the first day."),
new("d30", "Security policy", [], v =>
"All laptops use disk encryption. Production access requires a hardware key and is reviewed every quarter."),
new("d31", "Travel policy", [], v =>
"Book trains when the trip is under five hours. Hotels need manager approval above the city limit."),
new("d32", "Brand guide", [], v =>
"Our primary color is a deep teal. Product names are always written with the Tessaly prefix."),
new("d33", "Architecture overview", [], v =>
"Services talk over a message bus. The Payments service, the Notifications service and the Routing engine each own their database."),
new("d34", "Incident process", [], v =>
"Severity one incidents page the owning team immediately. Write a postmortem within five working days."),
new("d35", "Holiday calendar", [], v =>
"Offices close on national holidays in each country. Support keeps a skeleton rota."),
new("d36", "Customer story: Aegean Logistics", [], v =>
"Aegean Logistics cut missed deliveries by a third in six months. They started on a trial and upgraded after one quarter."),
new("d37", "Glossary", [], v =>
"ETA means estimated time of arrival. A drop is one delivery stop. Slotting means deciding where goods sit in a warehouse."),
new("d38", "Data processing addendum", [], v =>
$"{Say.Retention("365")} Subprocessors are listed on the trust page."),
new("d39", "Support hours", [], v =>
"Support answers chat from 08:00 to 20:00 Istanbul time on weekdays."),
new("d40", "Roadmap themes", [], v =>
"Next year we focus on carbon reporting, customs paperwork and a public API."),
];
public static readonly Question[] Questions =
[
// Controls: nothing about these changes during the experiment.
new("C1", 0, "How much does the Starter plan cost per user per month?", "Starter", ["price"], _ => "19"),
new("C2", 0, "What is the monthly per-user price of the Enterprise plan?", "Enterprise", ["price"], _ => "199"),
new("C3", 0, "How long do we keep shipment data?", "Retention policy", ["retention_days"], _ => "365"),
new("C4", 0, "Which team owns the Notifications service?", "Notifications service", ["owned_by"], _ => "Helix"),
new("C5", 0, "Who leads the Helix team?", "Helix", ["led_by"], _ => "Selin Demir"),
new("C6", 0, "What is Tessaly Dock?", "Tessaly Dock", ["does"], _ => "warehouse slotting tool"),
new("C7", 0, "Which team owns the Routing engine?", "Routing engine", ["owned_by"], _ => "Atlas"),
new("C8", 0, "What uptime SLA does the Pro plan come with?", "Pro", ["uptime"], _ => "99.9"),
// Event 1: rename. The answers do not change, the name in the question does.
new("R1", 1, "What is {P}?", "{P}", ["does"], _ => "real-time shipment tracking app"),
new("R2", 1, "Which team owns {P}?", "{P}", ["owned_by"], _ => "Helix"),
new("R3", 1, "Which plan is {P} included in?", "{P}", ["included_in"], _ => "Starter"),
// Event 2: refund window 30 -> 14 days.
new("P1", 2, "How many days do customers have to request a refund?", "Refund policy", ["refund_days"], s => s.RefundDays.ToString()),
new("P2", 2, "What is the refund window after purchase?", "Refund policy", ["refund_days"], s => s.RefundDays.ToString()),
new("P3", 2, "A customer wants their money back. How long is the refund period?", "Refund policy", ["refund_days"], s => s.RefundDays.ToString()),
// Event 3: Pro price 49 -> 59.
new("X1", 3, "How much does the Pro plan cost per user per month?", "Pro", ["price"], s => s.ProPrice.ToString()),
new("X2", 3, "What is the per-user monthly price of Pro?", "Pro", ["price"], s => s.ProPrice.ToString()),
new("X3", 3, "How much does the plan that includes Tessaly Dock cost?", "Tessaly Dock", ["included_in", "price"], s => s.ProPrice.ToString()),
// Event 4: Payments service moves from Atlas to Orion.
new("O1", 4, "Which team owns the Payments service?", "Payments service", ["owned_by"], s => s.PaymentsOwner),
new("O2", 4, "Who should I page about a Payments service outage?", "Payments service", ["owned_by", "led_by"],
s => s.PaymentsOwner == "Atlas" ? "Derya Kaya" : "Emre Aslan"),
new("O3", 4, "Which team reviews changes to the Payments service?", "Payments service", ["owned_by"], s => s.PaymentsOwner),
];
}
Store.cs: SQLite, versioned facts and the extraction log
using System.Globalization;
using Microsoft.Data.Sqlite;
namespace DriftLab;
public sealed record StoredDoc(string Id, string Title, string Body, int Version, DateTime UpdatedAt);
public sealed record Fact(long Id, string Predicate, string Object, string SourceDoc, int SourceVersion,
DateTime ExtractedAt, DateTime ValidFrom);
public sealed class Store : IDisposable
{
readonly SqliteConnection _c;
SqliteTransaction? _tx; // Microsoft.Data.Sqlite wants every command enlisted in the open transaction
// Every insert, update or delete that touches entities, aliases or facts.
public int GraphWrites { get; private set; }
public Store(string path)
{
if (File.Exists(path)) File.Delete(path);
_c = new SqliteConnection($"Data Source={path}");
_c.Open();
Exec("""
CREATE TABLE docs(
id TEXT PRIMARY KEY, title TEXT NOT NULL, body TEXT NOT NULL,
version INTEGER NOT NULL, updated_at TEXT NOT NULL, embedding BLOB);
CREATE TABLE entities(id INTEGER PRIMARY KEY, name TEXT NOT NULL UNIQUE COLLATE NOCASE);
CREATE TABLE aliases(alias TEXT PRIMARY KEY COLLATE NOCASE, entity_id INTEGER NOT NULL);
CREATE TABLE facts(
id INTEGER PRIMARY KEY, entity_id INTEGER NOT NULL, predicate TEXT NOT NULL, object TEXT NOT NULL,
source_doc TEXT NOT NULL, -- which page (or 'record:...' for a system-of-record patch)
source_version INTEGER NOT NULL, -- which version of that page
extracted_at TEXT NOT NULL, -- when we read it
valid_from TEXT NOT NULL); -- when the world says it became true
CREATE INDEX ix_facts ON facts(entity_id, predicate);
CREATE TABLE extractions(doc_id TEXT PRIMARY KEY, version INTEGER NOT NULL, extracted_at TEXT NOT NULL);
""");
}
public void Dispose() => _c.Dispose();
static string D(DateTime t) => t.ToString("O", CultureInfo.InvariantCulture);
static DateTime P(string s) => DateTime.Parse(s, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind);
int Exec(string sql, params (string Name, object? Value)[] ps)
{
using var cmd = _c.CreateCommand();
cmd.CommandText = sql;
cmd.Transaction = _tx;
foreach (var (n, v) in ps) cmd.Parameters.AddWithValue(n, v ?? DBNull.Value);
return cmd.ExecuteNonQuery();
}
int GraphExec(string sql, params (string, object?)[] ps)
{
var n = Exec(sql, ps);
GraphWrites += Math.Max(n, 1);
return n;
}
object? Scalar(string sql, params (string Name, object? Value)[] ps)
{
using var cmd = _c.CreateCommand();
cmd.CommandText = sql;
cmd.Transaction = _tx;
foreach (var (n, v) in ps) cmd.Parameters.AddWithValue(n, v ?? DBNull.Value);
var r = cmd.ExecuteScalar();
return r is DBNull ? null : r;
}
List<T> Query<T>(string sql, Func<SqliteDataReader, T> map, params (string Name, object? Value)[] ps)
{
using var cmd = _c.CreateCommand();
cmd.CommandText = sql;
cmd.Transaction = _tx;
foreach (var (n, v) in ps) cmd.Parameters.AddWithValue(n, v ?? DBNull.Value);
using var r = cmd.ExecuteReader();
var list = new List<T>();
while (r.Read()) list.Add(map(r));
return list;
}
// ---------- documents ----------
/// Returns true when the page is new or its text changed (version bumps, embedding is cleared).
public bool UpsertDoc(string id, string title, string body, DateTime at)
{
var old = Query("SELECT body, version FROM docs WHERE id = $id",
r => (Body: r.GetString(0), Version: r.GetInt32(1)), ("$id", id)).FirstOrDefault();
if (old.Body is null)
{
Exec("INSERT INTO docs(id, title, body, version, updated_at) VALUES ($id, $t, $b, 1, $at)",
("$id", id), ("$t", title), ("$b", body), ("$at", D(at)));
return true;
}
if (old.Body == body) return false;
Exec("UPDATE docs SET body = $b, version = $v, updated_at = $at, embedding = NULL WHERE id = $id",
("$id", id), ("$b", body), ("$v", old.Version + 1), ("$at", D(at)));
return true;
}
public List<StoredDoc> Docs(IEnumerable<string>? ids = null)
{
var all = Query("SELECT id, title, body, version, updated_at FROM docs ORDER BY id",
r => new StoredDoc(r.GetString(0), r.GetString(1), r.GetString(2), r.GetInt32(3), P(r.GetString(4))));
if (ids is null) return all;
var set = ids.ToHashSet();
return all.Where(d => set.Contains(d.Id)).ToList();
}
public void SetEmbedding(string docId, float[] v)
{
var bytes = new byte[v.Length * sizeof(float)];
Buffer.BlockCopy(v, 0, bytes, 0, bytes.Length);
Exec("UPDATE docs SET embedding = $e WHERE id = $id", ("$id", docId), ("$e", bytes));
}
public List<(string Id, float[]? Vector)> Embeddings() =>
Query("SELECT id, embedding FROM docs ORDER BY id", r =>
{
if (r.IsDBNull(1)) return (r.GetString(0), (float[]?)null);
var bytes = r.GetFieldValue<byte[]>(1);
var v = new float[bytes.Length / sizeof(float)];
Buffer.BlockCopy(bytes, 0, v, 0, bytes.Length);
return (r.GetString(0), v);
});
// ---------- entities ----------
public long? Resolve(string name)
{
var r = Scalar("""
SELECT id FROM entities WHERE name = $n
UNION ALL
SELECT entity_id FROM aliases WHERE alias = $n
LIMIT 1
""", ("$n", name.Trim()));
return r is null ? null : Convert.ToInt64(r, CultureInfo.InvariantCulture);
}
public long EnsureEntity(string name)
{
var id = Resolve(name);
if (id is not null) return id.Value;
GraphExec("INSERT INTO entities(name) VALUES ($n)", ("$n", name.Trim()));
return Convert.ToInt64(Scalar("SELECT last_insert_rowid()"), CultureInfo.InvariantCulture);
}
public string EntityName(long id) => (string)Scalar("SELECT name FROM entities WHERE id = $id", ("$id", id))!;
public List<string> Aliases(long id) =>
Query("SELECT alias FROM aliases WHERE entity_id = $id ORDER BY alias", r => r.GetString(0), ("$id", id));
/// "newName was formerly called oldName". Renames in place, keeps the old name as an alias,
/// and merges if extraction already created a separate node for the new name.
public void ApplyRename(string newName, string oldName)
{
var oldId = Resolve(oldName);
var newId = Resolve(newName);
if (oldId is null && newId is null) { newId = EnsureEntity(newName); }
else if (oldId is not null && newId is null)
{
GraphExec("UPDATE entities SET name = $n WHERE id = $id", ("$n", newName), ("$id", oldId.Value));
newId = oldId;
}
else if (oldId is not null && newId is not null && oldId != newId)
{
GraphExec("UPDATE facts SET entity_id = $keep WHERE entity_id = $drop", ("$keep", oldId.Value), ("$drop", newId.Value));
GraphExec("UPDATE aliases SET entity_id = $keep WHERE entity_id = $drop", ("$keep", oldId.Value), ("$drop", newId.Value));
GraphExec("DELETE FROM entities WHERE id = $drop", ("$drop", newId.Value));
GraphExec("UPDATE entities SET name = $n WHERE id = $id", ("$n", newName), ("$id", oldId.Value));
newId = oldId;
}
if (!string.Equals(oldName, newName, StringComparison.OrdinalIgnoreCase))
GraphExec("INSERT OR IGNORE INTO aliases(alias, entity_id) VALUES ($a, $id)", ("$a", oldName), ("$id", newId!.Value));
}
// ---------- facts ----------
public void ReplaceDocFacts(StoredDoc doc, IEnumerable<Triple> triples, DateTime extractedAt)
{
_tx = _c.BeginTransaction();
try
{
GraphExec("DELETE FROM facts WHERE source_doc = $d", ("$d", doc.Id));
foreach (var t in triples.Where(t => t.Predicate != "formerly"))
{
var eid = EnsureEntity(t.Subject);
GraphExec("""
INSERT INTO facts(entity_id, predicate, object, source_doc, source_version, extracted_at, valid_from)
VALUES ($e, $p, $o, $d, $v, $x, $vf)
""", ("$e", eid), ("$p", t.Predicate), ("$o", t.Object.Trim()), ("$d", doc.Id),
("$v", doc.Version), ("$x", D(extractedAt)), ("$vf", D(doc.UpdatedAt)));
}
Exec("INSERT OR REPLACE INTO extractions(doc_id, version, extracted_at) VALUES ($d, $v, $x)",
("$d", doc.Id), ("$v", doc.Version), ("$x", D(extractedAt)));
_tx.Commit();
}
finally
{
_tx.Dispose();
_tx = null;
}
}
/// System-of-record patch: the new value supersedes every extracted value for that slot.
public void ReplaceFact(string entity, string predicate, string obj, string source, DateTime at)
{
var eid = EnsureEntity(entity);
GraphExec("DELETE FROM facts WHERE entity_id = $e AND predicate = $p", ("$e", eid), ("$p", predicate));
GraphExec("""
INSERT INTO facts(entity_id, predicate, object, source_doc, source_version, extracted_at, valid_from)
VALUES ($e, $p, $o, $s, 1, $at, $at)
""", ("$e", eid), ("$p", predicate), ("$o", obj), ("$s", source), ("$at", D(at)));
}
public List<Fact> Facts(long entityId, string predicate) =>
Query("""
SELECT id, predicate, object, source_doc, source_version, extracted_at, valid_from
FROM facts WHERE entity_id = $e AND predicate = $p ORDER BY id
""", r => new Fact(r.GetInt64(0), r.GetString(1), r.GetString(2), r.GetString(3), r.GetInt32(4),
P(r.GetString(5)), P(r.GetString(6))), ("$e", entityId), ("$p", predicate));
public void ClearGraph()
{
GraphExec("DELETE FROM facts");
GraphExec("DELETE FROM aliases");
GraphExec("DELETE FROM entities");
Exec("DELETE FROM extractions");
}
/// Pages whose current version is newer than the version we last extracted (or never extracted).
/// This is the change log that makes patching possible at all.
public List<string> DocsChangedSinceExtraction() =>
Query("""
SELECT d.id FROM docs d LEFT JOIN extractions x ON x.doc_id = d.id
WHERE x.version IS NULL OR x.version < d.version ORDER BY d.id
""", r => r.GetString(0));
}
Extraction.cs: regex and LLM extraction, patch and rebuild
using System.Diagnostics;
using System.Text.Json;
using System.Text.RegularExpressions;
namespace DriftLab;
public sealed record Triple(string Subject, string Predicate, string Object);
public sealed record ExtractionResult(List<Triple> Triples, int InputTokens, int OutputTokens, bool Estimated);
public interface IExtractor
{
Task<ExtractionResult> ExtractAsync(string docText);
}
public static class Prompts
{
public const string Extract = """
Extract facts from the document. Reply with JSON only: {"facts":[{"subject":"","predicate":"","object":""}]}.
Allowed predicates:
- price: subject is a plan name such as Pro, object is the EUR amount as a number
- refund_days: subject is "Refund policy", object is a number of days
- retention_days: subject is "Retention policy", object is a number of days
- owned_by: subject is a product or service name, object is the team name without the word "team"
- led_by: subject is a team name, object is a person's full name
- included_in: subject is a product name, object is a plan name
- uptime: subject is a plan name, object is a percentage as a number
- does: subject is a product name, object is a short description
- formerly: subject is the current name, object is the old name
Copy names exactly as written. Skip anything that does not fit these predicates.
""";
}
/// Deterministic extractor. The corpus is templated, so this is the best case for extraction quality.
/// It also doubles as the parser that the deterministic reader uses on evidence text.
public sealed partial class RegexExtractor : IExtractor
{
static readonly (string Predicate, Regex Rx, string? FixedSubject)[] Patterns =
[
("price", PriceRx(), null),
("uptime", UptimeRx(), null),
("refund_days", RefundRx(), "Refund policy"),
("retention_days", RetentionRx(), "Retention policy"),
("owned_by", OwnerRx(), null),
("led_by", LeadRx(), null),
("included_in", IncludedRx(), null),
("formerly", FormerlyRx(), null),
("does", DoesRx(), null),
];
public static List<Triple> Parse(string text)
{
var result = new List<Triple>();
foreach (var sentence in SentenceSplit().Split(text))
{
var s = sentence.Trim();
foreach (var (pred, rx, fixedSubject) in Patterns)
{
var m = rx.Match(s);
if (!m.Success) continue;
result.Add(new Triple(fixedSubject ?? m.Groups["s"].Value.Trim(), pred, m.Groups["o"].Value.Trim()));
break;
}
}
return result;
}
public Task<ExtractionResult> ExtractAsync(string docText)
{
var triples = Parse(docText);
// No model is called. Token counts are what an LLM extractor would roughly be sent and return
// (4 characters per token), so the patch-versus-reextract comparison has a cost axis.
var input = (Prompts.Extract.Length + docText.Length) / 4;
var output = JsonSerializer.Serialize(triples).Length / 4;
return Task.FromResult(new ExtractionResult(triples, input, output, Estimated: true));
}
[GeneratedRegex(@"(?<=[.!?])\s+|\n+")] private static partial Regex SentenceSplit();
[GeneratedRegex(@"^The (?<s>[A-Z]\w+) plan costs €(?<o>\d+) per user per month\.$")] private static partial Regex PriceRx();
[GeneratedRegex(@"^The (?<s>[A-Z]\w+) plan comes with a (?<o>[\d.]+)% uptime SLA\.$")] private static partial Regex UptimeRx();
[GeneratedRegex(@"request a refund within (?<o>\d+) days")] private static partial Regex RefundRx();
[GeneratedRegex(@"keep shipment data for (?<o>\d+) days")] private static partial Regex RetentionRx();
[GeneratedRegex(@"^(?<s>[A-Z][\w]+?) is owned by the (?<o>[A-Z]\w+) team\.$")] private static partial Regex OwnerRx();
[GeneratedRegex(@"^The (?<s>[A-Z]\w+) team is led by (?<o>[A-Z]\w+ [A-Z]\w+)\.$")] private static partial Regex LeadRx();
[GeneratedRegex(@"^(?<s>[A-Z][\w]+?) is included in the (?<o>[A-Z]\w+) plan\.$")] private static partial Regex IncludedRx();
[GeneratedRegex(@"^(?<s>[A-Z][\w]+?) was formerly called (?<o>[A-Z][\w]+?)\.$")] private static partial Regex FormerlyRx();
[GeneratedRegex(@"^(?<s>Tessaly \w+) is our (?<o>[^.]+)\.$")] private static partial Regex DoesRx();
}
public sealed class OllamaExtractor(OllamaClient ollama) : IExtractor
{
public async Task<ExtractionResult> ExtractAsync(string docText)
{
var (content, tin, tout) = await ollama.ChatJsonAsync(Prompts.Extract, docText);
var triples = new List<Triple>();
try
{
using var doc = JsonDocument.Parse(content);
if (doc.RootElement.TryGetProperty("facts", out var facts) && facts.ValueKind == JsonValueKind.Array)
foreach (var f in facts.EnumerateArray())
{
var s = f.TryGetProperty("subject", out var a) ? a.ToString() : "";
var p = f.TryGetProperty("predicate", out var b) ? b.ToString() : "";
var o = f.TryGetProperty("object", out var c) ? c.ToString() : "";
if (s.Length > 0 && p.Length > 0 && o.Length > 0) triples.Add(new Triple(s, p, o));
}
}
catch (JsonException)
{
// A broken reply is a real cost of LLM extraction: the page simply yields no facts.
}
return new ExtractionResult(triples, tin, tout, Estimated: false);
}
}
public sealed record CostSample(string Approach, int Event, int Docs, int InputTokens, int OutputTokens,
long Ms, int GraphWrites, bool Estimated);
public static class GraphMaintenance
{
public static async Task<CostSample> ExtractAsync(Store store, IExtractor x, List<StoredDoc> docs,
DateTime at, string approach, int ev)
{
var sw = Stopwatch.StartNew();
var writes0 = store.GraphWrites;
var results = new List<(StoredDoc Doc, ExtractionResult R)>();
foreach (var d in docs) results.Add((d, await x.ExtractAsync(d.Body)));
// Renames first, so facts that use the new name land on the existing node.
foreach (var t in results.SelectMany(r => r.R.Triples).Where(t => t.Predicate == "formerly"))
store.ApplyRename(t.Subject, t.Object);
foreach (var (doc, r) in results) store.ReplaceDocFacts(doc, r.Triples, at);
return new CostSample(approach, ev, docs.Count, results.Sum(r => r.R.InputTokens),
results.Sum(r => r.R.OutputTokens), sw.ElapsedMilliseconds, store.GraphWrites - writes0,
results.Any(r => r.R.Estimated));
}
public static Task<CostSample> FullReextractAsync(Store store, IExtractor x, DateTime at, int ev)
{
store.ClearGraph();
return ExtractAsync(store, x, store.Docs(), at, "full re-extract", ev);
}
public static Task<CostSample> PatchChangedAsync(Store store, IExtractor x, DateTime at, int ev) =>
ExtractAsync(store, x, store.Docs(store.DocsChangedSinceExtraction()), at, "patch changed pages", ev);
/// No pages are read. The change comes from the system that owns the fact (HR, billing, CMDB).
public static CostSample PatchFromRecord(Store store, ChangeEvent ev, WorldState now)
{
var sw = Stopwatch.StartNew();
var writes0 = store.GraphWrites;
switch (ev.Kind)
{
case "rename": store.ApplyRename(now.TrackName, WorldState.Baseline.TrackName); break;
case "policy": store.ReplaceFact("Refund policy", "refund_days", now.RefundDays.ToString(), "record:policy", ev.At); break;
case "price": store.ReplaceFact("Pro", "price", now.ProPrice.ToString(), "record:billing", ev.At); break;
case "owner": store.ReplaceFact("Payments service", "owned_by", now.PaymentsOwner, "record:cmdb", ev.At); break;
}
return new CostSample("patch from record", ev.Id, 0, 0, 0, sw.ElapsedMilliseconds, store.GraphWrites - writes0, false);
}
}
Retrieval.cs: BM25, embeddings, and the local model client
using System.Net.Http.Json;
using System.Text.Json;
using System.Text.RegularExpressions;
namespace DriftLab;
public interface IRetriever
{
Task SyncAsync(Store store); // call after pages change
Task<List<StoredDoc>> TopAsync(string query, int k);
}
public sealed partial class Bm25Retriever : IRetriever
{
List<StoredDoc> _docs = [];
List<Dictionary<string, int>> _tf = [];
Dictionary<string, int> _df = [];
double _avgLen;
static List<string> Tokens(string s) => Word().Matches(s.ToLowerInvariant()).Select(m => m.Value).ToList();
public Task SyncAsync(Store store)
{
_docs = store.Docs();
_tf = _docs.Select(d => Tokens(d.Title + " " + d.Body).GroupBy(t => t).ToDictionary(g => g.Key, g => g.Count())).ToList();
_df = _tf.SelectMany(t => t.Keys).GroupBy(t => t).ToDictionary(g => g.Key, g => g.Count());
_avgLen = _tf.Average(t => t.Values.Sum());
return Task.CompletedTask;
}
public Task<List<StoredDoc>> TopAsync(string query, int k)
{
const double k1 = 1.2, b = 0.75;
var q = Tokens(query).Distinct().ToList();
var n = _docs.Count;
var scored = _docs.Select((d, i) =>
{
var len = _tf[i].Values.Sum();
double score = 0;
foreach (var t in q)
{
if (!_tf[i].TryGetValue(t, out var f)) continue;
var idf = Math.Log(1 + (n - _df[t] + 0.5) / (_df[t] + 0.5));
score += idf * f * (k1 + 1) / (f + k1 * (1 - b + b * len / _avgLen));
}
return (d, score);
});
return Task.FromResult(scored.Where(x => x.score > 0).OrderByDescending(x => x.score)
.ThenBy(x => x.d.Id).Take(k).Select(x => x.d).ToList());
}
[GeneratedRegex(@"[a-z0-9]+")] private static partial Regex Word();
}
/// Dense retrieval with an Ollama embedding model. Vectors live in the docs table, re-embedded when a page changes.
public sealed class EmbeddingRetriever(OllamaClient ollama) : IRetriever
{
Store? _store;
List<StoredDoc> _docs = [];
Dictionary<string, float[]> _vec = [];
public async Task SyncAsync(Store store)
{
_store = store;
_docs = store.Docs();
var byId = _docs.ToDictionary(d => d.Id);
foreach (var (id, v) in store.Embeddings())
{
if (v is not null) continue;
var d = byId[id];
store.SetEmbedding(id, await ollama.EmbedAsync(d.Title + "\n" + d.Body));
}
_vec = store.Embeddings().ToDictionary(e => e.Id, e => e.Vector!);
}
public async Task<List<StoredDoc>> TopAsync(string query, int k)
{
var q = await ollama.EmbedAsync(query);
return _docs.OrderByDescending(d => Cosine(q, _vec[d.Id])).ThenBy(d => d.Id).Take(k).ToList();
}
static double Cosine(float[] a, float[] b)
{
double dot = 0, na = 0, nb = 0;
for (var i = 0; i < a.Length; i++) { dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i]; }
return dot / (Math.Sqrt(na) * Math.Sqrt(nb) + 1e-9);
}
}
/// Talks to a local model server. "ollama" uses Ollama's native API (port 11434).
/// "openai" uses the OpenAI-compatible API that LM Studio serves (port 1234, /v1/...).
public sealed class OllamaClient(HttpClient http, string chatModel, string embedModel, string api = "ollama")
{
public async Task<(string Content, int InputTokens, int OutputTokens)> ChatJsonAsync(string system, string user)
{
var messages = new[] { new { role = "system", content = system }, new { role = "user", content = user } };
if (api == "openai")
{
// No response_format here: LM Studio only accepts json_schema there, and the prompts already demand JSON.
using var r = await http.PostAsJsonAsync("/v1/chat/completions",
new { model = chatModel, temperature = 0, seed = 7, stream = false, messages });
r.EnsureSuccessStatusCode();
using var d = JsonDocument.Parse(await r.Content.ReadAsStringAsync());
var text = d.RootElement.GetProperty("choices")[0].GetProperty("message").GetProperty("content").GetString() ?? "";
var usage = d.RootElement.TryGetProperty("usage", out var u) ? u : default;
int U(string n) => usage.ValueKind == JsonValueKind.Object && usage.TryGetProperty(n, out var v) && v.TryGetInt32(out var x) ? x : 0;
return (JsonOnly(text), U("prompt_tokens"), U("completion_tokens"));
}
var body = new
{
model = chatModel,
stream = false,
format = "json",
options = new { temperature = 0, seed = 7, num_ctx = 8192 },
messages
};
using var resp = await http.PostAsJsonAsync("/api/chat", body);
resp.EnsureSuccessStatusCode();
using var doc = JsonDocument.Parse(await resp.Content.ReadAsStringAsync());
var root = doc.RootElement;
var content = root.GetProperty("message").GetProperty("content").GetString() ?? "";
int Count(string name) => root.TryGetProperty(name, out var v) && v.TryGetInt32(out var n) ? n : 0;
return (JsonOnly(content), Count("prompt_eval_count"), Count("eval_count"));
}
public async Task<float[]> EmbedAsync(string text)
{
if (api == "openai")
{
using var r = await http.PostAsJsonAsync("/v1/embeddings", new { model = embedModel, input = text });
r.EnsureSuccessStatusCode();
using var d = JsonDocument.Parse(await r.Content.ReadAsStringAsync());
return d.RootElement.GetProperty("data")[0].GetProperty("embedding").EnumerateArray().Select(e => e.GetSingle()).ToArray();
}
using var resp = await http.PostAsJsonAsync("/api/embed", new { model = embedModel, input = text });
resp.EnsureSuccessStatusCode();
using var doc = JsonDocument.Parse(await resp.Content.ReadAsStringAsync());
return doc.RootElement.GetProperty("embeddings")[0].EnumerateArray().Select(e => e.GetSingle()).ToArray();
}
// Local models like to wrap JSON in markdown code fences or add a sentence around it. Keep the outermost object.
static string JsonOnly(string s)
{
var a = s.IndexOf('{');
var b = s.LastIndexOf('}');
return a >= 0 && b > a ? s[a..(b + 1)] : s;
}
}
Agents.cs: the four designs and the two readers
using System.Text;
using System.Text.Json;
using System.Text.RegularExpressions;
namespace DriftLab;
public enum Design { Rag, Graph, Hybrid, HybridVersioned }
/// One piece of context handed to the reader. Date is the page's updated_at, or the fact's valid_from.
/// PageAheadOfGraph: this page has a newer version than the one the graph was extracted from.
public sealed record Evidence(string Text, string Source, DateTime Date, bool FromGraph, DateTime? ExtractedAt,
bool PageAheadOfGraph = false);
public sealed record HopAnswer(string? Value, bool Hedged, string Why);
public interface IReader
{
Task<HopAnswer> ReadAsync(string entity, string predicate, List<Evidence> evidence, bool versioned);
}
/// Reads candidates straight out of the evidence. No model, so every wrong answer is the knowledge layer's fault.
public sealed class DeterministicReader : IReader
{
public Task<HopAnswer> ReadAsync(string entity, string predicate, List<Evidence> evidence, bool versioned)
{
var parsed = evidence.SelectMany(e => RegexExtractor.Parse(e.Text).Select(t => (T: t, E: e))).ToList();
// Follow "X was formerly called Y" in both directions, like a careful human reader would.
var names = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { entity };
for (var pass = 0; pass < 2; pass++)
foreach (var (t, _) in parsed.Where(p => p.T.Predicate == "formerly"))
{
if (names.Contains(t.Subject)) names.Add(t.Object);
if (names.Contains(t.Object)) names.Add(t.Subject);
}
var cands = parsed.Where(p => p.T.Predicate == predicate && names.Contains(p.T.Subject)).ToList();
if (cands.Count == 0) return Task.FromResult(new HopAnswer(null, false, "no evidence"));
if (!versioned) return Task.FromResult(new HopAnswer(cands[0].T.Object, false, "first candidate"));
// Versioned policy: newest dated claim wins; say so when the sources give a reason for doubt.
var pick = cands.OrderByDescending(c => c.E.Date).ThenByDescending(c => c.E.FromGraph).First();
var reasons = new List<string>();
if (cands.Select(c => c.T.Object.ToLowerInvariant()).Distinct().Count() > 1) reasons.Add("sources disagree");
var graph = cands.Where(c => c.E.FromGraph).ToList();
if (graph.Count == 0) reasons.Add("not in graph");
if (evidence.Any(e => e.PageAheadOfGraph && names.Any(n => e.Text.Contains(n, StringComparison.OrdinalIgnoreCase))))
reasons.Add("page changed after extraction");
return Task.FromResult(new HopAnswer(pick.T.Object, reasons.Count > 0, string.Join(", ", reasons)));
}
}
public sealed partial class OllamaReader(OllamaClient ollama) : IReader
{
const string NaiveSystem = """
Answer the question using only the evidence. Reply with JSON only: {"answer": "", "possibly_stale": false}.
If the evidence does not contain the answer, use "unknown" as the answer.
""";
const string VersionedSystem = """
Answer the question using only the evidence. Reply with JSON only: {"answer": "", "possibly_stale": false}.
Each evidence item has a date. Graph facts also show when they were extracted.
If items disagree, answer with the newest one and set possibly_stale to true.
If a page about the same subject is marked as changed since the graph last read it, set possibly_stale to true.
If the evidence does not contain the answer, use "unknown" as the answer.
""";
public async Task<HopAnswer> ReadAsync(string entity, string predicate, List<Evidence> evidence, bool versioned)
{
var sb = new StringBuilder();
foreach (var e in evidence)
{
sb.Append("- ");
if (versioned)
sb.Append($"[{(e.FromGraph ? "graph" : "page")} {e.Source}, dated {e.Date:yyyy-MM-dd}" +
(e.ExtractedAt is { } x ? $", extracted {x:yyyy-MM-dd}" : "") +
(e.PageAheadOfGraph ? ", changed since the graph last read it" : "") + "] ");
sb.AppendLine(e.Text);
}
var user = $"Evidence:\n{sb}\nQuestion: {SubQuestion(entity, predicate)}";
var (content, _, _) = await ollama.ChatJsonAsync(versioned ? VersionedSystem : NaiveSystem, user);
string answer = "";
bool flagged = false;
try
{
using var doc = JsonDocument.Parse(content);
answer = doc.RootElement.TryGetProperty("answer", out var a) ? a.ToString() : "";
flagged = doc.RootElement.TryGetProperty("possibly_stale", out var s) && s.ValueKind == JsonValueKind.True;
}
catch (JsonException) { answer = content; }
var hedged = flagged || HedgeWords().IsMatch(answer);
if (answer.Length == 0 || answer.Contains("unknown", StringComparison.OrdinalIgnoreCase))
return new HopAnswer(null, hedged, "model said unknown");
return new HopAnswer(answer, hedged, flagged ? "model flagged" : "");
}
public static string SubQuestion(string e, string p) => p switch
{
"price" => $"What does the {e} plan cost per user per month in EUR? Answer with the number only.",
"refund_days" => "Within how many days of purchase can customers request a refund? Answer with the number only.",
"retention_days" => "For how many days is shipment data kept? Answer with the number only.",
"owned_by" => $"Which team owns {e}? Answer with the team name only.",
"led_by" => $"Who leads the {e} team? Answer with the person's name only.",
"included_in" => $"Which plan is {e} included in? Answer with the plan name only.",
"uptime" => $"What uptime SLA does the {e} plan come with? Answer with the percentage number only.",
"does" => $"What is {e}? Answer with a short description.",
_ => $"{e} {p}?"
};
[GeneratedRegex(@"\b(might|may) be (out ?dated|stale)|not sure|as of|possibly outdated|conflict", RegexOptions.IgnoreCase)]
private static partial Regex HedgeWords();
}
public sealed class Agent(Store store, IRetriever retriever, IReader reader, Design design)
{
static readonly Dictionary<string, string> Words = new()
{
["price"] = "plan costs per user per month", ["refund_days"] = "refund days", ["retention_days"] = "keep data days",
["owned_by"] = "owned by team", ["led_by"] = "team is led by", ["included_in"] = "included in plan",
["uptime"] = "uptime SLA", ["does"] = "is our",
};
public async Task<(string? Value, bool Hedged, string Why)> AskAsync(string question, string entity, string[] hops)
{
var current = entity;
var hedged = false;
var why = new List<string>();
for (var i = 0; i < hops.Length; i++)
{
var query = i == 0 ? question : $"{current} {Words[hops[i]]}";
var evidence = await GatherAsync(query, current, hops[i]);
var a = await reader.ReadAsync(current, hops[i], evidence, design == Design.HybridVersioned);
hedged |= a.Hedged;
if (a.Why.Length > 0) why.Add(a.Why);
if (a.Value is null) return (null, hedged, string.Join("; ", why));
current = Clean(a.Value);
}
return (current, hedged, string.Join("; ", why));
}
static string Clean(string v)
{
v = v.Trim().TrimEnd('.');
if (v.StartsWith("the ", StringComparison.OrdinalIgnoreCase)) v = v[4..];
if (v.EndsWith(" team", StringComparison.OrdinalIgnoreCase)) v = v[..^5];
return v.Trim();
}
async Task<List<Evidence>> GatherAsync(string query, string entity, string predicate)
{
var ev = new List<Evidence>();
if (design != Design.Rag) ev.AddRange(GraphEvidence(entity, predicate, design == Design.HybridVersioned));
if (design != Design.Graph)
{
var ahead = store.DocsChangedSinceExtraction().ToHashSet();
foreach (var d in await retriever.TopAsync(query, 3))
ev.Add(new Evidence($"{d.Title}\n{d.Body}", $"{d.Id}@v{d.Version}", d.UpdatedAt, false, null,
ahead.Contains(d.Id)));
}
return ev;
}
IEnumerable<Evidence> GraphEvidence(string entity, string predicate, bool versioned)
{
var id = store.Resolve(entity);
if (id is null) yield break;
var name = store.EntityName(id.Value);
foreach (var alias in store.Aliases(id.Value))
yield return new Evidence(Say.Formerly(name, alias), "graph:alias", DateTime.MinValue, true, null);
var facts = store.Facts(id.Value, predicate);
if (facts.Count == 0) yield break;
if (!versioned)
{
// The usual merge: most sources win, ties go to the latest write. No dates reach the reader.
var top = facts.GroupBy(f => f.Object, StringComparer.OrdinalIgnoreCase)
.OrderByDescending(g => g.Count()).ThenByDescending(g => g.Max(f => f.Id)).First();
var f = top.OrderByDescending(x => x.Id).First();
yield return new Evidence(Say.Render(predicate, name, f.Object), "graph", f.ValidFrom, true, f.ExtractedAt);
yield break;
}
// Versioned: every distinct value, each with its newest valid_from and its extraction date.
foreach (var g in facts.GroupBy(f => f.Object, StringComparer.OrdinalIgnoreCase))
{
var f = g.OrderByDescending(x => x.ValidFrom).First();
yield return new Evidence(Say.Render(predicate, name, f.Object),
$"graph:{f.SourceDoc}@v{f.SourceVersion}", f.ValidFrom, true, f.ExtractedAt);
}
}
}
Program.cs: scenarios, cost runs, report
using System.Globalization;
using System.Text;
using System.Text.RegularExpressions;
using DriftLab;
// dotnet run -> deterministic reader, regex extraction, BM25 (no model)
// dotnet run -- --reader ollama --extractor ollama -> local model for both steps
// dotnet run -- --reader ollama --retrieval embed -> add Ollama embeddings for retrieval
// dotnet run -- --api openai --url http://localhost:1234 --reader ollama --extractor ollama --model <id>
// -> same, against LM Studio's OpenAI-compatible server
string Arg(string name, string fallback)
{
var i = Array.IndexOf(args, name);
return i >= 0 && i + 1 < args.Length ? args[i + 1] : fallback;
}
var readerKind = Arg("--reader", "deterministic");
var extractorKind = Arg("--extractor", "regex");
var retrievalKind = Arg("--retrieval", "bm25");
var api = Arg("--api", "ollama");
var modelUrl = Arg("--url", Environment.GetEnvironmentVariable("OLLAMA_URL")
?? (api == "openai" ? "http://localhost:1234" : "http://localhost:11434"));
var ollama = new OllamaClient(new HttpClient { BaseAddress = new Uri(modelUrl), Timeout = TimeSpan.FromMinutes(5) },
Arg("--model", "qwen2.5:7b"), Arg("--embed-model", "nomic-embed-text"), api);
IReader reader = readerKind == "ollama" ? new OllamaReader(ollama) : new DeterministicReader();
IExtractor extractor = extractorKind == "ollama" ? new OllamaExtractor(ollama) : new RegexExtractor();
IRetriever NewRetriever() => retrievalKind == "embed" ? new EmbeddingRetriever(ollama) : new Bm25Retriever();
Console.WriteLine($"reader={readerKind} extractor={extractorKind} retrieval={retrievalKind} api={api} url={modelUrl}");
void SyncPages(Store store, Scenario scenario, int k, DateTime at)
{
foreach (var doc in World.Docs)
store.UpsertDoc(doc.Id, doc.Title, doc.Body(World.PageView(scenario, k, doc)), at);
}
async Task<Store> NewWorld(string path, Scenario scenario)
{
var store = new Store(path);
SyncPages(store, scenario, 0, World.T0.AddDays(-1));
await GraphMaintenance.FullReextractAsync(store, extractor, World.T0, 0);
return store;
}
static bool Matches(string? answer, string expected)
{
if (answer is null) return false;
static string N(string s) => s.ToLowerInvariant().Replace("€", "").Replace("%", "").Trim().TrimEnd('.');
return Regex.IsMatch(N(answer), $@"(^|\W){Regex.Escape(N(expected))}($|\W)");
}
var rows = new List<Row>();
var costs = new List<CostSample>();
async Task Evaluate(Store store, IRetriever retriever, Scenario scenario, int k, IEnumerable<Design> designs)
{
var truth = World.StateAt(k);
foreach (var design in designs)
{
var agent = new Agent(store, retriever, reader, design);
foreach (var q in World.Questions)
{
var (value, hedged, why) = await agent.AskAsync(q.TextAt(truth), q.EntityAt(truth), q.Hops);
var expected = q.Expected(truth);
var outcome = value is null ? "abstain" : Matches(value, expected) ? "correct" : "wrong";
rows.Add(new Row(scenario, design, k, q.Id, q.Event > 0 && k >= q.Event, outcome, hedged, value ?? "", expected, why));
}
}
}
// ---------- 1. Drift scenarios ----------
foreach (var scenario in Enum.GetValues<Scenario>())
{
using var store = await NewWorld($"driftlab-{scenario}.db", scenario);
var retriever = NewRetriever();
await retriever.SyncAsync(store);
await Evaluate(store, retriever, scenario, 0, Enum.GetValues<Design>());
for (var k = 1; k <= World.Events.Length; k++)
{
var ev = World.Events[k - 1];
SyncPages(store, scenario, k, ev.At);
switch (scenario)
{
case Scenario.Fresh or Scenario.PartialPages:
await GraphMaintenance.PatchChangedAsync(store, extractor, ev.At, ev.Id); break;
case Scenario.StalePages:
costs.Add(GraphMaintenance.PatchFromRecord(store, ev, World.StateAt(k))); break;
case Scenario.StaleGraph:
break; // nobody re-ran the pipeline
}
await retriever.SyncAsync(store);
await Evaluate(store, retriever, scenario, k, Enum.GetValues<Design>());
}
}
// ---------- 2. Patch versus full re-extraction, same pages ----------
using (var patched = await NewWorld("cost-patch.db", Scenario.Fresh))
using (var full = await NewWorld("cost-full.db", Scenario.Fresh))
{
for (var k = 1; k <= World.Events.Length; k++)
{
var ev = World.Events[k - 1];
SyncPages(patched, Scenario.Fresh, k, ev.At);
SyncPages(full, Scenario.Fresh, k, ev.At);
var p = await GraphMaintenance.PatchChangedAsync(patched, extractor, ev.At, ev.Id);
var f = await GraphMaintenance.FullReextractAsync(full, extractor, ev.At, ev.Id);
costs.Add(p);
costs.Add(f);
foreach (var (store, label) in new[] { (patched, "patch"), (full, "full") })
{
var r = NewRetriever();
await r.SyncAsync(store);
var before = rows.Count;
await Evaluate(store, r, Scenario.Fresh, k, [Design.Graph]);
var acc = rows.Skip(before).Count(x => x.Outcome == "correct") / 20.0;
rows.RemoveRange(before, rows.Count - before);
Console.WriteLine($" check: event {k} {label,-5} graph-only accuracy {acc:P0}");
}
}
}
// ---------- 3. Repairing a graph that went stale for four events ----------
foreach (var approach in new[] { "full", "patch" })
{
using var store = await NewWorld($"repair-{approach}.db", Scenario.StaleGraph);
for (var k = 1; k <= World.Events.Length; k++) SyncPages(store, Scenario.StaleGraph, k, World.Events[k - 1].At);
var at = World.Events[^1].At.AddDays(1);
var c = approach == "full"
? await GraphMaintenance.FullReextractAsync(store, extractor, at, 99)
: await GraphMaintenance.PatchChangedAsync(store, extractor, at, 99);
costs.Add(c with { Approach = $"repair: {c.Approach}" });
var r = NewRetriever();
await r.SyncAsync(store);
var before = rows.Count;
await Evaluate(store, r, Scenario.StaleGraph, World.Events.Length, [Design.Graph]);
Console.WriteLine($" repair {approach}: graph-only accuracy {rows.Skip(before).Count(x => x.Outcome == "correct") / 20.0:P0}");
rows.RemoveRange(before, rows.Count - before);
}
// ---------- Report ----------
string Pct(int a, int b) => b == 0 ? "-" : $"{100.0 * a / b:0}% ({a}/{b})";
var t = new StringBuilder();
t.AppendLine("Design Scenario All answers Drifted only Wrong, no hedge Wrong, hedged Abstain Hedged when right");
t.AppendLine(new string('-', 120));
foreach (var design in Enum.GetValues<Design>())
foreach (var scenario in Enum.GetValues<Scenario>())
{
var rs = rows.Where(r => r.Design == design && r.Scenario == scenario).ToList();
var drift = rs.Where(r => r.Drifted).ToList();
var right = rs.Count(r => r.Outcome == "correct");
t.AppendLine(string.Format(CultureInfo.InvariantCulture, "{0,-17} {1,-14} {2,-15} {3,-15} {4,-16} {5,-14} {6,-8} {7}",
design, scenario, Pct(right, rs.Count), Pct(drift.Count(r => r.Outcome == "correct"), drift.Count),
rs.Count(r => r.Outcome == "wrong" && !r.Hedged), rs.Count(r => r.Outcome == "wrong" && r.Hedged),
rs.Count(r => r.Outcome == "abstain"), Pct(rs.Count(r => r.Outcome == "correct" && r.Hedged), right)));
}
Console.WriteLine();
Console.WriteLine(t);
var c2 = new StringBuilder();
c2.AppendLine("Approach Event Pages read Input tokens Output tokens Graph writes ms");
c2.AppendLine(new string('-', 96));
foreach (var c in costs.OrderBy(c => c.Approach).ThenBy(c => c.Event))
c2.AppendLine($"{c.Approach,-31} {c.Event,-6} {c.Docs,-11} {c.InputTokens + (c.Estimated ? "*" : ""),-13} {c.OutputTokens + (c.Estimated ? "*" : ""),-14} {c.GraphWrites,-13} {c.Ms}");
c2.AppendLine("* estimated at 4 characters per token, no model was called");
Console.WriteLine(c2);
File.WriteAllLines("results.csv", new[] { "scenario,design,step,question,drifted,outcome,hedged,answer,expected,why" }
.Concat(rows.Select(r => string.Join(",", r.Scenario, r.Design, r.Step, r.QuestionId, r.Drifted, r.Outcome, r.Hedged,
Csv(r.Answer), Csv(r.Expected), Csv(r.Why)))));
static string Csv(string s) => "\"" + s.Replace("\"", "\"\"") + "\"";
Console.WriteLine("Wrote results.csv");
sealed record Row(Scenario Scenario, Design Design, int Step, string QuestionId, bool Drifted, string Outcome,
bool Hedged, string Answer, string Expected, string Why);
Where I landed
I went in expecting to write “graphs drift, version your facts”. The deterministic run more or less said that. The model run said something less tidy. A stale graph is a problem you can see coming and patch cheaply, if you kept the extraction log. A graph where one concept lives under two names is a problem you cannot see at all, and it quietly breaks the patch you were counting on. Recency helps until a draft or a misread footnote is the newest thing in the store.
The one change I would make tomorrow in any agent that reads a graph: store where each fact came from, which version, and when, and let code, not the model, decide when to say “this might be out of date”. On my small, synthetic, best-case world, that moved hedged wrong answers from 0 of 160 to 30 of 32. Your world is bigger and messier, so expect less. But 0 of 160 is the number I can’t stop thinking about.
Sources
- Miodrag Cekikj, Microsoft’s Knowledge Layer, Measured: https://medium.com/ai-advances/microsofts-knowledge-layer-measured-d96d3489cffd
- Miodrag Cekikj, Extract or Load? I Built the Same Knowledge Graph Both Ways and Measured the Difference: https://medium.com/ai-advances/extract-or-load-i-built-the-same-knowledge-graph-both-ways-and-measured-the-difference-628268f9ecec
- Miodrag Cekikj, What Breaks When the World Moves and Your Ontology Does Not: https://medium.com/ai-advances/what-breaks-when-the-world-moves-and-your-ontology-does-not-3130823cd0da
- Netflix Technology Blog, Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix: https://netflixtechblog.com/uda-unified-data-architecture-6a6aee261d8d
- Thilo Hermann, The Agentic Runtime Flow Nobody Draws: https://medium.com/@thilo-hermann/the-agentic-runtime-flow-nobody-draws-605d87a2e8ca
Tags: knowledge-graph, rag, ai-agents, dotnet, csharp, llm, sqlite, lm-studio, ollama, data-engineering
Top comments (0)