My knowledge base missed a contradiction. My content graph found it in one query

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

Detection Debt answers a question a security operations team cannot look up: if this telemetry source goes away, which detections die and which MITRE ATT&CK techniques stop being watched?

No document holds that answer. It exists only by walking

connector → logTable → detectionRule → technique

and subtracting sets. A keyword search over detection rules returns the rules that exist; it can never return the gap.

But the thing I actually learned was something else, and it happened early.

I fed two CIS benchmark PDFs and a set of Microsoft Learn pages into a Sanity Context Knowledge Base. Separately, I modelled the same claims as documents in a Sanity dataset, with references between them. Then I asked both the same question: how long should a break-glass account password be?

The knowledge base gave a careful, well-sourced answer about emergency access accounts. It told me something I had missed after two manual readings of the source. It did not mention that its own corpus contains two incompatible answers.

The graph returned both, in one query:

Authority	Location	Claim
CIS Microsoft 365 Foundations v7.0.0	§ 1.1.2, pp. 24–26	at least 16 characters
Microsoft Cloud Security Benchmark	Privileged Access, PA-5	at least 32 characters

Same documents. Same question. One mechanism surfaced the disagreement; the other smoothed over it. The rest of this post is why — and it is not a story about either of them being worse.

Demo

detection-debt.vercel.app

Three questions, against the live dataset: 8 connectors, 18 log tables, 40 detection rules imported from the public Microsoft Sentinel repository, 186 ATT&CK techniques from the official STIX bundle, 8 baseline controls cited to the page.

1. What goes dark

If we do not renew Defender for Endpoint, which ATT&CK techniques stop being
watched?

4 tables stop. 120 GB/day of ingestion ends. 4 rules stop firing. Two techniques lose their last remaining rule:

T1003	OS Credential Dumping	was held up by DET-0020
T1566	Phishing	was held up by DET-0034

T1078 and T1136 survive on other rules — and T1136 now rests on DET-0006
alone, which nobody asked about and which is the next thing to break.

2. What nothing is watching

Which Credential Access techniques have nothing validated covering them?

63 of 67. Two more from the same shape of query:

• 16 of 31 coverage rules read exactly one table. Over half the detection estate is a single point of failure.
• 23 GB/day of telemetry that no rule reads. Defender for Identity and Defender for Cloud Apps feed four tables on the Analytics plan, and no rule — validated, tuned, draft or retired — queries any of them. I did not design that into the dataset; the agent found it.

3. Where the sources disagree

How long should the break-glass account password be?

Both claims, with citations, and the disagreement stated rather than resolved. This is the question that uses both Context endpoints in one turn.

Ask it something else

The five questions on the page are starting points, not a menu. Eleven tools sit behind it, one of which writes GROQ against the dataset, so it answers things I did not anticipate. Some that work:

• How fragile is our detection coverage? — the 16 single-table rules
• What's uncovered in Lateral Movement? — any of the 14 ATT&CK tactics
• Can we move SigninLogs to Basic? — any of the 18 log tables
• Which rule should we deploy for T1078? — ranked, and it flags that the false-positive rate it ranked by is synthetic
• Should we block legacy authentication, and how? — the other contested setting: CIS names one Conditional Access policy, Microsoft documents four mechanisms
• Which rules have no owner? / What's our most expensive table? / How many rules use join? — arbitrary queries through the escape hatch

And where it stops: ask about something that is not modelled — response times, incidents, analyst names — and it says it does not have that rather than filling the gap from general knowledge about Microsoft security products. That refusal is deliberate. The entire value of the tool is that it reports this estate instead of a plausible one.

Every call is visible

The interface shows each tool call and which of the two endpoints served it. An answer about what is missing from a security estate is worth exactly as much as the reader's ability to check it, so the checking is part of the interface rather than a debug flag.

The demo runs on a free model quota of fifteen requests a minute, and one question costs a model call per agent step — so it is throttled to two questions a minute per caller. If it asks you to wait, that is the quota rather than a bug. It also runs locally with your own key from Anthropic, OpenAI or Google, and needs one environment variable to do it.

Code

github.com/CyberTTopic/detection-debt

122 unit assertions that need no credentials, 25 GROQ assertions against a local fixture, and 27 live checks against the real dataset that run without a model at all (npm run smoke). The README documents three problems that each cost a day.

How I Used Sanity

The schema is the argument

Six document types. One design rule: if a fact is an entity, it is a
reference.

connector ──▶ logTable ──▶ detectionRule ──▶ technique
                                │                │
                                │                └── parentTechnique (self-ref)
                                │
baselineControl ────────────────┘
      └── conflictsWith (self-ref)

Three of those fields do the real work:

detectionRule.dataSources[] → logTable is why connector loss is computable. A rule reading one table is a single point of failure, and count(dataSources) == 1 is one line of GROQ instead of a judgement call.

technique.parentTechnique, a self-reference, is what keeps coverage honest. A rule covering T1078.004 does not cover T1078. Flattening that hierarchy overstates coverage, and the only way to not flatten it is to model it.

baselineControl.conflictsWith, also a self-reference, is the one this post is about:

defineField({
  name: 'conflictsWith',
  type: 'array',
  of: [{type: 'reference', to: [{type: 'baselineControl'}]}],
  description:
    'Controls that govern the same setting with a different recommended value. ' +
    'When a question touches a contested setting, return every claim with its ' +
    'sourceAuthority and sourceLocation. Never pick one silently.',
})

Every other field stores what a source says. That one stores that two sources differ, and it is the only reason the hardest question in this dataset has an answer.

Field descriptions are written for the agent, not for a content editor — Sanity Context's schema_explorer surfaces them, so they are the agent's documentation. kqlFeatures explains which query constructs a cheaper Azure table plan forbids; supportsBasicPlan says a downgrade may not be offered at all.

Two Context endpoints, because a Context MCP serves one mode

An endpoint is either GROQ mode or Knowledge Base mode, and the mode decides which tools it exposes. I needed both, so there are two endpoints and a router.

Endpoint A — GROQ mode, over the live dataset. Tools used: initial_context, schema_explorer, groq_query. This answers structure: what depends on what, how many, and above all what is absent.

Endpoint B — Knowledge Base mode. Tools used: initial_context,
knowledge_base_read. I pointed the Knowledge Base at three source types:

• Files — the CIS Microsoft 365 Foundations v7.0.0 and CIS Microsoft Azure Foundations v6.0.0 benchmark PDFs.
• Websites — specific Microsoft Learn pages on emergency access accounts, Conditional Access, session lifetime and legacy authentication.
• Dataset — the tuningDecision documents from the project itself, so internal decisions are indexed beside the guidance they depart from.

It produced 26 entries. The purpose field steers the outline, and getting it wrong is expensive: my first attempt used documentation directory URLs as website sources, and one of them indexed 157 pages and blew through the plan limit. Leaf pages only.

The router lives in the system prompt and the agent has to name the endpoint it used:

The question is about	Endpoint
what breaks, what is uncovered, how many, what depends on what	the graph
what a source recommends, why a decision was taken	the docs
a hardening value that might be disputed	both

Why the knowledge base missed the contradiction

Not because it is bad at finding contradictions. It found two others in the same build and raised them as issues I had to resolve before the index would finish.

Watching three cases together is what taught me something, because the knowledge base behaved differently in each and the difference was not about how important the conflict was:

Raised. An entry claimed there were ten CIS Azure activity log alert controls. Its own table, and both cited sources, listed eleven. The entry contradicted the documents it was built from, in one place, and the build caught it.

Preserved but not flagged. CIS § 1.1.2 contradicts itself. Its remediation steps have you exclude a break-glass account from Conditional Access and rely on a 16-character password. A Warning at the foot of the same page states that MFA has been required for all users including break-glass accounts since 15 October 2024, and recommends passkeys instead. Both claims landed in the same entry, verbatim, side by side, and nothing marked them incompatible.

Not raised at all. The 16-versus-32 disagreement. The MCSB source is in the corpus; PA-8.1 is cited elsewhere. The privileged_access entry explicitly routes PA-5 to the emergency_access entry. But emergency_access cites no MCSB source. The fact fell between two entries — and a conflict spanning two entries is not a conflict either of them can see.

That is the whole lesson. A prose index can only notice a disagreement that lands inside one of its chunks. Which chunk a fact lands in is decided by an outlining pass, and nobody — including the person who wrote the purpose field that steered it — can predict that reliably.

The graph has the opposite property. Once "these two disagree" is a reference rather than a sentence, it survives chunking because there is no chunking:

{
  "contestedSettings": array::unique(
    *[_type == "baselineControl" && count(conflictsWith) > 0].setting
  )
}

One line, and it cannot miss — not because GROQ is clever, but because the fact was made structural instead of textual.

And the other direction, honestly

The knowledge base also told me this, which I had read past twice:

If the Conditional Access exclusion is managed by a security group, that group
must be role-assignable or enrolled in PIM for Groups. A regular security group
allows Group Administrators to bypass Conditional Access entirely.

That is a conditional, three documents deep, and I had no field for it. I would have had to already know it mattered in order to model it.

Prose keeps the facts you did not anticipate. A schema keeps the facts you did. That is why the agent reads both, and why I would not replace either with the other.

The model does not do the arithmetic

coverage_delta fetches the graph, does the set algebra in TypeScript, and returns only what changed — the 186 techniques never enter the context window.

It takes a list of connectors, and that matters more than it looks:

A technique held up by one rule from Defender for Identity and one rule from
Defender for Cloud Apps survives losing either one. It goes dark when both go.

Run the single-connector query twice and union the answers, and that technique reports as safe. The unit test for this is the one I care about most:

check('the union of the single answers would have missed it',
  [...mde.techniquesGoingDark, ...mdi.techniquesGoingDark]
    .some((t) => t.attackId === 'T2000'),
  false)

And when asked about a single connector, the tool runs both implementations — the TypeScript, and an equivalent GROQ query inside Sanity — and compares them. Two independent expressions of the same set difference, checked against each other at answer time. When they disagree it says so rather than picking one.

That cross-check earned its keep on the first run against production data.

The bug that every test passed

techniques[]->attackId evaluates under groq-js to ["T1110","T1556"]. The same
projection through Context's groq_query returns
[{"attackId":"T1110"},{"attackId":"T1556"}]. Context rewrites projections —
results carry an _id nobody asked for — and dereferenced scalar fields arrive
wrapped. Nested paths are left alone, so connector->slug.current comes back as a
plain string from the same query. That inconsistency is why it took a while to
see.

Those wrapped values were being used as Map keys. Object keys compare by reference, so every lookup missed, every technique fell through the
"not in the snapshot" branch, and the tool reported that losing a connector killed nine rules and left no technique uncovered.

Silent. Plausible. Wrong in the direction that reads as good news.

Every unit test passed the whole time, because they run against a local fixture through groq-js — they were exercising a shape production never emits. What found it was the cross-check: GROQ said four techniques go dark, the TypeScript said none, and the tool refused to choose.

A fixture is not a substitute for running against the real thing. I knew that and still had to learn it.

Then my own agent made the same mistake as the knowledge base

The router was a paragraph in the prompt. That works on a capable model. Late in the build I moved to a smaller one, and asked the break-glass question again.

It called docs_outline, then knowledge_base_read on the emergency access entry, and answered: at least 16 characters, according to the CIS benchmark. It never touched the graph. Two tool calls, both to one endpoint, and an answer that was cited, confident, and missing the contradiction entirely.

Which is exactly the failure this project exists to expose, reproduced by the thing I built to expose it.

The fix was not a bigger model. It was to stop asking the model to remember. Every knowledge base response now carries the disagreement with it:

return {
  source: 'docs endpoint (Knowledge Base mode)',
  contested: await contestedWarning(graph),   // <- attached to the content
  entries: cap(text, 30_000),
}

Where contestedWarning is one cached GROQ query turned into a sentence the model receives alongside the prose it asked for:

IMPORTANT — the graph records 3 settings whose guidance is CONTESTED between
authorities: breakglass-password-length, block-legacy authentication,
admin-signin-frequency. A knowledge base entry states one claim and generally
does not say that another authority disagrees, because a conflict spanning two
entries is invisible to either of them. If your answer touches any of the
settings above, call claims_for_setting before answering.

Same question, same small model, after the change:

The guidance for break-glass account password length is contested between
authorities. CIS Microsoft 365 Foundations v7.0.0, § 1.1.2: at least 16
characters, randomly generated. Note that this benchmark entry contains
conflicting information: while the remediation steps rely on this password, a
warning at the end of the same section states that MFA is now required for all
users, including break-glass accounts. Microsoft Cloud Security Benchmark, v2
PA-5: at least 32 characters, never expiring, split under dual control. These
sources disagree on both the required length and the overall authentication
strategy.

It also got cheaper. The model read the contested list in the outline response and went straight to the graph, skipping the entry read altogether — two calls instead of three, because it no longer needed to read something it already knew was incomplete.

That is the same move twice, at two different layers. conflictsWith made a disagreement between sources structural instead of textual, so a query could find what a prose index could not. This made the warning structural instead of instructional, so a weak model could not skip what a paragraph had merely asked it to do.

Both times the lesson was the same: if a fact matters, do not ask the reader to remember it — put it where they cannot miss it. Which is, more or less, the whole argument for modelling content.

What is invented, and why I say so

The connectors, tables, rule logic, ATT&CK techniques and every cited benchmark control are real, imported from public sources and cited to the page.

Four fields on each detection rule are not. status, fpRate, lastValidated and ownerTeam are generated from a hash of the rule ID — deterministic, so the demo does not reshuffle, but invented. They describe operational history with a rule, and no public repository can know that.

So the agent is instructed to say so whenever one of them drives an answer, and it does:

Synthetic Data Note: the ownerTeam and status fields used to identify
these rules are synthetic demonstration data.

Ranking rules by a fabricated false-positive rate without mentioning that it is fabricated is the most dishonest thing this application could do, and it would look exactly like competence.

Sanity Project Details

Project ID	6qz0b6rp
Dataset	production
Studio (public)	detection-debt.sanity.studio
Document types	connector, logTable, detectionRule, technique, baselineControl, tuningDecision
Content	8 connectors, 18 log tables, 40 rules, 186 techniques, 8 controls
Knowledge Base	26 entries from 2 PDFs, Microsoft Learn pages, and the dataset

The Studio is public and navigable. Open a detection rule and its tables and techniques are links you can follow — which is a better look at the content model than a project ID is.

---

What I would tell myself at the start

Model the disagreement, not just the claim. conflictsWith is one array of references and it is the reason the hardest question in this dataset has an answer.

Keep the arithmetic out of the model. Not because models are bad at it, but because when they get it wrong the answer is fluent and nobody can see the error.

Put warnings in the tool output, not the system prompt. A prompt is read once, far from the moment it matters. A tool result arrives in the model's hands exactly where the decision gets made, and the smaller the model, the larger that difference gets.

And run two implementations of the thing you care about most. It found a bug that 36 unit tests did not.