Keyword search found the facts but lost the sources: an Oracle-to-Azure agent on a Sanity graph

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

Q: Some posts say DBMS_JOB was removed in 19c. True?

Agent: Verdict: False. DBMS_JOB was not removed in 19c — it is deprecated (since 12.2) but still functional through 26ai. There's a recorded dispute over exactly this misconception.

Dispute on record: "Did DBMS_JOB stop working in 19c?" … Resolution (resolved by DBMS_JOB, PL/SQL Packages and Types Reference 19c, Oracle): Both observations are compatible. From 19c, a schema calling DBMS_JOB.SUBMIT needs the new CREATE JOB privilege or it fails with ORA-27486 — that privilege change, not removal, explains most "stopped working" reports.

— the full answer, with every query the agent ran

That answer comes from joins: a feature, its deprecation release, a dispute document pointing at two caveats from two publishers, and a resolution with its own source. I write Uptime Architect, a blog and hands-on Oracle labs, and the migration question I see answered badly is never just "can this move to Azure?" It is "on which Oracle release, to which Azure service, and says who?" That is a join, not a search. So I built the join, and measured whether it mattered.

Same 18 questions. Keyword search found the right answer 14 times but cited the required sources only 4 times out of 16; the Sanity agent cited them 12 times. From memory, the model grounded 0 of 18 answers; the agent grounded all 18.

The full table (18 held-out questions, four ways of answering, graded blind):

Condition	Verdict correct	Cited every required publisher	Grounded	Refused when it should
Model alone, no tools	10/18	11/16	0/18	0/2
Keyword search (BM25) over the same documents	14/18	4/16	16/18	2/2
Sanity Context, GROQ	17/18	12/16	18/18	2/2
Sanity Context, GROQ + Knowledge Base	17/18	12/16	18/18	2/2

👉 Try it without installing anything: The Coroner's Reports replays every answer side by side, with every GROQ query the agent ran, the grader's note, turns, tokens and time.

What I Built

ora2az is a migration advisor for teams moving Oracle Database workloads to Azure. It answers only from a structured knowledge graph in Sanity, cites every claim, scopes every claim to Oracle versions and Azure services, shows both sides when sources disagree, and says "the dataset has no evidence on this" instead of guessing. In the eval, all 36 of its answers were grounded; that is measured, not guaranteed.

The graph is the product: 446 documents from public sources only (Oracle's documentation, Microsoft Learn, community tool docs, my own blog and labs).

Type	Count	What it holds
oracleFeature	50	introduced / deprecated / desupported release, Oracle's named successor
azureTarget	21	Azure SQL DB, Managed Instance, PostgreSQL Flexible Server, Oracle Database@Azure, Fabric, …
mapping	101	feature → target, fidelity (exact / partial / workaround / none), effort, steps, versions it applies to
caveat	88	a checkable statement scoped to Oracle versions, editions and Azure tiers, with one evidence source
dispute	5	two caveats whose sources disagree, and the resolution
source	155	title, publisher, URL, doc version, date read
glossary, pattern	16 + 10	definitions and how-to prose (the Knowledge Base)

The agent is Claude Code running headless (claude -p) with claude-sonnet-5, a system prompt (agent/persona.md), and one tool surface: two Sanity Context MCP endpoints. Shell, file and web tools are disabled. A small CLI wraps it: ora2az ask "…".

Demo

1. A dispute (above).
2. Version scoping. "We still run a non-CDB 19c database. Does that matter?" Non-CDB was deprecated in 12.1 and desupported in 21c. On Azure VMs you can lift it as-is, but only to postpone the conversion.
3. A refusal. "How many Azure regions offer Oracle Database@Azure today?" No document records a count, so it says so and points at the source pages instead of inventing a number.

Code

pyaroslav / ora2az

Oracle → Azure migration knowledge graph on Sanity: an agent over Sanity Context, and The Legacy Obituaries (DEV Sanity Challenge 2026)

ora2az — an Oracle → Azure migration knowledge graph on Sanity

A structured, sourced graph of how Oracle Database features map to Azure, and two things built on it for the DEV Sanity Challenge:

What	Try it
Path One	A migration-advisor agent that answers only from the graph, through two Sanity Context MCP endpoints, with a four-way eval	The Coroner's Reports: every eval answer, with the queries it ran. No install.
Path Two	The Legacy Obituaries: an obituary for every Oracle feature that didn't survive the move to Azure	ora2az.vercel.app

Sanity project udyjvgsk · dataset production (public; query it anonymously, e.g. all disputes). Studio at https://ora2az.sanity.studio needs a Sanity login.

The result in one table

18 held-out questions, graded blind (method and caveats):

Condition	Verdict correct	Cited every required publisher	Grounded
Model alone	10/18	11/16	0/18
Keyword search (BM25) over the same documents	14/18

…

View on GitHub

agent/ holds the rules, endpoint config, CLI and eval/. content/test/ has 14 assertions that run against the public dataset (npm test).

How I Used Sanity

Two Sanity Context endpoints, two jobs.

• ora2az-groq serves the whole dataset in GROQ mode: initial_context for the schema, schema_explorer for a type's fields, groq_query for the joins, array_field_reader for long Portable Text.
• ora2az-kb serves a Knowledge Base built from the pattern and glossary documents: 26 documents became 12 entries. I set its source through the GROQ query tab so every entry carries its citations: *[_type in ["glossary","pattern"]]{..., "sources": sources[]->{title, publisher, url, docVersion}}. The agent reads the outline with initial_context and pulls entries with knowledge_base_read.

The joins are the point. A typical answer needs one query like this:

*[_type == "mapping" && oracleFeature->slug.current == $slug]{
  fidelity, effort, appliesToOracleVersions,
  "target": azureTarget->name,
  "caveats": *[_type == "caveat" && references(^._id)]{
    severity, statement, appliesTo, "evidence": evidence->{title, url}
  }
}

A caveat without its evidence source is just an opinion. A mapping without the Oracle versions it applies to is wrong half the time.

The eval, and what it does and doesn't show

Setup. 18 questions written before the content was finished (one expected answer, q02, was corrected afterwards when research showed a tool default I had wrong; marked in the repo).

• Model alone: claude-opus-5-5, no tools.
• Keyword search: claude-opus-5-5 with the top 8 BM25 passages, one pass, one passage per document, references not followed.
• Sanity Context (both): claude-sonnet-5 as the agent above, up to 14 turns.

The baselines ran on the larger model (the runner didn't pass a model for them, so they got the default). That favours the baselines, not the graph. Answers were shuffled and relabelled, then graded blind by claude-opus-5-5 against a fixed rubric; citations were checked by script.

Condition	Median turns	Median tokens in / out	Median time
Model alone	1	16.5k / 2.1k	22 s
Keyword search	1	19.0k / 1.0k	10 s
GROQ	5	43.8k / 2.2k	28 s
GROQ + KB	7	62.7k / 2.5k	33 s

(Flat-rate subscription, so no per-run dollar figure.)

What I take from it:

• Keyword search found the facts but not their sources: 4 of 16. Citations live on source documents that one-pass keyword search never joins to the claim. The fair rebuttal is "denormalise the URLs into every passage" — which is rebuilding the graph by hand. This compares architectures (joins and multiple turns vs one retrieval pass), not just indexes.
• The model alone "cites" 11 of 16, but 9 of the 59 URLs it produced do not resolve.
• The eval found a hole in my data. On q04 every retrieval condition missed that Oracle Database@Azure keeps Oracle's empty-string-is-NULL behaviour, because I had never mapped that pair. The grounded agent declined to claim it rather than guess. I added the mapping and re-ran: all four conditions got it.

The Knowledge Base, honestly. It didn't move the frozen score, and on q13 it hurt: with it available, the agent read two entries about scheduler jobs, never queried the disputes, and dropped one side (frozen run). Prose competed with structure and won the turn budget. The fix was one rule — query disputes before Knowledge Base prose — and the re-run cites both sides. Then I added two how-to questions after the freeze, the kind of prose the Knowledge Base holds. On choosing a method for an AIX-to-Exadata move, the Knowledge Base agent read the three right entries in one call and cited every required publisher; GROQ alone reached the same verdict through eight calls and missed the Oracle citation. Two questions are a signal, not a result — but it's the pattern I'd expect: GROQ for facts and edges, the Knowledge Base for prose, disputes first. The frozen table above is unchanged; the later runs are shown separately in the Coroner.

A second grader from another vendor. To answer "the grader is the same family as the model", gemini-3.8-flash regraded the frozen answers under the same blind protocol. Google's free tier ran out of capacity partway, so this covers q01–q09 so far (36 answers); I'll update it when the rest finish:

Condition	Verdicts agreed	Grounding agreed
Model alone	7/9	2/9
Keyword search	9/9	7/9
Sanity Context, GROQ	9/9	9/9
Sanity Context, GROQ + KB	9/9	9/9

The graders agree on every judgement about the Sanity answers. The disagreements are all about the baselines, mostly "grounded": the second grader accepts a correct answer from memory as grounded, while the rubric counts only claims backed by retrieved evidence.

Limits. One run per question; questions written by me; second grader so far on half the questions.

One thing Sanity taught me the hard way: a dot in _id is a path prefix. My first seed wrote all 445 documents as feature.dbms-job, and every public query returned zero, because they were hidden the way drafts. documents are. Ids are now feature_dbms-job, and a test guards it.

Sanity Project Details

• Project ID: udyjvgsk · dataset production (public)
• Schema, as code: studio/schemaTypes (Studio at https://ora2az.sanity.studio needs a Sanity login; the dataset itself is public)
• Try a query: all five disputes
• The same dataset powers my Path Two entry, The Legacy Obituaries (live at https://ora2az.vercel.app; its post follows later this week)

---

Sources are public only: Oracle documentation, Microsoft Learn, community documentation, and my own public blog and labs. — Yaro