Submission for the DEV x Sanity Challenge, Path One.
What I Built
JudgeStack answers Magic: The Gathering rules questions, but the answer itself is only half the job. It also works out which source has the authority to decide that answer.
Depending on the question, that might mean retrieving the card's current Oracle wording, an older printing, a rule, a ruling, or a format announcement. If the evidence is missing, JudgeStack is supposed to say so instead of filling the gap from memory.
That matters because Magic does not have one simple source hierarchy that works for every question:
| Question | Governing authority |
|---|---|
| What does this card say now? | Current Oracle wording, CR 108.1 |
| How does it interact with the rules? | Oracle plus the current Comprehensive Rules |
| What did it do in an earlier year? | The rules and wording effective on that date |
| Is it legal now? | The current format-legality claim |
| Since when has it been legal, banned or restricted? | A dated format announcement, and nothing else |
| Why does my card say something different? | Printed wording against current Oracle wording |
The two legality rows are separate for a reason. Current card-legality data tells you that a card is banned in Modern today. It does not tell you that the ban began in 2017. That second claim needs a dated announcement.
If the system treats those as the same question, it can invent an effective date while sounding completely certain.
I also wanted to know whether this structure was actually helping. That is something I could test.
I held back ten questions that were never run during development, then tested two evidence-gathering systems using the same model and answer prompt. The answers were shuffled, stripped of their condition labels, and graded blind.
| one-shot keyword retrieval | Sanity Context | |
|---|---|---|
| verdict correct | 1/10 | 9/10 |
| reasoning rests on something unretrieved | 7/10 | 1/10 |
The method, limitations, and one metric I had to throw out are all below.
Demo
Code
https://github.com/joshuarg007/judgestack
There are two tagged versions of the evaluation.
eval-2026-09-22 is the code both conditions ran from.
eval-2026-09-22-artifacts contains the frozen outputs, which were committed after the run:
- Lexical results
- Structured results
- Blind grading result
- Blind rubric and pack builder
- The 30 questions
- Raw blind judgments
JUDGESTACK_PROVIDER=deepseek npm run eval -- --condition structured
How I Used Sanity
The corpus contains 496 documents across ten types:
card, printing, ruleParagraph, glossaryTerm, formatEvent, claim, decision, textDifference, adjudicationCase, and authoritySource.
That breaks down to thirty cards, seventy-seven printings, sixteen rule paragraphs, two hundred and eight legality claims, and seventy-four detected differences between printed wording and current Oracle text.
Two Sanity Context MCP endpoints sit on top of that content:
- A dataset endpoint for filtered GROQ queries over the structured documents
- A knowledge base endpoint that serves the Comprehensive Rules as a file source
They are separate because a Context endpoint with a dataset source ignores its knowledge base sources. Trying to combine everything into one endpoint would quietly remove access to the rules file.
I also had to think about Wizards' IP. The public dataset stores only the sixteen rule paragraphs cited by reviewed cases, which lets the system verify those citations against exact text. The complete rules file is used as a retrieval source instead of being republished as hundreds of separate dataset documents.
That does not settle the licensing question, and I am not claiming it does. It keeps the public structured dataset narrow and reduces how much Wizards text I duplicate.
Sanity Project Details
Project ID: g2mmasfd
Dataset: production
Studio: https://judgestack.sanity.studio/
The content model is public in the repository and does not require Studio access.
schemaTypes/ contains all ten document types.
eval/questions.json contains the thirty evaluation questions, including the card ids and rule numbers each question requires the system to retrieve.
No login is needed to use the app.
The Evaluation
It is easy to make a demo produce one convincing answer. I wanted to know whether the structured content was consistently doing useful work, so I built the comparison before I trusted the output.
The evaluation contains thirty questions across three categories:
- Printed wording versus current Oracle text
- Current format legality
- Historical rules changes
Ten questions were held out and never run during development.
The two conditions gather evidence differently.
The lexical condition runs one BM25 search over a flattened copy of the corpus and injects the top twelve chunks into the prompt. It gets one retrieval pass and one answer.
The structured condition can query the Sanity dataset with GROQ, read the rules knowledge base, and follow references between documents. It gets up to ten model steps.
Both conditions used deepseek-flash through api.deepseek.com with the same answer prompt. Temperature and output-token limits were left unset in both, so both used the provider defaults.
This is not a clean comparison of BM25 against GROQ. It is a comparison between two evidence-gathering architectures. One receives a single result set. The other can decide what to query next and follow references. It also gets more model turns because that is part of the agentic design being tested.
Calling it only a retrieval comparison would overstate what I controlled.
An evaluation that does not say which model produced the answers is not much of an evaluation.
Held-Out Results, Graded Blind
The twenty holdout answers, ten from each condition, were shuffled together.
They were graded by OpenAI GPT-5.6 Sol through the ChatGPT interface. The judge received the questions, expected verdicts, and grading rubric. It was not given the condition labels or told how many answers came from each system.
The answering model and judging model came from different vendors, and the judge was not involved in building either condition.
The complete rubric and pack builder are in eval/blind-pack.ts.
| lexical | structured | |
|---|---|---|
| verdict correct | 1/10 | 9/10 |
| reasoning rests on something unretrieved | 7/10 | 1/10 |
There is one reproducibility limitation here. The grading happened inside ChatGPT, so the exact judge build is not pinned the way the DeepSeek answer model is.
The pack, rubric, and raw judgments are all public, so anyone can repeat the grading with another judge.
Full-Suite Diagnostics
These numbers cover all thirty questions, including the twenty used during development. They are useful diagnostics, but they are not an unbiased test set.
| lexical | structured | |
|---|---|---|
| required rules cited | 19/30 | 30/30 |
| required cards retrieved | 22/30 | 30/30 |
| cited rules actually retrieved | 24/30 | 30/30 |
| unsupported citations | 6 | 0 |
The difference is large enough to justify more testing. Ten untouched questions are not enough to make broad claims about every Magic rules question.
A Metric I Am Withdrawing
I originally had a fifth automated check called "date discipline." The structured condition scored 10/10.
I am not reporting that number because the metric does not support it.
This was the check:
const hasFormatEvent = r.retrievedIds.some((id) => id.startsWith('formatEvent'))
It only asks whether the system retrieved any formatEvent. It never checks whether that event concerns the card in the question.
The corpus contains two format events, both for the same unrelated card. An answer could invent an effective date and still pass after retrieving either one.
The metric also cannot tell the difference between these two statements:
The card is banned as of 2026-09-18.
The ban became effective on 2026-09-18.
The first describes when the status was observed. The second claims to know when it took effect. That is the exact distinction the metric was supposed to enforce.
There was another problem. The saved evaluation rows included retrievedCount, but not retrievedIds, so the metric could not be recomputed from its own frozen artifacts.
Both defects are fixed in the repository. The original number remains withdrawn.
If I run those same ten questions again, I will report the result as a versioned regression run, not as another untouched holdout. Once I examined those questions and used them to repair the evaluator, they stopped being fresh. A new unbiased estimate needs a new frozen holdout.
Where the Grading Agreed
I graded the structured answers myself before running the blind evaluation. Since I built that condition, I was also the person most likely to give it the benefit of the doubt.
My labels and the blind labels matched on all ten verdict judgments and nine of the ten unsupported-reasoning judgments.
I had also graded the lexical answers earlier without blinding. That pass produced the same 1/10 verdict score and 7/10 unsupported-reasoning count as the blind evaluation.
Most importantly, the blind judge found the same structured failure I did.
The Failure I Am Keeping
The structured system was asked where Sol Ring could be registered.
It retrieved and quoted every relevant format claim:
- Legal in Commander
- Banned in Duel
- Banned in Legacy
- Banned in Oathbreaker
- Restricted in Vintage
Then it concluded:
The only format where it may be registered is Commander.
That is wrong. A restricted card can still be registered, usually with a limit of one copy.
The corpus recorded Sol Ring's status in each format, but it contained no document explaining what "restricted" means. The model resolved that missing definition incorrectly and effectively treated "restricted" as another word for "banned."
On another legality question, the same system noticed that the definition was missing and refused to infer it.
Same missing evidence. Two different behaviours.
That is the failure mode I care about. Retrieval systems do not always fail because they found nothing. Sometimes they find almost everything, and the missing piece is exactly the one needed to reach the right conclusion.
The evaluation did more than give the system a score. It found a defect in the schema.
The content model stores each card's legality status, but not the meaning of those statuses. The fix is a legalityTerm type defining legal, banned, and restricted. It also needs to explain how restrictions affect registration, whether sideboards count toward the limit, and which formats use restrictions at all.
I left the frozen result alone. Patching the corpus and rerunning the same questions would make the score look better, but the failure is more useful than that.
Three Things That Cost Me Hours
The AI SDK Versions Did Not Pair
My lockfile combined ai@5.0.261 with @ai-sdk/mcp@1.0.82. That resolved two major versions of @ai-sdk/provider-utils, 3.x and 4.x.
Tool calls then failed while validating the empty object {} with:
value is not a function
Upgrading to ai@6 removed the incompatible duplicate path.
Every @ai-sdk/mcp 1.x release I checked required provider-utils 4.x, so that dependency combination could not run the structured path.
Both Endpoints Had the Same Tool Name
Both Context endpoints expose a tool called initial_context.
I merged the two tool sets with an object spread. One silently replaced the other.
The tool I lost was the dataset schema overview, so the agent never learned that the card and printing document types existed. It retrieved almost no card data while continuing to cite rules correctly.
That looked like a retrieval-quality problem. It was an object-key collision.
The fix was to namespace the tools before merging them.
I Parsed the MCP Response Incorrectly
MCP puts the useful payload inside content[].text.
I stringified the whole tool result instead. That left the quotation marks escaped, so document ids no longer matched a regular expression looking for:
"_id":"..."
Rule numbers still matched because they were bare digits. That made the bug especially convincing. The system appeared capable of retrieving rules but bad at finding cards.
On the twenty tuning questions, card retrieval scored 1/20 before I flattened the MCP output to text. After the fix it scored 20/20, and 30/30 across the full suite.
Almost that entire improvement came from repairing the instrument, not improving the model. It would have been very easy to report it as a retrieval win.
The README contains the other integration problems, including the Studio deployment and transport issues.
Scope, Honestly
JudgeStack is a rules laboratory, not a replacement for a judge.
The current corpus has thirty cards, seventy-seven printings, and sixteen stored rule paragraphs. The project stays below the free-tier limits, and npm run budget fails the build if the corpus grows past its cap.
I also tested one local-model configuration using Qwen3. Across three runs, it made no successful calls to the dataset endpoint and produced invalid arguments for tools that take no parameters.
It could still use the structured corpus when I asked it to produce a retrieval plan as JSON and executed that plan separately. That third condition is included in the repository.
That result applies to one model configuration across three runs. It is not a claim that local models cannot use tools.
JudgeStack is unofficial Fan Content permitted under the Fan Content Policy. Not approved/endorsed by Wizards. Portions of the materials used are property of Wizards of the Coast. (c)Wizards of the Coast LLC. Card data via Scryfall and MTGJSON.
Top comments (0)