An Italian copy reviewer that cites its style guide, built on Sanity Context

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

I'm Italian, and part of my work is checking the Italian copy of software websites. The same mistakes keep shipping: "Fidato da" for "Trusted by", "Utensileria" (a hardware store) as the menu label for "Tools", "5 Mld+ Altoparlanti nativi" (native loudspeakers) for "5B+ native speakers". A general model catches some of these and misses others, and when it does catch one it cannot tell you which house rule it applied.

So I built a reviewer agent. You give it a string, where it appears and, if you have it, the English source. It reads the team's Italian style guide from a Sanity Knowledge Base, answers ok or error, proposes the corrected text, and lists the Knowledge Base entries it relied on. The style guide is structured content in Sanity: 21 rules and 24 glossary terms. Fourteen rules carry real strings that shipped on live product pages next to their native correction; six are house decisions the team made.

I also measured it, and I report every run. On the 12 held-out generic pairs the Knowledge Base never saw, the Knowledge Base and the model alone are within one pair of each other: with the same code filter behind both arms, the latest run gives 11 against 10 pairs fully right, and the Sep 30 run rescored the same way gives 9 against 10. A general model already knows most of what makes Italian copy wrong, so a style guide cannot add much there. The Knowledge Base earns its place on rules a model cannot know: house decisions that differ from generic Italian. On a separate slice of 16 pairs built only from such rules, the agent with the Knowledge Base gets 10 of 16 pairs fully right and the same model without it gets none. The details are below.

Demo

Three real reviews in the local web UI, grounded in the Knowledge Base. Each took 12 to 17 seconds on the free Gemini tier; the GIF cuts the wait, and every card shows the real time.

The same three reviews as a video: https://vimeo.com/1231260014

A review looks like this (real output from the Sep 30 test run, gemini-3.1-flash-lite, input Fidato da oltre 100.000 imprese):

{
  "verdict": "error",
  "reason": "The phrase 'Fidato da' is a literal calque of 'Trusted by' and is considered unnatural in Italian marketing copy.",
  "evidence": [
    "Fidato da"
  ],
  "corrected": "Scelto da oltre 100.000 imprese",
  "rules": [
    "Calques, False Friends & Idiom Errors/\"Trusted by\" calque (major)"
  ]
}

Code

https://github.com/exochard/italian-copy-reviewer

TypeScript on Node 22 with the Vercel AI SDK; a Sanity Studio schema; a small Python script that turns the rules into Sanity documents. npm test runs without network.

How I Used Sanity

The content model. Two document types in the Studio schema:

• l10nRule: title, slug, category (calque, agreement, number format…), severity, the rule in plain words, preferred forms, forms to avoid, and an array of real shipped examples (product type, where the text appears, English source, shipped Italian, native correction).
• glossaryTerm: the English term, the preferred Italian, the forms to avoid, a note.

The examples are the point. "Don't translate literally" is useless to a model; "a software company shipped Fidato da oltre 9,500 marchi e organizzazioni globali and a native reviewer changed it to Scelto da oltre 9.500 marchi e organizzazioni in tutto il mondo" is not.

The Knowledge Base. Two sources: the production dataset above, and Microsoft's public Italian Localization Style Guide (a PDF). They sometimes disagree. When it built the Knowledge Base, Sanity Context flagged three such conflicts as Issues instead of merging them silently; I read each one and resolved it in the dashboard before the final runs.

The agent. The AI SDK connects to the Context MCP endpoint in knowledge-base mode. The system prompt tells the model to call initial_context first and then knowledge_base_read on the entries that could apply. The output is a typed object (verdict, reason, the wrong fragments quoted from the text, corrected text, cited entry paths), so every answer says which rules it used.

The first measured run taught me something. With the Knowledge Base the agent flagged correct text more often than the model alone, because it applied entries whose trigger was not there: it flagged "AI" in a sentence with no "AI" in it, and asked for "oltre" on a number that had no "+". So every flag now has to quote the wrong words, and code drops a flag whose quote is not in the text. The prompt also says an entry applies only when the text contains what the entry is about. I tuned this on the training pairs only.

That was not enough. The second test run showed flags whose quote was the whole string: the agent read the capitalisation entry and called Automazione batch con un clic and RECENSIONI DEGLI UTENTI "Title Case". So a flag that relies only on the Title Case, "+" or AI entry now survives only when the text holds that entry's trigger: a capitalised word in mid-sentence (ALL CAPS never counts), a plus sign, or "AI" and "IA" mixed in one string, which is exactly what the AI entry says. Re-applied offline to the saved training outputs, this took the grounded agent from 12/17 to 16/17 without losing a caught bug. I first applied this filter to the grounded arm only, which is not a fair comparison: re-applied offline to the saved Sep 30 test run, the same filter moves the model-alone arm from 9/12 to 10/12. So the like-for-like numbers below put both arms behind the same filter (--shared-filter in src/eval.ts), and I also show the arms with no filter at all.

One fix came from reading a test failure, and I'd rather say so. On USER REVIEW both arms read the text between the prompt's <<< and >>> markers as an empty placeholder. The prompt now says the text between the markers is the literal string.

In every grounded run on Oct 1, 24 of 24 on the test pairs and 32 of 32 on the house pairs, the agent read at least one entry before answering.

Keeping the test honest. The benchmark has 29 pairs. The train half feeds the Knowledge Base; the test half never enters it, and a unit test asserts the two halves are disjoint. Each pair is scored only when the agent flags the shipped string and leaves the native fix alone, so a reviewer that flags everything scores zero.

Every test run, same 12 pairs, gemini-3.1-flash-lite, pairs fully right. These are the original runs, where only the Knowledge Base arm had the filter; the like-for-like table follows:

Test run	Changed before it (tuned on training pairs)	Model alone	+ Sanity Knowledge Base
Sep 28	first version	7/12	4/12
Sep 29	quote check; "entry applies only when triggered" prompt; two entries reworded	9/12	5/12
Sep 30	per-rule trigger check; literal-text prompt	9/12	9/12

The Sep 30 run in full:

	Bugs caught	Fixes left alone	Pairs fully right
gemini-3.1-flash-lite alone	11/12	10/12	9/12
gemini-3.1-flash-lite + Sanity Knowledge Base	10/12	11/12	9/12

Like for like, the Sep 30 run rescored offline with src/rescore.ts, pairs fully right:

	No filter	Same filter on both arms
Model alone	9/12	10/12
+ Sanity Knowledge Base	9/12	9/12

A fresh run on Oct 1 with the shared filter on, after the Knowledge Base was rebuilt with the house rules, gave 11/12 for the Knowledge Base and 10/12 for the model alone.

The model alone moves between runs too: both arms share the prompt, and the model is not fully repeatable even at temperature 0. Twelve pairs is a small test, so one pair is eight points.

What still fails in the Sep 30 run: both arms passed Transcrizione, which should be Trascrizione, so neither checks spelling letter by letter. With the Knowledge Base the agent called the correct Ottieni informazioni utili e pratiche a calque of "Get", and let an English string, Larger file per task, through as correct. The model alone failed on two correct strings instead: it carried the "+" of the English source into oltre 100 siti and called it redundant, and it objected to the preposition in per attività. What the Knowledge Base does well is the explanation: every flag names the entry it came from, so a reviewer can see why it fired and fix the entry instead of the prompt.

Where the Knowledge Base cannot be replaced. A keyword search over a style guide would not beat a model on generic Italian, and on that test it does not. So I added a held-out slice of 16 pairs (house in content/split.json, pairs in content/house.jsonl) whose verdict depends on six rules and five glossary terms the team chose and that exist only in the Knowledge Base: always "tu" and never "Lei", error messages that open with "Non siamo riusciti a", no exclamation marks in product copy, "persona" instead of "utente", no "per favore", the euro sign after the amount, "spazio di lavoro" for "workspace", "pagamento" for "checkout", plan names kept in English as "Free" and "Pro". Both halves of every pair are correct Italian in general; only the shipped one breaks a house rule. For example: Acceda al suo account per continuare against Accedi al tuo account per continuare, Invita un utente nel progetto against Invita una persona nel progetto, Da €29 al mese against Da 29 € al mese. No prompt, filter or rule was tuned on this slice, and none of its strings is in the seed (a unit test checks both the split and the seed). Pairs fully right, both arms behind the same filter: Knowledge Base 10/16, model alone 0/16. Neither arm flagged a single correct string. The six the agent missed: checkout, e-mail and two uses of utente went through although the glossary covers them, and so did an error opening with Errore: and an exclamation mark. I wrote these pairs myself to test the claim, so they say what the Knowledge Base can do when the rules are in it, not how often real copy breaks house rules.

Sanity Project Details

Project ID: 3ezm3tm1 (dataset production). The schema is in studio/schemaTypes/ and the seed documents in content/seed.ndjson.

The dataset is public, so anyone can query it without a token, for example the six house rules:

curl -G "https://3ezm3tm1.api.sanity.io/v2025-02-19/data/query/production" --data-urlencode 'query=*[_type=="l10nRule" && category=="house_style"]{title, rule, prefer, avoid}'

How this was made

I wrote the benchmark strings and their corrections from my own QA audits; they predate the challenge, and the brand names are masked. The code and the Knowledge Base content were written during the challenge with an AI coding assistant (Claude Code): I set the spec and the acceptance checks, it drafted the code, and I reviewed, ran and tested every part. The numbers above come from npm run eval runs (results/eval-*.json) and are reported as measured, including the ones that went against the Knowledge Base. src/rescore.ts re-applies the flag checks to a saved run without calling any model and prints the arms with and without the filter.