What I learned building a compliance answer engine that refuses to make things up

Adi CCH — Tue, 07 Jul 2026 21:53:18 +0000

Swiss business compliance is a genuinely hard information problem. There is no national minimum wage, VAT rules live at the federal level, social-insurance (AHV) admin runs through 26 cantonal offices, and most of the official guidance is only in German or French. If you are a foreign founder setting up a GmbH, you are stitching answers together from a dozen government PDFs.

I have been building Canton Compliance Hub, a free multilingual resource that tries to make that legible. The interesting engineering problem was not the content, it was trust: in a domain where a wrong number can cost someone real money, the system has to be citable and it must not hallucinate. Here is what actually worked.

Two-layer retrieval beats one

The naive RAG setup is: chunk documents, embed them, retrieve top-k, stuff into the prompt. It works until a legal term like AHV or MWST needs an exact match that a dense vector glosses over.

I run two layers on Postgres with pgvector:

Raw chunks are the ground truth. They keep the original wording and the source URL, so every answer can point back to a specific government page.
Fact cards are distilled atoms extracted from those chunks, embedded separately, and queried first.

Retrieval is hybrid: a BM25 pass catches the exact legal acronyms, a vector pass catches the paraphrases, and a metadata filter (canton, topic, language) runs before ranking so a Geneva question never pulls a Zurich-only rule.

Make hallucination structurally hard

"Please don't hallucinate" in a system prompt is not a control. What worked was making every factual claim carry a citation marker back to a chunk, then treating any claim without one as a defect.

Generation emits inline citation markers. A claim with no marker gets flagged.
A cite whose id is not in the retrieved set is a hallucinated citation, and the draft is rejected and retried at temperature 0.
For the numbers (rates, thresholds, CHF amounts), a separate verification pass marks each numeric claim supported, unsupported, or contradicting against the retrieved evidence only. Unsupported numbers block publication.

That last pass was the highest-leverage thing we added. Cheap draft models are confident and wrong exactly where it matters most in compliance: a plausible-looking rate.

Change detection is code, not AI

The corpus has to stay current, but re-summarising an unchanged page with an LLM is waste. Change detection is four boring signals: Last-Modified, ETag, an on-page timestamp, and a SHA-256 of the extracted text. The LLM only runs when the content genuinely changed.

Ship the primitives as tools

Two byproducts became reusable public artifacts, grounded in official sources and published as open datasets (CC BY 4.0): a minimum-wage-by-canton comparison, and an SME compliance-deadline calendar. The whole retrieval stack sits behind a free compliance self-check: describe your situation in plain language and get a personalised, source-cited overview.

Takeaways

Two-layer retrieval (atoms first, raw chunks as citation fallback) is worth the extra table.
Treat a missing or unknown citation as a build error, not a style preference.
Verify numbers as a separate pass. The draft model will keep inventing them; a checker against your own corpus is the fix.