RAG has a trust problem: a model answers fluently, cites nothing, and you can't tell fact from confident improvisation without manually checking the source docs. This gets worse, not better, with smaller models — they're more likely to fill gaps with something plausible, and less likely to hedge.
I've been building AIPK, a package format that tries to close that gap — not by making the model smarter, but by making it provably grounded, sentence by sentence, or explicit about the fact that it isn't.
What it actually is
An .aipk file is a single binary package containing a persona, a RAG knowledge base (chunks + embeddings, no external vector DB), skills, tool definitions, and an auditable set of claims: atomic, sourced factual statements extracted from the underlying documents.
aipk serve is a ~9 MB Rust binary that turns a .aipk file (or several, merged) into an OpenAI-compatible API on top of any backend that speaks that protocol — Ollama, vLLM, a cloud API, whatever. Point an existing chat UI at it and the package shows up as a model. Swap the backend model and the knowledge base, claims, and audit trail travel with it unchanged.
Sentence-level provenance, not vibes
Most "grounded RAG" setups stop at retrieval: stuff some chunks into the context window and hope the model sticks to them. AIPK adds an enforcement and auditing layer on top of that:
-
Claims extraction. An LLM pass reads the source documents and pulls out atomic factual statements — one per sentence-level fact — each tagged with its source. These go through a lifecycle (
extracted → reviewed → canonical → deprecated) before they're trusted. -
Strict-render. In this mode, the system prompt requires the model to append a
[claim_id]citation after every factual sentence it writes. No citation, no trust. -
aipk verify. A separate, independent pass parses the model's answer sentence by sentence, checks that every citation resolves to a canonical claim, and reports a coverage score: what fraction of the answer's factual sentences are actually backed by something in the package, versus asserted on the model's own authority.
None of this requires a word-for-word quote match — that would be brittle. It requires a valid ID pointing at a real, reviewed, canonical claim.
The benchmark
To find out if this moves the needle rather than just adding ceremony, I built a small fictional corpus — a company handbook, an incident-response SOP, and a product spec for a robot that doesn't exist — specifically so the model couldn't fall back on pretraining knowledge. 23 questions: 15 answerable from the corpus, 8 deliberately not.
Model: llama3.2:3b, run locally. Two conditions: vanilla RAG vs. --strict-render, both scored by aipk verify.
| slice | vanilla coverage | strict-render coverage |
|---|---|---|
| in-corpus (15 questions) | 0.933 | 0.983 |
| out-of-corpus (8 questions) | 0.166 | 0.000 |
| out-of-corpus refusal rate | 1/8 | 8/8 |
On the 8 questions the corpus genuinely can't answer, vanilla RAG produced a plausible-sounding response with some grounding 7 times out of 8 — it just quietly filled the rest with invention. Strict-render refused outright on all 8.
One concrete example
The benchmark corpus was synthetic on purpose. A more convincing test was real: a package built from the official Kubernetes documentation (CC-BY 4.0, 111 MB, 35,588 chunks, 1,354 canonical claims), asked about a kubectl flag that doesn't exist (--force-evict).
In normal mode, the model caught it cleanly and substituted the real command. With --strict-render on, it hedged honestly ("insufficient grounded information") — then invented a different fake flag while explaining why the first one was fake. Citation-only prompting doesn't stop a model from writing fiction; it just makes the fiction visible if you check:
_aipk: canonical_used=0 invalid_ids=[] coverage=0.00 uncited_sentences=4 fully_grounded=false
Zero canonical citations, zero coverage. The system didn't stop the model from bluffing — it caught the bluff and said so in a structured, machine-checkable way.
What dogfooding actually found
Building the benchmark and the Kubernetes package surfaced real defects synthetic unit tests hadn't:
-
The coverage report used to lie by omission. The original
fully_groundedcheck only verified that already-cited claim IDs were valid — an answer with zero citations passed trivially. Fixed to check what fraction of sentences carry a citation at all. -
A missing token limit could hang the whole request. Nothing capped generation length, so a model that occasionally fails to emit a stop token would generate indefinitely. Fixed with a
--max-tokenscap and a request timeout. - Decimal points were splitting sentences. "8.4 kWh" silently became two sentences, mangling claim extraction. Fixed across all three places doing sentence splitting.
None of these are exotic. They're the kind of bug that only shows up when you run the loop end to end at realistic scale.
Try it
.aipk files are self-contained — one file, offline-capable, works with any OpenAI-compatible local or cloud backend.
curl -fsSL https://aipk.dev/install.sh | sh
Repo: https://github.com/ArchDuran/aipk
Site + demo video: https://aipk.dev
Source-available under BSL 1.1 — free for evaluation, individuals, and small teams; commercial license for larger production use.
Feedback on the coverage methodology, the claim lifecycle, or where this still falls over is genuinely wanted.
Top comments (1)
Sounds great! I like the idea of making AI show exactly where its answers come from instead of just trusting what it says. Definitely worth keeping an eye on!