DEV Community

Amin Parva
Amin Parva

Posted on

Your hallucination checker only sees the final paragraph

Your hallucination checker only sees the final paragraph.

That’s the bleed. A fluent wrong number often starts earlier: empty retrieval, a swallowed tool error, a stale cache after a fact update. Score the prose alone and you miss the cause — then the model fills the gap confidently.

Old way

Most “anti-hallucination” tools optimize one job: grade the answer text (encoder / HHEM-class model / LLM-as-judge). Useful. Incomplete for agents.

What they usually can’t do:

  • Halt before generation when the preload is already broken
  • Name a cause-side failure (empty retrieval, tool fail, stale cache)
  • Return an auditable gate you can log in an incident channel — not only 0.87

New way

After the model speaks (and optionally before it does), run a verdict:

  1. Cause — handbook forensics on runtime evidence
  2. Effect — ground the answer against the preload you provided
  3. One outputShineVerdict: decision + named resolution_gate + evidence_hash

That product is PrismShine (Apache-2.0, pip install).

It is not a prompt-injection firewall (that’s PrismGuard). It is not an agent runtime (that’s ChorusGraph). One job: verify answers against evidence.

Proof (in-post — Shine-only receipt)

Public comparative vs Vectara HHEM-2.1-Open on HaluEval

(Azure ACI · ONNX Tier-3 · receipt 2026-07-20_run4_onnx):

System B1 QA F1 B2 numbers F1 B1 p50 LLM
prismshine-fast 0.831 1.000 (0 FP) ~90 ms 0
hhem-2.1-open 0.746 0.926 ~216 ms 0

Receipt folder on GitHub: benchmarks/progress/2026-07-20_run4_onnx (link in first comment).

Plan — try it in under a minute


bash
pip install "prismshine==0.2.2"
prismshine verify --demo
prismshine capabilities
Core path = Tiers 0–2, CPU, 0 LLM calls by default.

from prismshine import EvidenceBundle, PreloadChunk, ShineGate
gate = ShineGate.build(profile="default")
bundle = EvidenceBundle(
    run_id="demo",
    question="What was revenue?",
    answer="Revenue was $1000 in Q1.",
    preload=[
        PreloadChunk(
            chunk_id="c1",
            text="Revenue was $1000 in Q1.",
            source="retrieval",
        )
    ],
)
verdict = gate.verify(bundle)
print(verdict.decision, verdict.resolution_gate, verdict.evidence_hash)
0.2.2 drop-in helpers: validate_grounding · get_gate · enforce_mode_from_env
Shadow without blocking: PRISMSHINE_ENFORCE=0
Docs: INTEGRATION.md §0 (link in first comment).

There’s also a no-API-key browser demo (link in first comment) — pass → fabricated number block → empty-retrieval halt.

Honest limits (BIP)
PASS ≠ world-true — grounded in your preload only
Buffered answers (not mid-stream token verification)
Bare pip install prismshine ≠ Tier-3 span SotA; use prismshine[spans] + ONNX when you need that path
Wired runtime moat is a separate Docker receipt — don’t mix it unlabeled with the HHEM table above
Soft ask
If you try verify --demo and hit a snag, paste the traceback in the comments — I’ll help.

Where would you wire the gate first — after the LLM node, before generation (halt empty retrieval), or both?

### First comment (after publish)
Links (kept out of the lesson above on purpose):

Landing → https://www.insightits.com/products/prismshine.html Demo → https://insightitsgit.github.io/PrismShine/demo.html GitHub → https://github.com/insightitsGit/PrismShine PyPI 0.2.2 → https://pypi.org/project/prismshine/0.2.2/ Receipt → https://github.com/insightitsGit/PrismShine/tree/main/benchmarks/progress/2026-07-20_run4_onnx YouTube → https://www.youtube.com/watch?v=OZCelVhP844 Product Hunt → (paste live PH URL)

Smoke: pip install "prismshine==0.2.2" && prismshine verify --demo
Enter fullscreen mode Exit fullscreen mode

Top comments (0)