I ran the same refund-policy question through two models side by side, with the actual return policy pasted in as context. The local Llama model answered directly and scored 75% faithful to the source text. GPT-4o, given the exact same context, refused to answer at all — "I can't help with that" — and scored 0% faithful.
At first that looked like a bug. It wasn't. A refusal makes zero factual claims grounded in the context, so a faithfulness score of zero is the correct answer, not a broken one. That was the moment I trusted the scoring pipeline I'd spent the week building. It wasn't just parroting back "looks fine" — it was actually measuring whether the words leaving the model were tethered to the text I gave it.
v1.0 of this project stopped bad data going in — a PII firewall that blocks sensitive prompts before they reach a cloud model. v1.1 checks whether what comes out is actually true to the source material. This post covers how that works.
Code: github.com/sochaty/llm-governance-engine — tag governance-post-2.
The Gap PII Blocking Doesn't Cover
Blocking PII answers one question: did anything sensitive leave the building? It says nothing about a completely different failure mode — a RAG pipeline that retrieves the right passages and then has the model confidently state something those passages never said.
That's a hallucination, and it's arguably more dangerous than a PII leak in a support or compliance context, because it looks like a normal, well-formed answer. Nothing in the response format tells you it's wrong.
RAGAS (Retrieval Augmented Generation Assessment) is an open-source library built for exactly this: given a question, a model's response, and the retrieved context, it scores whether the response's claims are actually supported by that context (faithfulness), and separately whether the retrieved context was even relevant to the question (context utilization).
v1.1 wires both metrics into the same governance pipeline that already handles PII — same audit trail, same policy engine, same webhook alerts.
Architecture: Two Enforcement Points Now, Not One
The interesting engineering problem here isn't calling RAGAS — it's that faithfulness can only be known after the model has already responded, while PII enforcement has always run before. That's a fundamentally different point in the request lifecycle, and the governance layer had to grow a second enforcement path to handle it.
Everything under "PRE-response" is exactly what shipped in v1.0. Everything under "POST-response" is new, and it reuses the same PolicyEngine, the same PolicyViolation model, and the same webhook delivery path — just triggered from a different place, after the response body already exists.
Scoring a Response: FaithfulnessService
The service is intentionally narrow: given a prompt, a response, and the context that was supplied, it returns two scores or (None, None) — it never raises, because a judge-model hiccup should never break the response the user already received.
# backend/app/services/faithfulness_service.py
class FaithfulnessService:
"""Computes RAGAS faithfulness + context utilization scores for a response.
Scoring is opt-in: callers only invoke this when the caller supplied
retrieved `context` for the request, so plain prompt-only runs never
pay the extra judge-model latency/cost.
"""
def __init__(self, judge_model: str = "gpt-4o-mini") -> None:
self.judge_model = judge_model
async def score(
self, prompt: str, response: str, context: str
) -> Tuple[Optional[float], Optional[float]]:
if not context or not response:
return None, None
client = self._get_client()
if client is None:
logger.warning("Faithfulness scoring skipped: OPENAI_API_KEY not configured.")
return None, None
try:
from ragas.llms.base import llm_factory
from ragas.metrics.collections import ContextUtilization, Faithfulness
judge = llm_factory(self.judge_model, client=client)
retrieved_contexts = [context]
faithfulness_result = await Faithfulness(llm=judge).ascore(
user_input=prompt, response=response, retrieved_contexts=retrieved_contexts,
)
context_result = await ContextUtilization(llm=judge).ascore(
user_input=prompt, response=response, retrieved_contexts=retrieved_contexts,
)
return self._clean(faithfulness_result.value), self._clean(context_result.value)
except Exception as exc:
logger.error("Faithfulness scoring failed: %s", exc)
return None, None
Two design choices worth calling out:
-
Opt-in, not automatic. If you don't pass
context, none of this runs — no extra latency, no extra OpenAI call, identical behavior to v1.0. Faithfulness is meaningless without a context to check against, so there's no default to fall back to. -
The judge is a live-resolved OpenAI key, pulled from
settings_serviceon every call, the same wayllm_orchestratorresolves provider keys. Change the key in the Settings UI, next request picks it up — no restart.
RAGAS's modern ragas.metrics.collections API needed Faithfulness/ContextUtilization as async collection metrics rather than the older synchronous scorer classes — that's the 0.4.3-specific API surface this wraps.
A Governance Rule That Can't Actually Block
faithfulness_score_below is the fifth policy condition, alongside pii_detected, safety_score_below, cost_exceeds, and model_is — but it behaves differently from all of them:
# policies/default.yaml
- id: low-faithfulness-warn
name: "Low Faithfulness / Possible Hallucination"
description: >
Warns when a response's RAGAS faithfulness score against the supplied
retrieved context is below 0.6 — the response may contain claims not
grounded in the provided context.
condition: faithfulness_score_below
threshold: 0.6
action: warn
Every other condition in this policy engine evaluates before the model is called, so a block action means "the prompt never reaches the model." Faithfulness can't work that way — by the time you know a response wasn't grounded in its context, the response has already streamed to the client. There's nothing left to block.
So enforcement.py downgrades any block action fired post-response into an alert instead:
# backend/app/governance/policy/enforcement.py
# Actions that are still meaningful once the LLM response has already started
# streaming — a "block" verdict can no longer stop delivery post-response, so
# it is recorded as an alert instead (see record_violations()).
_POST_RESPONSE_ACTION_DOWNGRADE = {"block": "alert"}
async def _record_violation(
db, violation, context, webhook_url, *, post_response: bool = False,
) -> None:
action = violation.action
if post_response and action in _POST_RESPONSE_ACTION_DOWNGRADE:
logger.warning(
"Rule '%s' fired post-response with action=%s, which can no longer "
"block an in-flight stream — recording as '%s' instead.",
violation.rule_id, action, _POST_RESPONSE_ACTION_DOWNGRADE[action],
)
action = _POST_RESPONSE_ACTION_DOWNGRADE[action]
# ...persist PolicyViolation row, fire webhook if configured...
It's still fully audited and can still fire a webhook to Slack — it just can't stop something that already happened. record_violations() became a shared helper so both the pre-response dependency and this post-response path write to the same policy_violations table through the same code.
The Request Body Had to Change Shape
The streaming endpoint used to take prompt and provider as query parameters. Retrieved RAG context can be a few thousand characters — that doesn't fit comfortably in a URL, so /benchmark/stream moved from GET + query params to POST + JSON body:
# backend/app/schemas/benchmark.py
class BenchmarkRequest(BaseModel):
prompt: str = Field(..., min_length=1)
provider: str = Field("cloud", pattern="^(cloud|local)$")
provider_type: Optional[str] = Field(None, description="openai|anthropic|google|groq|ollama")
model_id: Optional[str] = Field(None, description="Model ID override, e.g. claude-sonnet-4-5")
context: Optional[str] = Field(
None, description="Retrieved RAG passages to score the response's faithfulness against"
)
If you're calling the API directly rather than through the dashboard, this is the one breaking change in v1.1 — update your client from query params to a JSON body.
What Shows Up in the Audit Vault
The History table and PDF export both grew two new columns: Faithfulness and Context Utilization. They read — for any run where no context was supplied (the vast majority of pre-existing rows, and any plain prompt-only run going forward), and a real percentage for scored RAG runs.
| Prompt | Provider | Faithfulness | Context Utilization | Verdict |
|---|---|---|---|---|
| "How many days do I have to request a refund?" | cloud (gpt-4o) | 0% | — | refused to answer |
| Same prompt | local (llama3.2) | 75% | 88% | passed |
That table is from the actual live test that shipped this release — real scores, not fixtures. The low-faithfulness-warn rule fired on the 0% row and showed up in /api/v1/policies/violations exactly as designed.
Two Bugs That Only Showed Up Against a Real Database
Everything passed 168/168 unit tests and 92% coverage locally before I ran this against Docker Compose. Two things still broke on first real run:
1. Unpinned transitive dependencies drifted between environments. pip install ragas resolved different langchain/instructor versions inside the Docker image than in my local venv, and the judge call failed with No module named 'langchain_community.chat_models.vertexai'. Fixed by pinning exact versions in requirements.txt — instructor==1.15.4, langchain==0.3.30, langchain-community==0.3.31, langchain-core==0.3.86, langchain-openai==0.3.35, langchain-text-splitters==0.3.11, langsmith==0.4.37, tiktoken==0.13.0. If you've been bitten by a RAGAS/langchain dependency mismatch before, this pin set is a reasonable starting point.
2. Existing Postgres volumes were missing the new column. This project doesn't run Alembic migrations yet — the schema is created once via Base.metadata.create_all(), which never alters a table that already exists. Anyone upgrading from v1.0.0 with data already in their policy_violations table needs to run this once by hand:
ALTER TABLE policy_violations ADD COLUMN IF NOT EXISTS faithfulness_score FLOAT;
Both are called out in the v1.1.0 release notes — but "it passed CI" and "it works against a real database someone has been using for a month" turned out to be two different claims, which is exactly why this got a live Docker Compose test before being called done.
Running It
git clone https://github.com/sochaty/llm-governance-engine
git checkout governance-post-2
cp .env.example .env
# OPENAI_API_KEY is required now even if you're only benchmarking other
# providers — it's the RAGAS judge model for faithfulness scoring.
docker compose up
Dashboard → http://localhost:4200
Open the dashboard, paste a short policy or FAQ excerpt into the new context field, ask a question it does or doesn't actually answer, and run it against both a cloud and a local model. Watch the Faithfulness column diverge.
One gotcha if you're testing with your own prompts: avoid place names ("France," "the Paris office"). Presidio flags country/city names as LOCATION PII at high confidence, and the existing pii-cloud-block rule (threshold 0.7) will block the request before it ever reaches a model to be scored. A refund-policy or FAQ-style prompt sidesteps this entirely.
What's Next
- v1.2 — FinOps dashboard: daily cost trends per model, Z-score anomaly detection, budget circuit breakers.
- v2.0 — Multi-tenant workspaces with JWT + RBAC, PostgreSQL row-level security for tenant isolation.
The original roadmap for this release described a local Ollama model as a free faithfulness judge. That's deferred, not shipped — RAGAS's modern collection-metric API needs an OPENAI_API_KEY for reliable structured-output scoring at this stage, so v1.1 ships with OpenAI as the judge and a local-judge option stays on the list for a follow-up rather than going out half-working.
Full code: github.com/sochaty/llm-governance-engine
Reproduce this post exactly: git checkout governance-post-2
If you're running your own RAGAS-based scoring pipeline and have found a reliable local judge model that doesn't need a cloud API key, I'd genuinely like to hear about it — open an issue or a Discussion on the repo.
Top comments (0)