DEV Community

Harish Kotra (he/him)
Harish Kotra (he/him)

Posted on AI-assisted

FreshGauge: a freshness control plane inside the LLM gateway

Every LLM answer is a snapshot of the model's training data. Ask for a price, a current model name, a CEO, or a fine cap and you get something that was true — stated with exactly the same confidence as something that still is. Retrieval doesn't fix this on its own: RAG grounds an answer in documents, but nobody checks whether the document the answer leans on is two years old.

FreshGauge is a Bifrost gateway plugin that does that check on the way out. Every completion that passes through the gateway gets its factual claims extracted, verified against live web evidence, scored by authority-weighted recency, and then acted on — logged, corrected in flight, or pre-empted with a dated brief before the model is even called.

This is a write-up of how it's built, the decisions that mattered, and the traps that cost real debugging time.


1. Why the gateway is the right place

You could build this as a wrapper around one SDK. It breaks the moment a team uses three providers, streams tokens, or fails over between models. A gateway sees all of that by construction: one process, one config, every request.

Bifrost's plugin ABI gives exactly the hooks needed:

  • PostLLMHook — the response, after the provider call. Where the audit is born.
  • HTTPTransportPreHook — every HTTP request to the gateway. Where the audit API lives.

Because HTTPTransportPreHook runs after Bifrost's auth, the plugin's admin routes inherit virtual-key auth for free. There is no second auth system to get wrong.


2. Architecture

                         ┌──────────────────────────────────────────────┐
  client ──POST─────────▶│            Bifrost gateway (:8080)            │
  /v1/chat/completions   │                                              │
                         │  HTTPTransportPreHook ──▶ /audit, /stats, …  │
                         │        │                       (auth'd by VK) │
                         │        ▼                                     │
                         │  provider call (OpenAI / Anthropic / X / …)  │
                         │        │                                     │
                         │  PostLLMHook                                 │
                         └────────┬─────────────────────────────────────┘
                                  │ 1. extract claims (≤8)
                                  │ 2. correct in flight?  ◀── ledger read (0 credits)
                                  │ 3. dedupe by request/content
                                  ▼
                        ┌──────────────────────┐
                        │  auditor (2 workers) │
                        │  per claim:          │
                        │   Tavily search  ────┼──▶ api.tavily.com (1–2 credits)
                        │   fetch + quote      │
                        │   stance: +1 / −1 / 0│
                        │   vote = auth·e^(−age/τ)
                        └──────────┬───────────┘
                                   ▼
                        ┌──────────────────────┐
                        │ SQLite ledger        │
                        │ audits · claims ·    │
                        │ evidence · metrics   │
                        └──────────┬───────────┘
                                   ▼
                   GET /audit/{id}.html  (flat paper report)
                   GET /stats            (corrections, grounded)
Enter fullscreen mode Exit fullscreen mode

Three modes sit on one engine:

Mode When Cost What the user sees
shadow after the answer Tavily credits nothing; the ledger only
correct inside PostLLMHook 0 credits (ledger read) dated correction block appended
brief in HTTPTransportPreHook 1 Tavily search answer born current

The cost asymmetry is the point. Detection needs the web. Enforcement does not — it needs the memory of what was already detected.


3. Claim extraction

The extractor pulls sentences that make checkable assertions about the world and classifies them:

const (
    Pricing     Category = "pricing"
    APIVersion  Category = "api_version"
    Leadership  Category = "leadership"
    Regulation  Category = "regulation"
    Limits      Category = "limits"
)
Enter fullscreen mode Exit fullscreen mode

Categories are not cosmetic. Each one carries a confidence half-life, τ, that decides how fast evidence decays:

func Tau(c extract.Category) float64 {
    switch c {
    case extract.Pricing:
        // A five-week-old vendor price page is still current. At tau=30 a 39-day-old
        // page scores 0.27 and fails MinVote, which reports a plainly stale price as
        // UNVERIFIABLE — abstention where the evidence is good enough to decide is a
        // worse error than a slightly generous half-life.
        return 120
    case extract.Limits:
        return 180
    case extract.APIVersion:
        return 90
    case extract.Leadership:
        return 365
    case extract.Regulation:
        return 730
    }
    return 180
}
Enter fullscreen mode Exit fullscreen mode

A statute moving on a decade timescale and a price sheet moving on a quarterly one cannot share a decay constant. That table is the whole domain model, and it is the difference between a detector that argues and one that abstains.


4. The verdict: authority × recency, not a vote tally

Most "is this stale?" implementations count sources. Counting is wrong: ten 2023 blog posts should not outvote one vendor pricing page published last month.

So each piece of evidence casts a weighted vote:

// Recency decays from 1.0 at the publish/fetch date to ~0.37 one tau later.
func Recency(e Evidence, cat extract.Category, now time.Time) float64 {
    date := e.Published
    if date.IsZero() {
        date = e.Fetched
    }
    if date.IsZero() || date.After(now) {
        return 1
    }
    ageDays := now.Sub(date).Hours() / 24
    return math.Exp(-ageDays / Tau(cat))
}

// Vote is the authority-weighted recency used for every decision.
func Vote(e Evidence, cat extract.Category, now time.Time) float64 {
    if e.Stance == 0 {
        return 0
    }
    return clamp01(e.Authority) * Recency(e, cat, now)
}
Enter fullscreen mode Exit fullscreen mode

Decisions follow from that single scale:

  • bestCon >= MinVote (0.35) and bestCon >= bestSup → CONTRADICTED
  • otherwise bestSup >= MinVote → SUPPORTED
  • otherwise → UNVERIFIABLE (abstain rather than guess)

And the freshness score is not a vibe — it's recomputable by hand from the ledger:

f := 100 * (1 - math.Min(1, bestCon))
Enter fullscreen mode Exit fullscreen mode

For the recorded demo: the vendor page (acme.com/pricing, authority 1.0, published 2026-10-02) says $35 and contradicts; a 2024 Medium post (authority 0.4) supports the old $20. The contradiction's vote wins by an order of magnitude, and the score is 28/100 — recent enough to be decisive, old enough that the claim it contradicts is plainly stale.


5. The pitfall that produces false accusations

The first live run against anthropic.com flagged $2 / million input tokens as CONTRADICTED. The page did say $15 / million input tokens — for a different model, in a multi-model table.

Naive comparison of "a number appeared near the claim" turns every pricing table into a contradiction factory. The fix gates comparison on three cheap invariants before any number is compared:

  1. Same subject. The claim's subject tokens (the product or model it names) must overlap the evidence line's subject (subjectOverlap, sameSubject).
  2. Same model version. Claude Sonnet 5 ≠ Claude Sonnet 4.
  3. Same price kind. per million input tokens ≠ per million output tokens ≠ per month (priceKind, taggedPrices).

And the scan is line-first: candidate evidence is the single line containing the number, not the whole page. After the fix, the same page and the same single Tavily search produce $15/M input → CONTRADICTED and $2/M input → SUPPORTED. There's a regression test for exactly that pair.


6. Enforcement without spending credits: in-flight correction

response_mode: "correct" rewrites the answer before the caller sees it. The correction is drawn entirely from the ledger — the plugin's own memory of contradictions it has already established — so it costs zero Tavily credits.

claims := extract.Claims(text, p.cfg.MaxClaims)

// In-flight correction runs before the audit is enqueued, and only ever reads the local
// ledger. Order matters: the audit below must never see its own correction block as a
// new claim, or every corrected answer would spawn fresh work for text the plugin wrote
// itself.
if !streaming && p.cfg.ResponseMode == "correct" {
    p.correctInFlight(resp.ChatResponse, claims)
}
Enter fullscreen mode Exit fullscreen mode

Two details earn their comments:

  • Order. Correct before enqueue, and strip the marker when reading text back (stripCorrection). Otherwise the block the plugin appended becomes a claim on the next audit and the plugin audits itself forever.
  • Ledger match. normalizeClaim collapses punctuation and casing so the same fact expressed twice hits the same row; priors are capped by correction_max_age_hours (24h default) so a correction never cites evidence the operator considers expired.

The output is appended under a fixed marker, three lines max by default:

---
**Freshness check (FreshGauge).**
- Correction: "Acme Pro costs $20 per month" is stale. https://acme.com/pricing says
  "Pro — $35 per user / month" (authority 1.00, published 2026-10-02).
Enter fullscreen mode Exit fullscreen mode

/stats exposes corrections and grounded_requests next to audit totals, so "the gateway changed answers" is a measurable event rather than a claim in a README.


7. Prevention: a dated brief before the call

grounding_mode: "brief" runs in HTTPTransportPreHook, only for freshness-sensitive questions (looksFreshnessSensitive), inside a time box (grounding_timeout_ms), and injects a short block of dated evidence into the request:

Current evidence (verified 2026-11-21):
- acme.com/pricing (2026-10-02): "Pro — $35 per user / month"
Enter fullscreen mode Exit fullscreen mode

The model answers from that instead of from its weights. One search, opt-in, and it shows up on /stats as grounded_requests.


8. Streaming is not a normal completion

Stream chunks carry a Delta, not a Message. Reaching for Choices[0].Message dereferences a nil embedded pointer — inside a hook, that surfaces as a recovered panic in the host log and a silently missing audit, which is the worst failure mode: it looks like it works.

The hook detects the stream case and assembles deltas per request:

switch {
case c0.ChatNonStreamResponseChoice != nil:
    text = stripCorrection(messageText(resp.ChatResponse))
case c0.ChatStreamResponseChoice != nil:
    streaming = true
    text = p.streamText(ctx, c0)   // buffers deltas until finish_reason
}
Enter fullscreen mode Exit fullscreen mode

Streams are audited once, at finish, and never mutated — there is no message object to amend, so correct is non-streaming by design. Buffers are reaped on completion, and there's a test that feeds a synthetic SSE sequence through the hook to keep it that way.

Deduplication also needs care: the host does not always put a request ID on the context, so the fallback is content-<hash>-<minute>. Two failover attempts producing identical prose collapse into one audit; a genuine repeat a day later gets a fresh one with fresh evidence dates.


9. The ABI is the hard part

Bifrost plugins are Go .so files loaded into the host process. That imposes an exact-match ABI: same Go toolchain and an identical module graph, replace directives included. A one-character drift in a transitive dependency and the host refuses to load the plugin — with an error that does not name the offending package.

What worked:

  • Pin the toolchain (go1.27.3) and Bifrost (core v1.15.2, transports v2.5.1).
  • Mirror the host's replace for mcp-go exactly: replace github.com/mark3labs/mcp-go => github.com/maximhq/mcp-go v0.0.0-20261102181017-b7da3bc145a1
  • Build both from the same module cache, then check the .so loads before touching logic.

Runtime config lives in <app-dir>/config.json; the plugin reads it in Init(raw any) and never breaks a completion when its own storage fails:

auditID, cerr := p.st.CreateAudit(model, requestID, claims)
if cerr != nil {
    return resp, bifrostErr, nil // never break a completion because the ledger failed
}
Enter fullscreen mode Exit fullscreen mode

A freshness check that takes down the gateway is worse than no freshness check.


10. The audit API

Mounted through HTTPTransportPreHook, so it inherits gateway auth:

GET  /health
GET  /stats                 # audits, claims, contradictions, corrections, grounded
GET  /audits                # recent list
GET  /audit/{id}            # JSON
GET  /audit/{id}.html       # flat paper report, printable
POST /audit                 # { "text": "…" } → { "audit_id": "…", "status": "pending" }
GET  /expiring              # corrections about to age out of the ledger window
Enter fullscreen mode Exit fullscreen mode
curl -s localhost:8091/audit -d '{"text":"Acme Pro costs $20 per month."}' | jq -r .audit_id
# → a_58d9ce122048 ; pending is async, poll /audit/{id}
Enter fullscreen mode Exit fullscreen mode

The HTML report is deliberately unglamorous: paper background, hairline rules, one block per claim with the decisive evidence quote, its authority, its publish date, and the arithmetic that produced the score.


11. Benchmarking honesty

A detector is only as good as its labeled set, and the temptation is to label after seeing predictions. The bench avoids that:

  • bench/claims.csv — one claim per row with is_true_now, label_url, label_quote, label_date, adversarial.
  • Labels are established by opening the primary source and copying the decisive string, not from memory or from the model's answer.
  • bench/run.py posts each claim to POST /audit against live Tavily; bench/score.py compares predicted to label and reports accuracy plus per-verdict precision/recall.
  • Adversarial rows (correct-but-differently-worded, and stale-but-official-looking) are kept in and scored, not filtered out.

The interesting failure mode is not "wrong verdict on a stale claim" — it's "confident verdict on a claim whose subject the search never actually resolved." Abstention is a feature, and the report says UNVERIFIABLE rather than inventing certainty.


12. What to steal

  1. Put the check where all traffic already flows. A gateway plugin beats an SDK wrapper the moment there's a second provider or a stream.
  2. Weight evidence, don't count it. authority × e^(−age/τ) per category, with an explicit abstain band.
  3. Separate detection cost from enforcement cost. Detect once with the web; enforce forever from the ledger at zero marginal cost.
  4. Gate numeric comparison on subject, version, and unit before comparing values, or your detector will accuse correct answers.
  5. Never let the freshness plane break the data plane. Every storage/search error degrades to "no audit," never to "no completion."

The whole thing is a few thousand lines of Go: extractor, scorer, Tavily client, SQLite ledger, auditor workers, an HTML report, and two hooks.

Code & more: https://www.dailybuild.xyz/project/321-fresh-guage

Top comments (0)