DEV Community

Cover image for Stop Serving Raw Cosine Scores: Explainable RAG Confidence Scoring at Query Time
Eric Tetzlaff
Eric Tetzlaff

Posted on

Stop Serving Raw Cosine Scores: Explainable RAG Confidence Scoring at Query Time

#llm #rag #ai #typescript

The gap nobody talks about

You've got a RAG pipeline. It retrieves documents, generates an answer, and hands it to your users. Somewhere between "retrieved 0.84 cosine similarity" and "show answer in UI" you've made a judgment call — usually implicit — about whether that answer is trustworthy.

The problem is:

  • Cosine similarity measures proximity, not quality. 0.84 doesn't mean the answer is correct, grounded, or complete.
  • LLM self-confidence is uncalibrated. "I'm fairly confident..." is invisible to your API, your logs, and your UI.
  • There's no standard for expressing RAG answer quality in a way that's auditable, machine-readable, and actionable at runtime.

The gap between retrieval and trust has no standard fill. Until now (hopefully).

transparent-confidence

transparent-confidence is a zero-dependency TypeScript library that computes a structured 0–100 scorecard for any RAG answer at query time — no additional model calls, no separate infrastructure.

npm install transparent-confidence
Enter fullscreen mode Exit fullscreen mode

Try it in your browser — no install: StackBlitz playground

It takes the signals your pipeline already produces — retrieval scores, document metadata, a support classification — and returns a typed scorecard with per-dimension breakdowns, machine-readable warnings, and a recommended action.

import { computeConfidence } from 'transparent-confidence';

const scorecard = computeConfidence({
  supportLevel: 'high',
  hasConflict: false,
  citationCount: 3,
  candidates: [
    { retrievalScores: { semantic: 0.88, keyword: 0.72 }, combinedScore: 0.88, documentId: 'doc-001' },
    { retrievalScores: { semantic: 0.85, keyword: 0.68 }, combinedScore: 0.85, documentId: 'doc-002' },
    { retrievalScores: { semantic: 0.82, keyword: 0.65 }, combinedScore: 0.82, documentId: 'doc-003' },
  ],
});

console.log(scorecard.total);             // 100
console.log(scorecard.label);             // 'Strong'
console.log(scorecard.recommendedAction); // 'answer'
Enter fullscreen mode Exit fullscreen mode

How it works: 3 core + 5 optional dimensions

The score is built from three core dimensions (always active) and up to five optional extensions. Raw points are summed across active dimensions and normalized to 0–100.

Core (always on)

1. Answer Grounding (max 30 pts)

Scores how well the LLM answer is grounded in source documents.

Starts from supportLevel ('high' / 'medium' / 'low'), then applies penalties and ceilings:

  • queryComplexity sets a ceiling — a multi-hop question gets a lower max than a direct lookup
  • faithfulnessScore (from RAGAs or similar) applies a modifier: if the model hallucinated, grounding drops even when sources are strong
  • hasConflict deducts 5 pts
  • citationCount adds up to 2 bonus pts
  • Invalid citations deduct pts

2. Retrieval Confidence (max 25 pts)

Three sub-signals:

  • Method Agreement — how many retrieval methods (semantic, keyword, rerank) confirm each candidate
  • Score Magnitude — average combinedScore of top-K candidates
  • Source Diversity — unique documentId values retrieved; rewards casting a wide net

3. Evidence Consistency (max 10 pts)

Score stability (standard deviation of combinedScore across candidates) plus explicit conflict signal. Omitting hasConflict generates a missing-conflict-signal warning and scores conservatively — silence is not treated as agreement.

Optional extensions

4. Answer Relevance (max 15 pts) — activates when you pass answerRelevanceScore. Scores whether the answer addresses the user's question, independent of grounding. A grounded answer can still be off-topic.

5. Source Authority (max 20 pts) — for document hierarchies. Define tiers with keywords, score by weighted average across candidates. Useful for legal, compliance, governance, policy domains.

6. Corpus Completeness (max 15 pts) — surfaces the risk that a correct answer exists but the documents needed to find it haven't been uploaded. Counts document types present vs. expected.

7. Document Freshness (max 15 pts) — configurable age window with median/oldest/newest aggregation modes. Compliance-oriented pipelines can use 'oldest' for a conservative read.

8. Index Integrity (max 15 pts, Tier 2) — scores index operational readiness: embedding model version match, stale-indexed-document ratio, failed ingestion count, ACL filter confirmation, deleted-source leakage. Opt-in; inactive unless config.indexIntegrity is present. This is the dimension that catches "the index was built with a different embedding model than the one serving queries" before your users do.

The output shape 

{
  "total": 74,
  "label": "Moderate",
  "labelColor": "amber",
  "recommendedAction": "review",
  "actionReason": "Warning missing-conflict-signal triggers review policy.",
  "tier1": { "score": 80, "label": "Moderate", "color": "amber" },
  "tier2": { "score": 60, "label": "Good", "color": "amber" },
  "dimensions": {
    "grounding":   { "raw": 21, "max": 30, "normalized": 70, "explanation": "..." },
    "retrieval":   { "raw": 22, "max": 25, "normalized": 88, "explanation": "..." },
    "consistency": { "raw":  6, "max": 10, "normalized": 60, "explanation": "..." },
    "authority":   { "raw": 15, "max": 20, "normalized": 75, "explanation": "..." },
    "freshness":   { "raw": 10, "max": 15, "normalized": 67, "explanation": "..." }
  },
  "meta": {
    "algorithmVersion": "0.3.0",
    "schemaVersion": "0.3",
    "rawTotal": 74,
    "maxPossible": 100,
    "activeDimensions": ["grounding", "retrieval", "consistency", "authority", "freshness"],
    "warnings": [
      { "code": "missing-conflict-signal", "severity": "warn", "message": "..." }
    ],
    "missingSignals": ["conflictSignal"],
    "weights": { "grounding": 30, "retrieval": 25, "consistency": 10, "authority": 20, "freshness": 15 }
  }
}
Enter fullscreen mode Exit fullscreen mode

Every dimension also returns a breakdown object:

{
  "components":  { "supportBase": 30 },
  "adjustments": { "complexityCeiling": -9, "citationBonus": 2 },
  "uncappedRaw": 23,
  "raw": 21
}
Enter fullscreen mode Exit fullscreen mode

breakdown.raw === DimensionScore.raw is enforced by the test suite. Every point on the scorecard has a reason attached to it.

The runtime gating pattern

This is the part worth building toward:

const scorecard = computeConfidence(inputs, config);

if (scorecard.recommendedAction === 'abstain') {
  return {
    answer: null,
    reason: scorecard.actionReason,
    confidence: scorecard.total,
  };
}

if (scorecard.recommendedAction === 'review') {
  return {
    answer,
    reviewRequired: true,
    confidence: scorecard.total,
    warnings: scorecard.meta.warnings.map(w => w.code),
  };
}

return { answer, confidence: scorecard.total };
Enter fullscreen mode Exit fullscreen mode

The default policy runs an 8-rule cascade: explicit documentsSilent → abstain-on-warning codes → score below 40 → Tier 1 floor → review-on-warning codes → score above 65 → review band → fallback. Every rule is configurable.

The v0.3 production toolkit

v0.3 hardened the scorer for production use without adding runtime dependencies or model calls:

Production preset — preset: 'production-v0.3' requires answerRelevanceScore, a conflict signal, and either faithfulnessScore or claimSupport. When any are missing, the scorecard carries required-signal-missing warnings and the action is forced to review. No silent degradation.

Signal policy — declare which signals are required and what happens when they're absent:

computeConfidence(inputs, {
  signalPolicy: {
    require: ['answerRelevanceScore', 'conflictSignal', 'citationCoverageScore'],
    reviewWhenMissing: ['answerRelevanceScore', 'conflictSignal'],
    abstainWhenMissing: ['citationCoverageScore'],
    minCitationCoverageScore: 0.75,
    maxInvalidCitationCount: 0,
  },
});
Enter fullscreen mode Exit fullscreen mode

Calibration utilities — analyzeCalibration(samples) takes historical score/outcome pairs and returns empirical positive rates per score band plus recommended answerAt / reviewAt / abstainBelow thresholds. It tunes the action gate against your own data; it does not change the scoring algorithm.

Evaluator signal bridge — fromRagasLike / fromDeepEvalLike / fromTruLensLike / fromCustomJudge map offline-eval output into scoring inputs. Plain-object adapters; no evaluator SDK is imported.

import { fromRagasLike, mergeEvaluationSignals } from 'transparent-confidence';

const signals = fromRagasLike(ragasResult);
const { inputs: enriched } = mergeEvaluationSignals(inputs, signals);
const scorecard = computeConfidence(enriched);
Enter fullscreen mode Exit fullscreen mode

Integrating with your retriever

Candidate[] maps directly to what most retrievers return.

pgvector / Supabase:

// SELECT *, 1 - (embedding <=> $query_embedding) AS score FROM documents
const candidates = rows.map(row => ({
  retrievalScores: { semantic: row.score },
  combinedScore:   row.score,
  documentId:      row.id,
  documentType:    row.document_type,
  lastUpdated:     row.updated_at ? new Date(row.updated_at) : undefined,
}));
Enter fullscreen mode Exit fullscreen mode

LangChain / LlamaIndex:

const candidates = retrievedDocs.map(doc => ({
  retrievalScores: {
    semantic: doc.metadata.score ?? doc.score,
    keyword:  doc.metadata.bm25Score ?? 0,
  },
  combinedScore:  doc.metadata.score ?? doc.score,
  documentId:     doc.metadata.source ?? doc.id,
  documentType:   doc.metadata.documentType,
  lastUpdated:    doc.metadata.lastUpdated
                    ? new Date(doc.metadata.lastUpdated)
                    : undefined,
}));
Enter fullscreen mode Exit fullscreen mode

For single-vector pipelines (pgvector without hybrid retrieval), set minConfirmedMethods: 1 — otherwise all candidates fail the default "2+ methods" check and method agreement scores at its lowest band.

computeConfidence(inputs, {
  retrieval: { minConfirmedMethods: 1 },
});
Enter fullscreen mode Exit fullscreen mode

vs. RAGAs / TruLens / DeepEval

These are evaluation frameworks — they run asynchronously, usually in a separate eval pipeline, and call LLMs to judge answer quality. That's the right tool for offline benchmarking and fine-tuned model evaluation.

transparent-confidence runs inline at query time using signals your pipeline already has. Different use case — runtime gating vs. offline eval. They complement each other. The README comparison table:

transparent-confidence RAGAs TruLens DeepEval
Runs at query time ✅ ⚠️ async ⚠️ async ⚠️ async
Requires LLM calls ✅ none ❌ yes ❌ yes ❌ yes
Zero dependencies ✅ ❌ ❌ ❌
TypeScript-native ✅ ❌ ❌ partial
Authority / corpus / freshness ✅ ❌ ❌ ❌

Observability

Log the full scorecard as structured JSON. Every observable dimension, warning code, and action decision is in one flat object:

logger.info('rag_confidence', {
  timestamp:         new Date().toISOString(),
  algorithmVersion:  scorecard.meta.algorithmVersion,
  total:             scorecard.total,
  recommendedAction: scorecard.recommendedAction,
  warningCodes:      scorecard.meta.warnings.map(w => w.code),
  dimensions: Object.fromEntries(
    scorecard.meta.activeDimensions.map(name => [
      name,
      {
        raw:        scorecard.dimensions[name]?.raw,
        normalized: scorecard.dimensions[name]?.normalized,
      },
    ])
  ),
});
Enter fullscreen mode Exit fullscreen mode

Plug into OpenTelemetry, Datadog, Splunk — whatever you're already using for LLM observability.

Where it came from

I built transparent-confidence while building BoardPath, an HOA governance intelligence platform. HOA documents have rigid hierarchies — CC&Rs override Bylaws override Rules & Regs — and the question "which document is authoritative?" turns out to be a real engineering problem, not a hypothetical one.

The Source Authority extension exists because I needed it. The Corpus Completeness extension exists because I kept watching queries fail silently when a required document type wasn't uploaded yet. The missing-conflict-signal warning exists because amendment conflicts are common and dangerous to miss.

v0.3.1 ships 3 core + 5 optional dimensions, 412 passing tests across 19 files, a configurable actionPolicy and signalPolicy, calibration utilities, and an evaluator bridge — still with zero runtime dependencies.

Links

Feedback, issues, and PRs welcome. Next milestone is v1.0 — locking the public API surface. The Python port is a candidate on the roadmap.

Top comments (0)