Corrective RAG for billing: the bug is not retrieval, it's the model narrating correct numbers wrong

Most RAG demos are graded by an audience that cannot check the answer. Ask a docs bot something, get a fluent paragraph back, nobody in the room knows whether sentence three is invented. The demo lands.

Billing is not like that. When a subscriber asks "why is my bill higher this month" they are holding the invoice. They will check.

We built a WhatsApp assistant for an ISP that answers plan and billing questions. This post is about the two things that turned out to matter, neither of which is "we used CRAG":

1. A document corpus and a billing API are both called "retrieval" and they fail in completely different ways.
2. The failure that actually bites is not bad retrieval. It is the model taking correct numbers and narrating them wrong.

Code is from the real system. Comments translated from Spanish, otherwise untouched.

The pipeline

inbound message
   ↓
state gate ──────────────► human agent active? bot writes nothing
   ↓
hybrid retrieval
   ├─ pgvector cosine ─┐
   └─ Postgres FTS ────┴─► RRF fusion ─► CRAG prune
   ↓
capability call (billing API) ─► circuit breaker
   ↓
no context AND no tool data? ──────────► escalate (before spending a token)
   ↓
LLM, structured output
   ↓
grounding verifier ────────► not grounded? ─► escalate
   ↓
humanizer ─────────────────► forbidden phrase? ─► escalate
   ↓
send

Every arrow pointing right is a refusal path. That is the design.

1. Fusing two rankings that aren't comparable

Vector similarity and Postgres ts_rank live on different scales. Adding them is meaningless. Reciprocal Rank Fusion only looks at position, so it fuses them without pretending the scores are commensurable:

DEFAULT_RRF_K = 60
"""Smoothing constant. Damps the weight of the top positions so a single
ranking cannot dominate the fusion. 60 is the value from the original paper."""


def reciprocal_rank_fusion(
    rankings: Sequence[Sequence[str]], *, k: int = DEFAULT_RRF_K
) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for position, id_ in enumerate(ranking, start=1):
            scores[id_] = scores.get(id_, 0.0) + 1.0 / (k + position)
    return scores

Ties break on the id, deterministically. A result that reorders between identical runs makes tests flaky and debugging impossible.

Why full-text at all, when you have embeddings? Because account numbers, tax IDs and national ID numbers have no semantic meaning. An embedding cannot score digits. FTS is the only path by which those are ever retrieved.

2. CRAG's refinement step, without the second model

CRAG (Yan et al., 2024) evaluates retrieval relevance and drops the noise before generating. The paper uses a separate lightweight evaluator model.

We skipped the evaluator. The retriever already computed cosine similarity and knows whether full-text hit — running a per-chunk model call every turn is the expensive part of CRAG and the least useful part when you already have scores.

The problem it solves is concrete: top_k asks for 6 chunks and RRF always returns 6, whether or not the sixth makes sense. Fusion orders, it does not filter. That filler enters the prompt at full size, burns tokens every turn, and dilutes the good evidence.

DEFAULT_MARGIN = 0.12
"""How far below the best chunk we still tolerate.

Deliberately a *relative* margin. A fixed threshold behaves badly at both ends:
on a well-answered query (best = 0.80) it lets a 0.40 through as "good" when
next to the best it's noise, and on a weak query (best = 0.42) it would prune
everything. The margin adapts: each chunk is compared to the best found on
*this* turn."""

DEFAULT_MIN_CHUNKS = 2


def prune_irrelevant[Chunk: _Scorable](
    chunks: Sequence[Chunk],
    *,
    margin: float = DEFAULT_MARGIN,
    min_chunks: int = DEFAULT_MIN_CHUNKS,
) -> list[Chunk]:
    if len(chunks) <= min_chunks:
        return list(chunks)

    best = max((c.vector_score or 0.0 for c in chunks), default=0.0)
    cutoff = best - margin

    kept = {i for i, c in enumerate(chunks) if _is_relevant(c, cutoff)}

    # Chunks arrive ordered best-to-worst, so backfilling from index 0
    # adds precisely the best of the discarded ones.
    for i in range(len(chunks)):
        if len(kept) >= min_chunks:
            break
        kept.add(i)

    return [c for i, c in enumerate(chunks) if i in kept]


def _is_relevant(chunk: _Scorable, cutoff: float) -> bool:
    """A full-text hit always survives; everything else competes on similarity.

    Full-text immunity is not a convenient exception: `plainto_tsquery` requires
    the terms to actually appear in the chunk, so the hit is itself proof of
    relevance. It is also the only path by which account numbers and tax IDs are
    retrieved — digits carry no semantic meaning, so they arrive here with a low
    cosine that would otherwise condemn them.
    """
    if chunk.matched_by_text:
        return True
    return (chunk.vector_score or 0.0) >= cutoff

Note the bias, because it is the opposite of the verifier below: when in doubt, keep. Over-pruning would delete the one line that answered the question and escalate a case the bot could have solved — worse than wasting a few tokens.

3. Tool results are first-class evidence

The real balance is not in any document. It comes from the billing API, and it has to count as citable evidence or the verifier in step 4 would reject the correct answer:

@dataclass(frozen=True, slots=True)
class ToolResult:
    capability: str
    ok: bool
    evidence_text: str | None = None
    """Text ready to pass to the LLM as evidence; None if there was no data."""

    data: dict[str, Any] | None = None
    error: str | None = None
    should_escalate: bool = False
    """`True` if the failure is infrastructure (billing API down) → escalate.
    A "customer not found" does NOT escalate: it's data that simply isn't there."""

That last field is a distinction worth stealing. "The API is down" and "this customer doesn't exist" both produce no data, and they are not the same event. One is an outage, the other is an answer.

except CrmBusinessError as exc:
    # The API works and said "doesn't exist": missing data, not a hard escalation.
    return ToolResult(capability=capability.name, ok=False,
                      error=exc.message, should_escalate=False)
except CRMError as exc:
    # Infrastructure: timeout, 5xx, open circuit → escalate.
    return ToolResult(capability=capability.name, ok=False,
                      error=str(exc), should_escalate=True)

When the API really is down, retrying makes it worse — every inbound message burns 15s of timeout × 3 attempts, the worker queue backs up, and the API gets a stampede exactly while it is trying to recover. So there is a circuit breaker, and for the bot "API down" simply translates to escalate fast:

def record_failure(self) -> None:
    """A failure counts; in HALF_OPEN it reopens immediately.

    Reopening on the first failure in HALF_OPEN is deliberate: if the probe
    request fails, the API is still bad and there is no point spending the
    rest of the threshold to confirm it.
    """
    self._failures += 1
    if self._state is CircuitState.HALF_OPEN or self._failures >= self._threshold:
        self._state = CircuitState.OPEN
        self._opened_at = self._now()

The clock is injected so tests can verify reopening without actually sleeping. A test that does sleep(60) gets deleted or marked slow, and the most delicate part of the system loses its coverage with it.

4. The part that actually matters

Everything above is retrieval hygiene. This is the piece I would push hardest on in a review of anyone else's billing assistant, because it survives every model upgrade.

The prompt asks the model not to invent. This verifies it:

class GroundingVerifier:
    def verify(
        self, *, answer: str, used_chunk_ids: list[str],
        retrieved_chunk_ids: list[str], evidence_texts: list[str],
        tool_result_texts: list[str] | None = None,
    ) -> GroundingResult:

Two checks:

1. Every citation points at real evidence. A model citing a chunk id that was never retrieved is inventing its own backing.
2. No number appears without support. Every numeric token in the answer — amounts, account numbers, phone numbers, dates — must appear in the retrieved evidence or in a tool result.

The bias is deliberate and the inverse of the pruner: when in doubt, reject. A rejected answer escalates to a human, which is annoying but safe. A hallucination that ships is expensive and costs trust. That module is held to 100% test coverage.

The bug that makes this non-trivial

Naive version: strip non-digits from the number, check if those digits appear in the evidence.

That is wrong, and here is the counterexample that killed it:

Evidence (from the billing API):  45.00
Model says:                       4.500
Digits of both:                   "4500"  ✓ passes

Two different amounts. Same digits. In Venezuelan notation 4.500 reads as four thousand five hundred; 45.00 is forty-five. Digit equality green-lights a hallucination that changes the magnitude of someone's bill by two orders of magnitude.

So amounts are compared by canonical value, not by digits:

_AMOUNT = re.compile(r"^\d{1,3}(?:[.,\s]\d{3})*[.,]\d{1,2}$|^\d+[.,]\d{1,2}$")


def _amount_value(token: str) -> str | None:
    """Normalise a currency token to canonical `integer.decimals`.

    Returns None if the token is not amount-shaped. This is what stops `45.00`
    and `4.500` being considered equal.

    Treats the LAST `.`/`,` as the decimal separator and the rest as thousands,
    which is the Venezuelan convention (`1.234,56`) and also the English one
    (`1,234.56`).
    """
    if not _AMOUNT.match(token):
        return None
    clean = token.replace(" ", "")
    cut = max(clean.rfind("."), clean.rfind(","))
    integer = re.sub(r"\D", "", clean[:cut])
    decimals = clean[cut + 1:]
    return f"{int(integer or 0)}.{decimals}"

Three rules, strictest first:

def _number_supported(token, evidence, evidence_digits, evidence_amounts) -> bool:
    # 1. Literal, with digit boundaries. `12345678` is NOT accepted just because
    #    it appears inside a longer `123456789`.
    if re.search(r"(?<!\d)" + re.escape(token) + r"(?!\d)", evidence):
        return True

    # 2. Amount value. `45,00` (Venezuelan decimal comma) matches `45.00` from
    #    the API, but `4.500` does NOT match `45.00` — different values.
    value = _amount_value(token)
    if value is not None:
        return value in evidence_amounts

    # 3. Digit equality — ONLY for identifiers (phones, accounts, tax IDs:
    #    7+ digits), where separators are cosmetic and don't change the value.
    if len(_digits_only(token)) >= _MIN_IDENTIFIER_DIGITS:
        return _digits_only(token) in evidence_digits
    return False

Rule 3 is deliberately narrow. A short number that matched neither literally nor as an amount is rejected — 4.500 must not pass just because it shares digits with 45.00.

The refusal path is the product

The instinct is to treat escalations as the failure rate and drive them to zero. That optimises the wrong thing. Every escalation you remove by loosening the verifier becomes a confident answer about a bill, and some of those are wrong in a way the customer notices.

Each reason carries a summary the human agent reads, so they don't re-read the whole thread:

_AGENT_SUMMARY: dict[EscalationReason, str] = {
    EscalationReason.NO_CONTEXT:
        "The customer asked something that is not in the knowledge base.",
    EscalationReason.UNGROUNDED_OUTPUT:
        "The bot was about to give an unsupported figure; it was stopped for safety.",
    EscalationReason.USER_REQUEST:
        "The customer asked to speak to a person.",
    EscalationReason.TOOL_ERROR:
        "Could not query the billing system (possible outage).",
    EscalationReason.CAPABILITY_MISSING:
        "The customer wants data the bot cannot query yet.",
}

USER_REQUEST fires unconditionally. There is always pressure to have the bot try once more before releasing the conversation — it improves your containment metric. It also means the subscriber who typed "I want to talk to a human" gets a paragraph they didn't ask for. On a channel people use to talk to people, ignoring that request teaches them the bot is an obstacle, not a front door.

What we'd tell you before you build one

• Route before you retrieve. "What does this line item mean" is a document question. "Why is my bill $14 higher" needs account state. Answering the second from the rate card is wrong for every subscriber whose delta came from somewhere else.
• Numbers from a system of record should reach the user as those numbers. The model's job is the sentence around the figure, not the figure. Any delta, total or date comparison is a computation, and computations belong in testable code.
• Instrument refusal rate as a dial, not a defect. Tune it against complaints.
• The metric that matters is verified-wrong answers, and it comes from support, not from your eval harness.

One honest note on scope: this is one system for one operator, described as architecture rather than as a benchmark. We are not going to show you a deflection-rate chart from a sample of one.

Also worth stating plainly: the billing API integration is contract-first. There is a documented HTTP contract and a simulator implementing it; the production system on the other side is still being built. Everything above — capability resolution, circuit breaker, tool-results-as-evidence — runs against that contract. We think that is the right order (the contract is the thing both sides agree on), but you should know it when you read the code.

Longer version, including why WhatsApp specifically removes every UI affordance you'd normally use to show uncertainty: Corrective RAG for Billing Questions on WhatsApp