DEV Community: Softronic

Corrective RAG for billing: the bug is not retrieval, it's the model narrating correct numbers wrong

Softronic — Thu, 30 Jul 2026 16:49:38 +0000

Most RAG demos are graded by an audience that cannot check the answer. Ask a docs bot something, get a fluent paragraph back, nobody in the room knows whether sentence three is invented. The demo lands.

Billing is not like that. When a subscriber asks "why is my bill higher this month" they are holding the invoice. They will check.

We built a WhatsApp assistant for an ISP that answers plan and billing questions. This post is about the two things that turned out to matter, neither of which is "we used CRAG":

A document corpus and a billing API are both called "retrieval" and they fail in completely different ways.
The failure that actually bites is not bad retrieval. It is the model taking correct numbers and narrating them wrong.

Code is from the real system. Comments translated from Spanish, otherwise untouched.

The pipeline

inbound message
   ↓
state gate ──────────────► human agent active? bot writes nothing
   ↓
hybrid retrieval
   ├─ pgvector cosine ─┐
   └─ Postgres FTS ────┴─► RRF fusion ─► CRAG prune
   ↓
capability call (billing API) ─► circuit breaker
   ↓
no context AND no tool data? ──────────► escalate (before spending a token)
   ↓
LLM, structured output
   ↓
grounding verifier ────────► not grounded? ─► escalate
   ↓
humanizer ─────────────────► forbidden phrase? ─► escalate
   ↓
send

Every arrow pointing right is a refusal path. That is the design.

1. Fusing two rankings that aren't comparable

Vector similarity and Postgres ts_rank live on different scales. Adding them is meaningless. Reciprocal Rank Fusion only looks at position, so it fuses them without pretending the scores are commensurable:

DEFAULT_RRF_K = 60
"""Smoothing constant. Damps the weight of the top positions so a single
ranking cannot dominate the fusion. 60 is the value from the original paper."""


def reciprocal_rank_fusion(
    rankings: Sequence[Sequence[str]], *, k: int = DEFAULT_RRF_K
) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for position, id_ in enumerate(ranking, start=1):
            scores[id_] = scores.get(id_, 0.0) + 1.0 / (k + position)
    return scores

Ties break on the id, deterministically. A result that reorders between identical runs makes tests flaky and debugging impossible.

Why full-text at all, when you have embeddings? Because account numbers, tax IDs and national ID numbers have no semantic meaning. An embedding cannot score digits. FTS is the only path by which those are ever retrieved.

2. CRAG's refinement step, without the second model

CRAG (Yan et al., 2024) evaluates retrieval relevance and drops the noise before generating. The paper uses a separate lightweight evaluator model.

We skipped the evaluator. The retriever already computed cosine similarity and knows whether full-text hit — running a per-chunk model call every turn is the expensive part of CRAG and the least useful part when you already have scores.

The problem it solves is concrete: top_k asks for 6 chunks and RRF always returns 6, whether or not the sixth makes sense. Fusion orders, it does not filter. That filler enters the prompt at full size, burns tokens every turn, and dilutes the good evidence.

DEFAULT_MARGIN = 0.12
"""How far below the best chunk we still tolerate.

Deliberately a *relative* margin. A fixed threshold behaves badly at both ends:
on a well-answered query (best = 0.80) it lets a 0.40 through as "good" when
next to the best it's noise, and on a weak query (best = 0.42) it would prune
everything. The margin adapts: each chunk is compared to the best found on
*this* turn."""

DEFAULT_MIN_CHUNKS = 2


def prune_irrelevant[Chunk: _Scorable](
    chunks: Sequence[Chunk],
    *,
    margin: float = DEFAULT_MARGIN,
    min_chunks: int = DEFAULT_MIN_CHUNKS,
) -> list[Chunk]:
    if len(chunks) <= min_chunks:
        return list(chunks)

    best = max((c.vector_score or 0.0 for c in chunks), default=0.0)
    cutoff = best - margin

    kept = {i for i, c in enumerate(chunks) if _is_relevant(c, cutoff)}

    # Chunks arrive ordered best-to-worst, so backfilling from index 0
    # adds precisely the best of the discarded ones.
    for i in range(len(chunks)):
        if len(kept) >= min_chunks:
            break
        kept.add(i)

    return [c for i, c in enumerate(chunks) if i in kept]


def _is_relevant(chunk: _Scorable, cutoff: float) -> bool:
    """A full-text hit always survives; everything else competes on similarity.

    Full-text immunity is not a convenient exception: `plainto_tsquery` requires
    the terms to actually appear in the chunk, so the hit is itself proof of
    relevance. It is also the only path by which account numbers and tax IDs are
    retrieved — digits carry no semantic meaning, so they arrive here with a low
    cosine that would otherwise condemn them.
    """
    if chunk.matched_by_text:
        return True
    return (chunk.vector_score or 0.0) >= cutoff

Note the bias, because it is the opposite of the verifier below: when in doubt, keep. Over-pruning would delete the one line that answered the question and escalate a case the bot could have solved — worse than wasting a few tokens.

3. Tool results are first-class evidence

The real balance is not in any document. It comes from the billing API, and it has to count as citable evidence or the verifier in step 4 would reject the correct answer:

@dataclass(frozen=True, slots=True)
class ToolResult:
    capability: str
    ok: bool
    evidence_text: str | None = None
    """Text ready to pass to the LLM as evidence; None if there was no data."""

    data: dict[str, Any] | None = None
    error: str | None = None
    should_escalate: bool = False
    """`True` if the failure is infrastructure (billing API down) → escalate.
    A "customer not found" does NOT escalate: it's data that simply isn't there."""

That last field is a distinction worth stealing. "The API is down" and "this customer doesn't exist" both produce no data, and they are not the same event. One is an outage, the other is an answer.

except CrmBusinessError as exc:
    # The API works and said "doesn't exist": missing data, not a hard escalation.
    return ToolResult(capability=capability.name, ok=False,
                      error=exc.message, should_escalate=False)
except CRMError as exc:
    # Infrastructure: timeout, 5xx, open circuit → escalate.
    return ToolResult(capability=capability.name, ok=False,
                      error=str(exc), should_escalate=True)

When the API really is down, retrying makes it worse — every inbound message burns 15s of timeout × 3 attempts, the worker queue backs up, and the API gets a stampede exactly while it is trying to recover. So there is a circuit breaker, and for the bot "API down" simply translates to escalate fast:

def record_failure(self) -> None:
    """A failure counts; in HALF_OPEN it reopens immediately.

    Reopening on the first failure in HALF_OPEN is deliberate: if the probe
    request fails, the API is still bad and there is no point spending the
    rest of the threshold to confirm it.
    """
    self._failures += 1
    if self._state is CircuitState.HALF_OPEN or self._failures >= self._threshold:
        self._state = CircuitState.OPEN
        self._opened_at = self._now()

The clock is injected so tests can verify reopening without actually sleeping. A test that does sleep(60) gets deleted or marked slow, and the most delicate part of the system loses its coverage with it.

4. The part that actually matters

Everything above is retrieval hygiene. This is the piece I would push hardest on in a review of anyone else's billing assistant, because it survives every model upgrade.

The prompt asks the model not to invent. This verifies it:

class GroundingVerifier:
    def verify(
        self, *, answer: str, used_chunk_ids: list[str],
        retrieved_chunk_ids: list[str], evidence_texts: list[str],
        tool_result_texts: list[str] | None = None,
    ) -> GroundingResult:

Two checks:

Every citation points at real evidence. A model citing a chunk id that was never retrieved is inventing its own backing.
No number appears without support. Every numeric token in the answer — amounts, account numbers, phone numbers, dates — must appear in the retrieved evidence or in a tool result.

The bias is deliberate and the inverse of the pruner: when in doubt, reject. A rejected answer escalates to a human, which is annoying but safe. A hallucination that ships is expensive and costs trust. That module is held to 100% test coverage.

The bug that makes this non-trivial

Naive version: strip non-digits from the number, check if those digits appear in the evidence.

That is wrong, and here is the counterexample that killed it:

Evidence (from the billing API):  45.00
Model says:                       4.500
Digits of both:                   "4500"  ✓ passes

Two different amounts. Same digits. In Venezuelan notation 4.500 reads as four thousand five hundred; 45.00 is forty-five. Digit equality green-lights a hallucination that changes the magnitude of someone's bill by two orders of magnitude.

So amounts are compared by canonical value, not by digits:

_AMOUNT = re.compile(r"^\d{1,3}(?:[.,\s]\d{3})*[.,]\d{1,2}$|^\d+[.,]\d{1,2}$")


def _amount_value(token: str) -> str | None:
    """Normalise a currency token to canonical `integer.decimals`.

    Returns None if the token is not amount-shaped. This is what stops `45.00`
    and `4.500` being considered equal.

    Treats the LAST `.`/`,` as the decimal separator and the rest as thousands,
    which is the Venezuelan convention (`1.234,56`) and also the English one
    (`1,234.56`).
    """
    if not _AMOUNT.match(token):
        return None
    clean = token.replace(" ", "")
    cut = max(clean.rfind("."), clean.rfind(","))
    integer = re.sub(r"\D", "", clean[:cut])
    decimals = clean[cut + 1:]
    return f"{int(integer or 0)}.{decimals}"

Three rules, strictest first:

def _number_supported(token, evidence, evidence_digits, evidence_amounts) -> bool:
    # 1. Literal, with digit boundaries. `12345678` is NOT accepted just because
    #    it appears inside a longer `123456789`.
    if re.search(r"(?<!\d)" + re.escape(token) + r"(?!\d)", evidence):
        return True

    # 2. Amount value. `45,00` (Venezuelan decimal comma) matches `45.00` from
    #    the API, but `4.500` does NOT match `45.00` — different values.
    value = _amount_value(token)
    if value is not None:
        return value in evidence_amounts

    # 3. Digit equality — ONLY for identifiers (phones, accounts, tax IDs:
    #    7+ digits), where separators are cosmetic and don't change the value.
    if len(_digits_only(token)) >= _MIN_IDENTIFIER_DIGITS:
        return _digits_only(token) in evidence_digits
    return False

Rule 3 is deliberately narrow. A short number that matched neither literally nor as an amount is rejected — 4.500 must not pass just because it shares digits with 45.00.

The refusal path is the product

The instinct is to treat escalations as the failure rate and drive them to zero. That optimises the wrong thing. Every escalation you remove by loosening the verifier becomes a confident answer about a bill, and some of those are wrong in a way the customer notices.

Each reason carries a summary the human agent reads, so they don't re-read the whole thread:

_AGENT_SUMMARY: dict[EscalationReason, str] = {
    EscalationReason.NO_CONTEXT:
        "The customer asked something that is not in the knowledge base.",
    EscalationReason.UNGROUNDED_OUTPUT:
        "The bot was about to give an unsupported figure; it was stopped for safety.",
    EscalationReason.USER_REQUEST:
        "The customer asked to speak to a person.",
    EscalationReason.TOOL_ERROR:
        "Could not query the billing system (possible outage).",
    EscalationReason.CAPABILITY_MISSING:
        "The customer wants data the bot cannot query yet.",
}

USER_REQUEST fires unconditionally. There is always pressure to have the bot try once more before releasing the conversation — it improves your containment metric. It also means the subscriber who typed "I want to talk to a human" gets a paragraph they didn't ask for. On a channel people use to talk to people, ignoring that request teaches them the bot is an obstacle, not a front door.

What we'd tell you before you build one

Route before you retrieve. "What does this line item mean" is a document question. "Why is my bill $14 higher" needs account state. Answering the second from the rate card is wrong for every subscriber whose delta came from somewhere else.
Numbers from a system of record should reach the user as those numbers. The model's job is the sentence around the figure, not the figure. Any delta, total or date comparison is a computation, and computations belong in testable code.
Instrument refusal rate as a dial, not a defect. Tune it against complaints.
The metric that matters is verified-wrong answers, and it comes from support, not from your eval harness.

One honest note on scope: this is one system for one operator, described as architecture rather than as a benchmark. We are not going to show you a deflection-rate chart from a sample of one.

Also worth stating plainly: the billing API integration is contract-first. There is a documented HTTP contract and a simulator implementing it; the production system on the other side is still being built. Everything above — capability resolution, circuit breaker, tool-results-as-evidence — runs against that contract. We think that is the right order (the contract is the thing both sides agree on), but you should know it when you read the code.

Longer version, including why WhatsApp specifically removes every UI affordance you'd normally use to show uncertainty: Corrective RAG for Billing Questions on WhatsApp

Bun vs Node.js in 2026: Should You Actually Migrate?

Softronic — Tue, 28 Jul 2026 14:35:41 +0000

Originally published on softronic.dev.

Bun finished 2025 at the top of the JavaScript Rising Stars chart, ahead of every framework, library, and tool that year. By Q1 2026 it has hit production at a meaningful number of mid-sized engineering teams. The question on every CTO's desk: should we actually migrate, or is this another runtime that gets quietly archived next year?

We use Bun every day for tooling, scripts, and CLIs. We still run Node in production for most client servers. Here's the honest, opinionated case for and against — with the benchmarks that matter and the gotchas that bit us.

Key takeaways

Bun's speed advantage is real and largest in install and test loops, which is where developer time actually goes.
The gotchas are concentrated in native modules and less-common Node APIs — pilot before committing a production service.
The 5% of cases where Bun isn't a drop-in will cost more than the 95% where it is, if you don't plan for them.
A hybrid is legitimate: Bun for tooling and tests, Node for the runtime, until your dependency graph is clean. ## Why Bun keeps winning

Bun's pitch is simple: faster, all-in-one, drop-in for most Node code. Each part of that has substance.

Faster. Bun is built on the JavaScriptCore engine (the same one in Safari) rather than V8, plus a Zig-based runtime designed from scratch for speed. The differences are not subtle.

All-in-one. Bun is the runtime, the package manager, the bundler, the test runner, and a TypeScript transpiler in one binary. You can delete npm, tsx, jest/vitest, webpack/esbuild, and a fair amount of package.json config when you adopt it.

Drop-in (mostly). Bun implements the Node API surface — fs, path, http, child_process, npm package resolution, the works. For greenfield code it usually just runs.

The combination is genuinely productive for a category of work we'll come back to.

Worth pairing with the case for TypeScript as the default, since the two decisions usually land in the same sprint.

Real benchmarks from our own work

We re-ran the benchmarks on our own internal services in March 2026, on identical hardware (Apple M2 Pro, 16 GB) and identical workloads. These aren't synthetic microbenchmarks. These are real tasks we run dozens of times per day.

Task	Node 22 LTS	Bun 1.2	Bun speedup
`package.json` install (medium project)	18.4 s	2.1 s	8.8x
Test suite (Vitest vs `bun test`)	9.2 s	1.7 s	5.4x
TypeScript CLI script startup	380 ms	35 ms	10.8x
HTTP server, 50K req hello-world	12.4 s	4.9 s	2.5x
HTTP server, 50K req with DB + JSON parse	28.1 s	22.7 s	1.2x
Build (esbuild on Node vs `bun build`)	4.6 s	1.4 s	3.3x

Two patterns to notice:

Bun crushes Node on cold starts, install times, and test runs. This is where the all-in-one tooling and fast startup compound.
For long-running servers doing I/O against a database, the gap narrows to 20-25%. Most production workloads are bottlenecked on Postgres or Redis, not the JavaScript runtime. Bun still wins, but not enough to justify a migration on its own.

The takeaway: if your engineering productivity is bottlenecked on tooling speed (install, test, build, dev server), Bun has a strong case. If your production cost is bottlenecked on database latency, Bun's win is smaller than the chart suggests.

Production gotchas that bit us

We've shipped Bun to production for three client engagements in the last 9 months. Here's what surprised us, in roughly the order of pain.

Native modules. Bun supports N-API and most native modules, but coverage is not 100%. We hit one issue with a legacy MongoDB driver and one with a niche image-processing library. The fix in both cases was upgrading to a maintained alternative, but that's a migration cost you need to budget.

npm script semantics. Most package.json scripts run, but some assume node_modules/.bin lookup order or specific environment variable propagation that differs slightly under Bun. The fixes are easy. Finding them in a CI failure at 11 PM is not.

Monitoring and APM. Datadog, New Relic, and Sentry all officially support Bun now, but the auto-instrumentation has lagged. Some hooks (especially around async context propagation) require manual setup. Plan for a half-day per service to wire it up properly.

Cluster mode and process management. Node's cluster module is a common production pattern. Bun has its own primitives for this and they work, but the migration is not zero-effort. If you're behind PM2 or a Kubernetes deployment that handles process management, this is a non-issue. If you have homegrown cluster code, plan for it.

Long-tail npm package compatibility. Anecdotally we hit a compat issue roughly once per 50 packages. Almost always a niche package, almost always already replaced upstream. But if your codebase has a node_modules with 800 packages, statistically you will hit a few.

Memory characteristics differ. Bun and Node have different garbage collection profiles. We saw one workload where memory usage was 30% lower under Bun, and one where it was 15% higher. Profile before assuming.

The decision matrix

After running Bun in production for the last 9 months and Node for the last 12 years, here's how we decide.

Scenario	Default choice	Reasoning
New CLI tool or developer script	Bun	Startup speed, single binary, zero config
Internal tooling, dev scripts, codemods	Bun	Same
New Cloudflare Workers / edge service	Workers runtime	Edge runtime, not Node or Bun
New monolith API server	Node	Maturity, ecosystem, APM, hiring pool
Greenfield microservice (low risk)	Either	Pick what your team knows
Existing Node service, working fine	Stay on Node	No migration value unless cold-start matters
Existing Node service, slow CI	Bun for CI	Run Bun in CI/test, Node in prod = best of both
Latency-sensitive serverless function	Bun	Cold start is the win
Heavy database I/O service	Node	DB is the bottleneck; Node's maturity wins
Test runner / build tooling	Bun	Massive speedup, low risk

The pattern: Bun wins decisively for short-lived processes and tooling. Node still wins for long-lived production servers where ecosystem maturity beats raw speed. A lot of teams can get most of Bun's value by running it in CI and dev environments while leaving production on Node.

Migration tactics that actually work

If you decide to migrate, here's the order that's worked for us and our clients.

Start with CI. Switch npm install and your test runner to Bun. Zero production risk, immediate developer-experience win. This step alone saves most teams 10-30 minutes per CI run.
Move local dev next. Use Bun as the local dev server. Catch any compatibility issues against your codebase without any production exposure.
Migrate scripts and codemods. Anything in scripts/ that runs on a developer's machine. Bun's startup speed compounds across hundreds of script invocations a day.
Pilot one production service. Pick something non-critical and re-runnable. Run it on Bun in parallel with the Node version, compare metrics for two weeks.
Roll out gradually, service by service. Don't do a big-bang migration. Each service gets its own go/no-go.

A team that does step 1 alone gets 60% of the benefit with 5% of the risk. That's where most of our clients land.

What we actually run at Softronic

For internal tooling, scripts, our own design system build, and most client-facing dev environments: Bun.

For client production servers: usually Node 22 LTS. The exception is latency-sensitive serverless work, where we'll run Bun on Fly.io machines or Cloudflare Containers.

For Cloudflare Workers (which is where a lot of our 2026 work lives): the Workers runtime, which is neither Bun nor Node and has its own constraints.

This is a deliberately boring answer. We watch the Bun roadmap closely. The window where it makes sense as the default production runtime is narrowing every quarter, and there's a version — probably Bun 2.0 — where we'll start defaulting to it for greenfield production services too.

A few things we'd push back on in the hype cycle

We're broadly bullish on Bun. We also notice the same pattern with every fast-rising runtime: the chart leaders attract a wave of "you're crazy not to migrate" content that glosses over real costs. Worth pushing back on a few claims that have been making the rounds.

"Bun is a drop-in replacement for Node." Mostly yes, sometimes no. The 5% of cases where it isn't will cost you more than the 95% where it is, if you don't plan for them. Pilot before you commit.

"You can replace your whole toolchain with Bun." True for greenfield work and small teams. For larger teams, the cost of standardizing the toolchain across 30 engineers, CI pipelines, internal templates, and editor configs is real. We've seen teams half-migrate and end up worse off than if they'd picked a side and stuck with it.

"Bun's HTTP server is 4x faster than Node's, period." It's faster on hello-world benchmarks. In real workloads bottlenecked on a database, the difference shrinks substantially. Benchmark your workload, not the synthetic test that comes with the runtime.

"Bun supports all of npm." It supports most of npm. The packages where it doesn't are concentrated in the long tail and in older or niche native modules. For modern, popular packages — Express, Fastify, Hono, Prisma, Drizzle, Zod, all the framework ecosystems — it's effectively complete.

The runtime wars in JavaScript over the last decade have rewarded patience. io.js merged back into Node. Deno is still around but didn't take over the way the early hype suggested. Bun has a real shot at being different, and even if it isn't, the parts of it we're using today (the test runner, the package manager, the CLI runtime) have already paid for themselves.

When you'd want help

If your team is weighing a Bun migration or trying to figure out where in your stack it actually pays off, we can save you the discovery work. We've done the painful version on real codebases.

Pick the runtime that fits the workload, not the chart. Bun is real, useful, and worth adopting today — just not always for the part of the stack you assumed.