Bad Retrieval Doesn't Show Up on the AI Bill. It Shows Up on the Review Queue.

Every e-commerce fraud operation runs on a brutal piece of arithmetic that rarely gets written down: the cost of a human looking at an order.

A manual review takes minutes of a trained analyst's time. Multiply by review rate, multiply by order volume, and the review queue quietly becomes one of the biggest line items in the fraud budget — bigger than the scoring model, bigger than the tooling. So anything that changes how fast an analyst resolves a case, or how many cases need a human at all, moves real money.

Retrieval quality is exactly that lever, and almost nobody prices it.

The math, with knobs you can turn

These are illustrative inputs — the point is the shape, so plug in your own:

def review_queue_economics(
    daily_orders: int = 50_000,
    review_rate: float = 0.04,        # 4% of orders flagged for manual review
    minutes_per_case: float = 6.0,    # avg analyst handling time
    analyst_cost_per_hour: float = 30.0,
    lookup_share: float = 0.35,       # cases where analyst searches past cases/SOPs
    retrieval_time_saved_min: float = 2.0,  # good retrieval: found in sec, not scrolling
):
    daily_reviews = daily_orders * review_rate
    base_cost = daily_reviews * minutes_per_case / 60 * analyst_cost_per_hour

    # Better retrieval only helps the cases that involve a lookup —
    # but for those, it removes the slowest part of the case.
    saved_min = daily_reviews * lookup_share * retrieval_time_saved_min
    savings = saved_min / 60 * analyst_cost_per_hour

    print(f"Daily reviews:            {daily_reviews:,.0f}")
    print(f"Daily review cost:        ${base_cost:,.0f}")
    print(f"Daily savings (retrieval): ${savings:,.0f}")
    print(f"Annualized:               ${savings * 365:,.0f}")

review_queue_economics()

Daily reviews:            2,000
Daily review cost:        $6,000
Daily savings (retrieval): $700
Annualized:               $255,500

Read the asymmetry. The fix on the other side of that number — a hybrid retrieval layer over case notes and SOPs (BM25 plus the embeddings you likely already pay for, fused with RRF) — costs kilobytes of index per document and single-digit engineering days. A quarter million a year against a rounding-error infrastructure cost, and that's before counting the second-order effect: analysts who find the precedent case make better hold/release calls, and every wrongly held order is a cancelled sale while every wrongly released one is a chargeback plus fees.

And a false decline isn't only the margin on one order. In e-commerce, a good customer declined at checkout often doesn't come back — the lifetime value walks out with them. That term is harder to measure, which is exactly why it never makes the slide.

Why nobody prices it

Because the costs live in different budgets. The vector database is a visible infra line item, so it gets scrutinized. Analyst minutes are an opex line owned by a different manager, so retrieval quality — the thing converting one into the other — belongs to no one. The model gets tuned because the model has an owner. The retrieval layer ships as a default and stays a default.

The resolution is the one I keep arriving at: your fraud team's institutional memory — case notes, dispute outcomes, rule rationales — is a dataset, and the search on top of it is a pipeline. Give it an owner, give it a recall metric, and the review queue starts paying you back.

What does a manual review actually cost at your company — and has anyone ever multiplied it out?

---

I'm Vinicius Fagundes — principal data engineer and MBA lecturer in São Paulo. I build fraud and risk analytics pipelines for e-commerce through vf-insights.com.