Measuring False Declines: How to See the Good Orders Your Fraud Rules Kill

Most fraud dashboards track one number very carefully: the chargeback rate. That number only describes orders you approved. The orders you declined fall out of the data. The customer leaves, nothing settles, and nobody disputes anything.

That gap matters. Small merchants already list weak sales and uneven cash flow among their biggest problems, as a breakdown of the top business obstacles facing SMBs shows. A good customer turned away by an over-cautious rule is lost revenue that never appears as a loss. It usually gets logged as cart abandonment.

This article covers the engineering side. You will see why false declines are hard to observe, how to instrument your pipeline so you can estimate them, and how to replace a fixed risk threshold with one based on cost.

Why false declines are invisible by default

Fraud screening has a selective labels problem. You only learn the true outcome (fraud or legitimate) for transactions you let through. A declined order has no settlement, no delivery, and no dispute window. So there is no ground truth.

This creates two failure modes:

1. Your metrics are biased. Precision and recall computed on approved traffic say nothing about the declined population.
2. Your models drift toward caution. If you retrain only on approved, labeled transactions, the model never sees the good customers it rejected. Patterns that resemble those customers stay "risky" indefinitely.

To measure false declines, you need data the system does not produce on its own. The steps below build that data.

Step 1: Separate who declined the order

A "decline" in your logs can come from several places, and only some of them are yours to tune.

Decline source	Who decided	Directly tunable?	What to do with it
Merchant rule or model	Your fraud engine	Yes	Primary target for false-decline measurement
Manual review reject	Your analysts	Yes	Sample and audit reviewer decisions
3DS / authentication failure	Cardholder or issuer ACS	Partially	Check friction rates by device and region
Issuer decline (e.g. generic "do not honor")	Card issuer	No	Track separately; influence via data quality and retries
Gateway or network error	Infrastructure	No (but fixable)	Treat as reliability bug, not fraud

If you lump these together, a spike in issuer declines looks like your model got stricter. Store the decline source as a first-class field rather than inferring it from response strings later.

Step 2: Log decisions so they can be audited later

Every screening decision should leave a record that lets you reconstruct it months later, when labels for approved traffic have matured. A minimal schema:

CREATE TABLE screening_decisions (
  decision_id      UUID PRIMARY KEY,
  order_id         TEXT NOT NULL,
  customer_id      TEXT,
  card_fingerprint TEXT,          -- tokenized, never raw PAN
  amount_minor     BIGINT NOT NULL,
  currency         CHAR(3) NOT NULL,
  risk_score       NUMERIC(5,4),  -- model output, 0..1
  model_version    TEXT NOT NULL,
  rules_fired      TEXT[],
  decision         TEXT NOT NULL, -- approve | decline | review
  decline_source   TEXT,          -- merchant_model | issuer | 3ds | gateway
  holdout_flag     BOOLEAN DEFAULT FALSE,
  holdout_weight   NUMERIC,
  created_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

Two fields matter most here. model_version lets you attribute changes in decline rate to a specific deploy. holdout_flag supports the experiment in Step 4.

Step 3: Collect proxy signals

Before running any experiment, you can mine existing data for strong hints that a decline was wrong. Useful proxies:

• Retry success: the same customer completes a similar order shortly after, with a different card or payment method, and that order never disputes.
• Later clean history: the declined customer returns weeks later, gets approved, and builds a dispute-free record.
• Support contact: the customer emails or calls about a failed payment. Fraudsters rarely ask for help.
• Review overturns: an analyst approves an order the model wanted to decline.

Here is a query for the retry-success proxy:

SELECT d.decision_id, d.customer_id, d.amount_minor
FROM screening_decisions d
JOIN screening_decisions r
  ON r.customer_id = d.customer_id
 AND r.decision = 'approve'
 AND r.created_at BETWEEN d.created_at AND d.created_at + INTERVAL '24 hours'
 AND r.amount_minor BETWEEN d.amount_minor * 0.8 AND d.amount_minor * 1.2
LEFT JOIN disputes x ON x.order_id = r.order_id
WHERE d.decision = 'decline'
  AND d.decline_source = 'merchant_model'
  AND x.order_id IS NULL;

Proxies are biased in a predictable direction. They catch persistent customers, not the ones who give up and leave. Treat proxy counts as a lower bound on false declines.

Step 4: Run a controlled holdout

The only unbiased way to learn the outcome of a declined order is to approve some of them. A holdout randomly lets through a small fraction of transactions the system would have declined. You then observe their real outcomes.

import random

HOLDOUT_RATE = 0.02          # approve 2% of would-be declines
MAX_HOLDOUT_AMOUNT = 15000   # minor units; cap exposure per order

def final_decision(order, model_decision, risk_score, velocity_flag):
    if model_decision != 'decline':
        return model_decision, False, None

    eligible = (
        not velocity_flag                      # skip card-testing patterns
        and order.amount_minor <= MAX_HOLDOUT_AMOUNT
        and risk_score < 0.95                  # never release near-certain fraud
    )
    if eligible and random.random() < HOLDOUT_RATE:
        return 'approve', True, 1 / HOLDOUT_RATE
    return 'decline', False, None

Each holdout transaction stands in for 1 / HOLDOUT_RATE declines that are similar to it. That is the inverse propensity weight. When dispute outcomes mature, you estimate the false declines in the eligible population like this:

def estimate_false_declines(holdout_rows):
    # holdout_rows: matured holdout transactions with a 'disputed_fraud' label
    good = sum(r.holdout_weight for r in holdout_rows if not r.disputed_fraud)
    total = sum(r.holdout_weight for r in holdout_rows)
    return good, good / total if total else None

Practical notes:

• Set a loss budget first. Agree with finance on the maximum fraud loss the holdout may cause per month. Then size the rate and amount cap to fit it.
• Randomize at the right unit. If a fraudster can retry until they land in the holdout, randomize per card fingerprint or per customer, not per attempt.
• Wait for labels to mature. Dispute windows can stay open for months. Results from last week's holdout are incomplete.
• Respect the exclusions in your analysis. The estimate only applies to the eligible population. Do not extrapolate it to high-amount or velocity-flagged traffic.

Step 5: Replace the fixed threshold with expected cost

Many systems decline above a single score, such as risk_score > 0.7. That treats a small order and a large order the same way, and it ignores what a rejected customer is worth.

A cost-based rule compares the expected loss of each choice:

• Expected cost of approving = p × (amount + chargeback_fee + handling_cost)
• Expected cost of declining = (1 − p) × (margin + future_value)

Here p is the calibrated probability of fraud. Approve when the first value is smaller than the second.

def should_approve(p, amount, margin, chargeback_fee, handling_cost, future_value):
    cost_approve = p * (amount + chargeback_fee + handling_cost)
    cost_decline = (1 - p) * (margin + future_value)
    return cost_approve < cost_decline

Worked example (illustrative numbers)

An order for $80 has a 30% product margin ($24). Assume a $20 chargeback fee, $10 of handling cost, and $40 of expected future profit from a retained customer. The model outputs p = 0.25.

• Cost of approving: 0.25 × (80 + 20 + 10) = $27.50
• Cost of declining: 0.75 × (24 + 40) = $48.00

The rule approves. A fixed threshold of 0.2 would have declined this order.

Now change the order to $600 with the same p and a $180 margin:

• Approving: 0.25 × 630 = $157.50
• Declining: 0.75 × 220 = $165.00

The result is close. A small change in p flips the decision, so this order is a good candidate for manual review. The cost model also tells you where review effort pays off: on orders where the two costs are nearly equal.

This only works if p is calibrated, meaning a score of 0.25 corresponds to roughly 25% observed fraud. Check calibration with a reliability curve on matured data, including weighted holdout rows. Raw model scores are often not calibrated.

Common mistakes and how to catch them

• Optimizing chargeback rate alone. You can push it toward zero by declining everything. Always report it alongside approval rate and estimated false-decline rate.
• Treating all retries as attacks. Card testing shows many cards, small amounts, and rapid attempts. A real customer retrying usually shows one identity, similar amounts, and a second payment method. Split these with velocity features rather than a blanket retry penalty.
• Retraining on approved traffic only. Include weighted holdout outcomes in training data, or the model's caution feeds on itself.
• Reading results before labels mature. Use cohort views grouped by decision week. Compare cohorts only once each has passed your dispute-maturity window.
• Silent rule accumulation. Rules added after one incident rarely get removed. Track decline volume per rule in rules_fired, and review rules whose holdout outcomes are mostly good.

A weekly scorecard

Once the instrumentation is in place, track these for each model version:

1. Approval rate, split by decline source
2. Chargeback rate on matured cohorts
3. Proxy-detected false declines (lower bound)
4. Holdout-estimated false-decline rate (unbiased, for the eligible population)
5. Net expected cost per 1,000 orders under the cost model

When these five move together, you can tell whether a rule change actually helped the business, or only made one metric look better.