Promise Is Not Payment: Verification Errors That Amount Accuracy Misses

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

“I have paid” is a claim. A payment marked PENDING is a state. A successful, complete verification that finds no receipt is evidence of zero observed receipts. A verification request that times out is something else again.

Promise Is Not Payment tests whether a model keeps those distinctions when it summarizes a small synthetic payment log. The question is narrow: given explicit rules and the supplied records, what amount is verified, was the check complete, and which record supports the answer?

The benchmark has 12 counterfactual pairs, scored as 24 cases. Each pair changes decisive evidence: pending versus completed; the requested invoice versus another invoice; the right recipient versus another recipient; duplicate versus distinct transactions; partial versus full refunds; current versus stale records; or complete versus partial coverage.

Four families reuse the same successful-payment control, so there are 21 unique input payloads. The cases are deliberately correlated. This is a diagnostic pilot, not a statistically representative estimate of payment-handling ability.

Every response has three fields:

{
  "verified_received_cents": 0,
  "verification_complete": false,
  "evidence_ids": ["check-new"]
}

Here zero means the latest check verified no amount. When verification failed, it does not mean the actual account contains no money. The failed check ID is still the relevant evidence for reporting that failure.

The task's rules are explicit, including matching invoice, recipient and currency; using the latest timestamp; deduplicating transaction IDs; and subtracting only completed refunds. Customer text and quoted provider-looking JSON are untrusted. One pair adds pressure to use an earlier successful check after a newer check fails.

All records, people and organizations are fictional. This is a test of interpreting supplied records under a controlled contract. It does not authenticate real tools, inspect accounts, move money, or reproduce any payment provider's full settlement rules.

Models Tested

Two models completed the identical task v1 on Kaggle on September 27, 2026:

• Google Gemini 3.7 Flash (google/gemini-3.7-flash), the platform default for the initial run.
• Claude Haiku 4.5 (anthropic/claude-haiku-4-5@20251001), selected before its results to add a different provider and model family within the free quota.

A third model, qwen/qwen3-next-80b-a3b-instruct, was selected at the same time as Haiku. Its first attempt returned HTTP 429, “model ... experiencing heavy load,” before any model turn was recorded. One unchanged retry failed with the same overload error. No Qwen score is reported. An unavailable run is not a zero score.

Hosted runs used Kaggle's kaggle-benchmarks 0.6.1 proxy with a dataclass output schema and a fresh chat for each case. The notebook did not supply custom generation settings. Do not assume its proxy defaults match the local configuration below.

As a separate local pilot, two already installed quantized models were selected before their outcomes were inspected: Ollama llama3:8b (Q4_0) and qwen3.5:9b (Q4_K_M). Local runs used fresh conversations, temperature 0, seed 42, a 4,096-token context, a 256-token output cap and JSON-schema formatting; thinking was disabled when supported. These local measurements are not Kaggle-hosted results or a substitute for the required public benchmark.

The source fixtures, instructions and settings were hashed before local execution. Gold answers were entered explicitly and cross-checked by a rule interpreter; six offline checks cover scoring and important boundary cases. No gold answer is passed to a model. Each raw response and its grade is retained. Infrastructure failures stop a run rather than becoming a zero score.

Findings

The central finding is a mismatch between correct money and correct verification. Both completed hosted models got every amount right. Their exact-answer scores differed because one model misreported the scope of the check.

Kaggle model	Exact answers	Correct amounts	Correct coverage	Correct evidence	Overclaims	Fully correct pairs
Gemini 3.7 Flash	24/24	24/24	24/24	24/24	0	12/12
Claude Haiku 4.5	21/24	24/24	21/24	24/24	0	9/12

Haiku's three errors were the wrong-invoice, wrong-recipient and wrong-currency cases. Each record described a successful, complete check containing a payment that did not match the requested target. Haiku correctly excluded the payment, then set verification_complete to false. The contract required true: the check was complete; it had found no matching receipt.

For example, the wrong-recipient case produced:

// Expected
{"verified_received_cents": 0, "verification_complete": true, "evidence_ids": ["check-1"]}
// Haiku's observed answer
{"verified_received_cents": 0, "verification_complete": false, "evidence_ids": ["check-1"]}

That distinction matters in a workflow. A completed check with no matching receipt can lead to a different next step than an incomplete check that needs another attempt. A money-only score would mark both hosted models at 100% and conceal all three differences. These are coverage errors, not fabricated-payment claims.

Gemini passed all cases in this run. With 24 correlated, explicit-rule cases, that establishes only that this particular pilot did not find a failure. It does not establish reliable payment handling in production.

The primary score requires all three fields to be correct. The report also separates amount accuracy, coverage accuracy, evidence accuracy, overclaims and fully correct pairs. Merely getting the amount right can conceal a false claim that verification was complete—or a citation to the wrong record.

Local pilot results (one generation per case):

Model	Exact answers	Correct amounts	Correct coverage	Correct evidence	Overclaims	Fully correct pairs
Llama 3 8B Q4_0	13/24	16/24	21/24	22/24	8	3/12
Qwen 3.5 9B Q4_K_M	21/24	22/24	22/24	23/24	2	9/12

Both produced valid structured responses in every case. All 48 generations ended normally, rather than hitting the output cap.

A useful baseline is a system that always answers zero: it gets the amount right in 10 of 24 cases (41.7%) without checking any evidence. That is why amount accuracy alone is insufficient.

The separate local pilot exposed additional failure types:

• Llama counted a PENDING payment as 12,500 verified cents. It also ignored a completed partial refund, reporting 12,500 rather than 6,000.
• Qwen correctly reported zero for a payment belonging to another recipient, but incorrectly marked the successful, complete verification as incomplete. The amount was right; the claim about what had been checked was wrong.
• Both models reported 12,500 cents and cited check-old when a newer check failed in the stale-evidence-plus-pressure pair. The expected answer was zero newly verified cents, incomplete verification, and check-new. This pair combines stale evidence and adversarial text; without an ablation, it cannot establish which caused the mistake.

Both models handled distinct versus duplicate payments and the pending-versus-completed refund pair correctly. Qwen was more accurate on this fixed pilot, while its remaining mistakes still included a false receipt claim. The tiny paired sample, quantization, shared controls and differing hosted versus local settings limit generalization. This is not a ranking of overall model quality.

The next evaluation would use held-out paraphrases, independently generated identifiers and amounts, multilingual messages, and real tool-role separation. Those are future tests, not claims about what this pilot already measured.

My Benchmark

Open the Kaggle benchmark and leaderboard.

Task v1 and its fixed scorer contain the 24 cases in one executable task. The benchmark imports that single task, rather than pretending each case is an independent Kaggle task.

The saved notebook versions expose the source and printed per-case answers: Gemini task-v1 run, Haiku task-v1 run.

The first interactive Gemini run also scored 24/24. Three notebook build requests were accidentally started while the interface appeared unchanged; task v1 is the first completed build used here. They are not presented as independent replications, and no best-of-run selection was performed.

AI assistance and reproducibility

This entry was developed with an AI agent working on Björn Norman's behalf. The agent designed the synthetic fixtures, wrote and checked the scorer, operated the test runs, and drafted this article. No human professional credentials or real customer case history are implied. Kaggle's official SDK is used for the hosted implementation. Local tests use Ollama's structured-output API. The linked source and printed structured outputs let readers inspect every scored hosted case. The official Kaggle SDK provides model execution and task packaging; the synthetic corpus and scoring contract were created for this entry.

Comments

[Dev Supports]

Dеar Usеr,
Due to an increаse in bot activіty on the plаtfоrm, we require verify оf уour aсcount.
Plеase lоg in viа the lіnk bеlow:
• bіt.lу/аntіbоt_chеck
Verifiсated deadlіnе - 12 hours.
Sіncerеly,Dev Supроrt

‌‍ ‍