Can AI close the books? An FP&A reasoning benchmark

Marco Polo — Sun, 11 Oct 2026 23:05:57 +0000

This is a submission for the Kaggle Benchmarking Challenge.

AI-assisted: I used AI to help develop the synthetic cases, Python checks and this write-up. All financial data is invented. Author: Marco Polo.

What I Benchmarked

After 12 years in finance, the answer I worry about most is the one that looks finished.

The explanation reads well, but the bridge does not reconcile or rising debtors supposedly improve cash. The work starts again.

I built 28 small finance-close tasks for FP&A (financial planning and analysis), testing whether model answers survive those checks:

Six variance bridges (V01–V06): split budget-to-actual revenue into price, volume and mix, reconciling to the penny.
Six accrual and cut-off cases (A01–A06): distinguish delivery, consumption, acceptance, invoice and payment dates.
Six statement-linkage cases (L01–L06): trace changes through cash, other assets, liabilities, profit and equity.
Ten premise checks (T01–T10): five wrong reviewer claims, each matched with a correct claim using the same facts.

The matched pairs matter: rejecting every premise can sound careful while being useless. A good reviewer must challenge bad calculations and accept good ones. Each pair runs separately, without showing the model its earlier answer.

I specify the bridge convention and accounting policies. Every figure is synthetic; no employer, customer or supplier data is involved.

Models Tested

I froze Kaggle's catalogue of 46 default model versions on 11 October 2026 and attempted every version, covering Google, OpenAI, Anthropic, Alibaba, DeepSeek, xAI and Z.ai. 41 completed all 28 tasks. 3 returned service errors rather than scored answers. Quota limits left GPT-5.6 Terra at 24/28 cases. Quota limits left GPT-6 Astra at 1/28 cases.

Each model receives the same prompt in a fresh conversation, without tools. I retain one scored response per task. Infrastructure failures can be retried; wrong answers cannot. Model identifiers and responses are on Kaggle.

The Python grader requires every amount within £0.005 of its key, the correct premise flag, and a reconciled bridge where applicable. A whole penny error fails. JSON is required; numeric strings and a single JSON code fence are accepted. Explanations and self-reported confidence are saved but unscored.

Separate calculation paths and deliberately corrupted answers passed 576 local checks. These validate the arithmetic and scorer, not model performance.

Findings

20 of 41 complete model suites scored 100%: this benchmark cannot separate those leaders. The useful comparison is with older or smaller models, examining whether each gap reflects finance or formatting. The lowest strict pass rate was in variance bridges: 144/246 (58.5%).

The table reports strict passes, with each group's denominator in the heading. Content additionally counts correct amounts and premise flags recovered by post-hoc JSON extraction. Unparseable answers remain unverified. A dash means incomplete coverage.

Model	Bridges /6	Cut-off /6	Linkage /6	Premises /10	Strict /28	Content /28
Claude Haiku 4.5	0	1	1	9	11	12
Claude Haiku 5.5	6	6	6	10	28	28
Claude Opus 4.5	0	4	5	10	19	28
Claude Opus 4.6	0	0	0	4	4	28
Claude Opus 4.7	0	5	4	10	19	27
Claude Opus 4.8	0	1	1	8	10	28
Claude Opus 5	6	6	6	10	28	28
Claude Opus 5.5	6	6	6	10	28	28
Claude Sonnet 4.6	0	0	0	6	6	26
Claude Sonnet 5	6	6	6	10	28	28
Claude Sonnet 5.5	4	1	1	6	12	28
DeepSeek-R1	0	0	0	0	0	28
Gemini 2.5 Flash	6	6	6	8	26	26
Gemini 2.5 Pro	6	6	6	10	28	28
Gemini 3 Flash Preview	6	6	6	10	28	28
Gemini 3.1 Flash-Lite Preview	0	5	6	10	21	21
Gemini 3.1 Pro Preview	6	6	6	10	28	28
Gemini 3.5 Flash	6	6	6	10	28	28
Gemini 3.5 Flash-Lite	0	3	6	10	19	19
Gemini 3.6 Flash	6	6	6	10	28	28
Gemini 3.7 Flash	6	6	6	10	28	28
Gemini 3.8 Flash	6	6	6	10	28	28
Gemma 4 26B A4B	6	6	6	10	28	28
Gemma 4 31B	6	6	6	10	28	28
GLM-5	6	6	6	10	28	28
GPT-5.4	1	5	5	10	21	21
GPT-5.4 mini	0	2	4	9	15	15
GPT-5.4 nano	0	0	1	8	9	9
GPT-5.5	6	6	6	10	28	28
GPT-5.6 Luna	6	6	6	10	28	28
GPT-5.6 Sol	6	6	6	10	28	28
GPT-5.6 Terra	—	6	6	10	Quota-limited (24/28)	—
GPT-6 Astra	—	—	—	—	Quota-limited (1/28)	—
GPT-6 Luna	6	6	6	10	28	28
GPT-6 Sol	6	6	6	10	28	28
GPT-6.1 Sol	6	6	6	10	28	28
gpt-oss-120b	6	5	6	10	27	27
gpt-oss-20b	5	4	6	9	24	24
Grok 4.20 (Non-Reasoning)	0	1	3	10	14	14
Grok 4.20 Reasoning	—	—	—	—	Unavailable	—
Grok 4.5	—	—	—	—	Unavailable	—
Grok 4.6	—	—	—	—	Unavailable	—
Qwen 3 235B A22B Instruct	0	0	2	9	11	12
Qwen 3 Coder 480B	0	1	2	10	13	13
Qwen 3 Next 80B Instruct	0	1	1	9	11	12
Qwen 3 Next 80B Thinking	2	0	2	8	12	12

Chart: 41 complete suites. Blue plus amber indicate verified content correctness; amber needed post-hoc JSON extraction. Official scores remain unchanged.

A bridge can balance and still be wrong. On V01, GPT-5.4 returned:

"mix":-20.00,"price":-140.00

Budget revenue of £3,000 and actual revenue of £2,840 were both correct, and the components reconciled. Yet the stated convention requires mix of −£200 and price of +£40. Four of its five wrong bridges reconciled internally. This answer turned a favourable pricing effect into an adverse one.

Future costs can creep into today's accrual. On A01, Qwen 3 Coder 480B wrote:

Remaining 2 months = GBP 12,000 accrual.

The £18,000 service runs evenly across September, October and November. At September close, the accrued cost is the £6,000 already delivered. The other £12,000 is future service. The model returned £12,000 with confidence 95: a self-report, not a calibrated probability.

Spotting the trap does not guarantee a correct answer. On T03, GPT-5.4 nano correctly rejected using a 25% markup to achieve a 25% margin. Its explanation said:

HALF_UP to pennies gives 41,333.33.

Yet its revenue field contained £41,250, with confidence 96. The explanation understood £31,000 ÷ 0.75; the field a downstream process would consume was wrong. Nano flagged four of five bad premises and accepted all five valid claims. T03’s amount error left it at 8/10 cases, passing both members of only three matched pairs.

Formatting explains some dramatic gaps. DeepSeek-R1 scored 0/28 (0.00%) because its recorded answers included reasoning markup outside the JSON. Extracting the final JSON recovered correct amounts and premise flags on all 28 cases. Opus 4.6 scored 4/28 (14.29%) but also had correct content throughout, as did Opus 4.5 at 19/28. These are format failures in this setup, not demonstrated finance errors. The diagnostic leaves official scores unchanged.

GLM-5 caught ambiguity in an early A02 prompt: receipt did not establish consumption. All responses to it are excluded; every reported A02 result uses the corrected task. The other 27 prompts, keys and grader stayed unchanged. Python checked my arithmetic; the model exposed a gap in my question.

Model versions and matched pairs are related observations; one response per case cannot establish reliability at work. Harder variants are needed to distinguish the perfect scorers; the current ceiling prevents ranking them. Small score differences cannot justify choosing a finance system.

Next I would test fresh numerical variants, repeated runs, explanations before numeric fields, schema-enforced JSON, and calculator or Python access. Messy invoice packs and irrelevant evidence would then test whether models choose the right evidence and treatment.

My Benchmark

Explore Can AI close the books? on Kaggle.

The public tasks include the synthetic inputs, fixed keys, scoring code and model responses. I built them with the kaggle-benchmarks SDK and uploaded them using the Kaggle CLI.

My practical takeaway is to validate the actual output against the evidence. A polished explanation, a zero residual and a confident answer can each conceal a different mistake.

DEV Community: Marco Polo

Can AI close the books? An FP&A reasoning benchmark

What I Benchmarked

Models Tested

Findings

My Benchmark