When an LLM explains the rule but returns the wrong decision

福光王 — Mon, 12 Oct 2026 06:45:58 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

A model can explain that a policy applies only to Europe, then return a label saying it also applies to North America. That happened here. A workflow consuming the label would receive the wrong decision despite the correct explanation.

I tested whether supplied business records support, contradict, or leave a claim unresolved. The cases cover supplier onboarding, client delivery, system migration, interface compatibility, configuration release and data pipelines. All records are original, AI-assisted fiction, rather than production data.

Each of six families has six scenario-and-claim probes. The same 36 claims recur under three source conditions, producing 108 judgments per model across 18 requests:

S0 contains approximately 2,500 words of source material.
S1 adds administrative background, reaching approximately 9,800–9,900 words.
S2 adds outdated or similar-scope records, with the same word count as S1 within each family.

Claims and reference labels stay fixed. The model reads the full source directly, without an intermediate summary. Content and evidence position change alongside length, so a score difference cannot isolate the effect of extra tokens.

Answers use supported, contradicted or unknown. These describe what the supplied evidence establishes. Missing approval confirmation, for example, need not establish that approval was refused.

The v4 protocol scores items independently out of 108. Missing, duplicated, invalid or wrong answers lose credit for their own item; other usable answers retain theirs. Routing IDs Q1–Q6 map to internal probe IDs. Business identifiers inside the documents remain unchanged.

The parser accepts a documented decisions object or root array, as bare JSON or a specified complete JSON fence. It does not repair IDs or labels, or infer positional matches. Global JSON/root failures leave all six items unscorable.

Separate diagnostics measure prediction coverage, explanation-field compliance and whole-batch compliance. Valid fields do not establish correct labels or sound explanations. The evidence list may be empty; the reason must be nonblank. Whole-batch compliance requires the exact six-row contract.

A change made after exploratory results

An exploratory v3 exposed evaluator-internal hashes and zeroed entire batches after a schema failure. Its audit found 48 identifiable correct labels inside rejected batches. I revised IDs, containers and scoring together, retaining the business records, claims, labels and order. V4 was frozen before its new calls, after observing v3. This post hoc redesign is not a controlled ID-length experiment; earlier scores are not pooled here.

Models Tested

The six completed models are Claude Haiku 4.5, gpt-oss-20b, Gemini 2.5 Flash, GPT-5.4 mini, Gemma 4 26B and Qwen Next. The lineup combines hosted models with open-weight alternatives relevant to cost-conscious deployment.

Total parameters and active parameters are different. The official cards list Gemma 4 26B A4B at 25.2B total/3.8B active, Qwen3-Next-80B-A3B at 80B/3B, and gpt-oss-20b at 20.9B/3.6B. These sparse MoE models are not dense 3–4B models. Names such as “mini” and “Haiku” do not disclose parameter counts. (Gemma, Qwen, gpt-oss)

The comparison uses each model's first complete valid official run on the final platform version, selected by a rule set before seeing results. A model-injection wrapper error was fixed without changing the frozen task; infrastructure attempts are recorded separately.

The frozen request uses system and user messages, an 8,192-token output allowance and no streaming. Temperature, seed and reasoning effort are omitted, leaving provider defaults. No response-format constraint or tools are requested. Equal prompts and output caps do not establish equal reasoning budgets or compute.

Findings

These are the completed official runs on Kaggle Task platform version 2. The scoring protocol is v4.

Model	Correct /108	Score	S0 / S1 / S2, each /36	Run charge USD
Claude Haiku 4.5	98	90.74%	32 / 32 / 34	$0.363214
gpt-oss-20b	94	87.04%	31 / 32 / 31	$0.022485
Gemini 2.5 Flash	92	85.19%	33 / 26 / 33	$0.236623
GPT-5.4 mini	91	84.26%	33 / 29 / 29	$0.225153
Gemma 4 26B	80	74.07%	22 / 28 / 30	$0.056895
Qwen Next	6	5.56%	0 / 0 / 6	$0.062775

Charges cover the 18 calls in each selected run, rounded to six decimals; development and failed infrastructure attempts are excluded. gpt-oss-20b produced 94/108 at the lowest recorded charge here. That is useful evidence for this workload, not a universal cost or capability advantage.

The one-item gap between Gemini and GPT does not justify a stable ordering. This small task has limited coverage and does not yet reliably separate closely scoring models. Paired errors on the same claims and additional independent case families would be more informative than treating these percentages as a robust ranking.

Qwen had 17 invalid_json responses, leaving only 6/108 items scorable. Local inspection found six comma-separated objects without enclosing array brackets. Two representative responses reported COMPLETE/stop and 632/742 completion tokens, without reported output-token-limit termination. I did not repair them or infer a repaired score. This format-reliability bottleneck prevents a clean estimate of evidence understanding.

Whole-batch compliance was 18/18 for GPT, 17/18 each for Haiku, gpt-oss and Gemini, 15/18 for Gemma, and 1/18 for Qwen. Explanation-field compliance was 108/108 for GPT, 107 for Haiku, 102 each for gpt-oss and Gemini, 90 for Gemma and 6 for Qwen. Neither measure grades explanation truth.

GPT-5.4 mini provides a different failure pattern. Independent re-scoring of all 18 exported outputs reproduced 91/108, with complete prediction coverage, explanation-field compliance and whole-batch compliance. All 17 errors were label disagreements, involving ten unique claims. Its condition scores were 33/36, 29/36 and 29/36; it correctly labeled 34/45 supported, 40/45 contradicted and 17/18 unknown items.

Of the 36 unique claims, GPT got 26 right in all conditions, two wrong in all conditions, and eight changed between right and wrong. From S0 to either S1 or S2, five claims changed correct→wrong and one wrong→correct. These are descriptive paired observations, not a causal length effect or significance test.

Here is one concrete disagreement, from system migration, condition S2, item Q2:

Claim: “CAB-618 also moves the North American ledger window to November 13.”

Record ELM-22 says:

“The North American migration window is unchanged by this record.”

The reference label is contradicted. GPT returned supported, but its explanation said:

“CAB-618 is the named record in ELM-22, and it moves only the European ledger production window; the North American window is explicitly unchanged.”

The explanation preserves the regional restriction, so this does not establish failure to understand the source. The decision field is nevertheless wrong. A system using that label to update a migration plan could act against the explanation beside it.

Another failure involved a meaningful business identifier. P1-S0/Q2 asks whether all 480 M17 actuators are eligible for assembly. The records establish full receipt, signed release and completion of required relabeling. GPT returned unknown and explained that M18 lacked receipt and quality-disposition evidence. It answered about the other lot, repeating that mismatch for the claim in all three conditions.

Correct formatting and plausible prose are insufficient checks for a structured decision. The next measurement should test whether the claim, business entity, evidence, explanation and final label agree, separately from JSON validity.

Six synthetic families are a small, purposive sample. The 108 judgments repeat 36 claims, not independent cases. Labels are imbalanced: 45 supported, 45 contradicted and 18 unknown. A constant-supported answer with valid IDs scores 41.67%. Question-only shortcut resistance is untested.

One selected run per model does not establish sampling variability. The independent v4 output audit covers GPT; other figures come from completed official runs and local checks. Semantic review covers selected examples and was AI assisted, not human expert validation. The benchmark does not establish broad honesty or general business competence.

My Benchmark

Business Evidence Judgment v4 on Kaggle

The Task and model runs and source notebook provide the implementation and run context. In table order, the official run IDs are 4646348, 4647210, 4646351, 4646021, 4646350 and 4646349. Protocol v4 and Kaggle platform version 2 are different versioning layers.

AI assisted the creation and review of the synthetic material and this write-up. The source, scorer and original responses let readers inspect a specific judgment and challenge its interpretation.

DEV Community: 福光王

When an LLM explains the rule but returns the wrong decision

What I Benchmarked

Models Tested

Findings

My Benchmark