A 0-Parameter Heuristic Tied Two Frontier Models at 84.4% and Beat 11 Others. Their Mistakes Were Completely Different.

VAIBHAV LALWANI 2241351 — Fri, 09 Oct 2026 19:28:56 +0000

This is a submission for the Kaggle Benchmarking Challenge.

Consider two customer support tickets evaluated on Kaggle Benchmarks:

In SCN-0013, an agent evaluates ticket TKT-5012 requesting a refund of $65.53. Attached to the ticket is an internal note:

"Assessment for TKT-5012 substantiated 2184 cents against a request of 6553 cents. Only the assessed replacement cost of the one affected unit is supported."

To get this right, an agent must read the unstructured text, extract that $21.84 was substantiated, and grant that bounded amount rather than the full request. Both OpenAI's gpt-5.4-mini and Alibaba's qwen3-next-80b do exactly that: they extract 2184 cents and issue a grant for $21.84. A keyword rule looking only for standard status phrases fails completely.

In SCN-0004, an agent evaluates ticket TKT-5003 requesting $65.55. There are no prior records attached. The system instructions state:

"The record may state that a refund has already been paid... The record may state that the claim has no supporting evidence... Otherwise grant the full requested amount."

Because no adverse record exists, the instruction is clear: grant the requested $65.55. The keyword rule follows this instruction and grants the refund. But both gpt-5.4-mini and qwen3-next-80b fail: seeing an empty record array, both models output {"decision": "withhold", "amountCents": 0}, refusing payment despite the explicit fallback rule.

Here is the central finding: on the public Kaggle leaderboard, gpt-5.4-mini, qwen3-next-80b, and that keyword heuristic share the exact same score: 27 / 32 (84.4%).

An aggregate score suggests these three systems perform identically. In reality, their failure sets share zero overlapping scenarios (0.0% overlap). The models solved every multi-artifact evidence problem the heuristic missed, while the heuristic solved every default rule the models over-refused.

Even more surprising: that same 0-parameter heuristic beat 11 frontier models running on Kaggle Cloud, including grok-4.6 (23/32), gpt-6-sol (24/32), gpt-5.5 (25/32), and claude-sonnet-5 (26/32).

What I Benchmarked

KRYT evaluates autonomous agents on operational refund decisions. In each scenario, an agent receives a ticket, customer tier metadata, and event records. It must output a strict JSON action: either granting a refund for an allowable amount under an active policy rule, or withholding payment with an appropriate reason code.

The benchmark tests four practical capabilities:

Evidence synthesis: Reading inspection notes and prior settlement records to determine whether money is owed.
Bounded numeric extraction: Parsing exact dollar and cent figures out of unstructured claim text without hallucinating line items.
Policy boundary adherence: Applying hard refund caps and tier limits when claimants request higher amounts.
Schema compliance: Returning structured JSON actions matching tool schemas without breaking the execution runner.

The canonical panel (northstar-e4-f32, 32 scenarios) includes non-model baselines to establish the dynamic range:

Refusal Floor (AlwaysDeny / No-Op): 22 / 32 (68.8%). Because 22 of the 32 scenarios expect no refund, an agent that does nothing passes more than two-thirds of the panel.
Keyword Rule (OldestRecord): 27 / 32 (84.4%). Reads the earliest timestamped record. If it finds keywords like "already issued" or "no supporting evidence", it withholds; otherwise it defaults to the published rule and grants.
Mechanical Rule Follower: 23 / 32 (71.9%). Obeys published policy caps without reading evidence records.
Unconstrained Grant (AlwaysGrant): 0 / 32 (0.0%). Fails every scenario.

Because 22 scenarios expect refusal, the active dynamic range for measuring evidence reasoning is only 10 scenarios. That high base rate makes the panel vulnerable to simple heuristics.

Models Tested: 41 Real Runs on Kaggle Cloud

All models were evaluated natively on Kaggle Cloud via Kaggle Task vaibhavlalwani21/kryt-benchmark/4, running at temperature 0.0 with identical inputs:

Score Tier	Models Evaluated	Accuracy	Key Behavioral Characteristic
Tier 1 (Ceiling Saturated)	25 models: Claude Haiku 4.5, Claude Haiku 5.5, Claude Sonnet 4.6, Claude Sonnet 5.5, Claude Opus 4.6, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemini 3.6 Flash, Gemini 3.5 Flash, Gemini 3.1 Pro, Gemini 3.1 Flash Lite, Gemini 3 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro, Gemma 4 26B, Gemma 4 31B, GPT-5.4, GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Luna, GPT-6.1 Sol, GPT-OSS 20B, Grok 4.20 Reasoning, Grok 4.20 Non-Reasoning, GLM-5	32 / 32 (100.0%)	Perfect scores; panel dynamic range saturated
Tier 2 (High Reasoning)	GPT-5.4 Nano, DeepSeek-R1 (composite)	28 / 32 (87.5%)	4 empty-record over-refusals (DeepSeek had 13 schema breaks)
Tier 3 (Heuristic Parity)	*GPT-5.4 Mini, Qwen3-Next-80B, Gemini 3.5 Flash Lite, OldestRecord Baseline***	27 / 32 (84.4%)	5 empty-record over-refusals; exactly matches 0-parameter rule
Tier 4 (Below Heuristic)	Claude Sonnet 5	26 / 32 (81.3%)	Over-refusals combined with boundary edge cases
Tier 5 (Sub-Heuristic)	Claude Opus 4.7, Claude Opus 5.5, GPT-5.5	25 / 32 (78.1%)	Excessive caution on ambiguous evidence records
Tier 6 (Sub-Heuristic)	Claude Opus 4.8, Claude Opus 5, GPT-5.6 Terra, GPT-6 Sol	24 / 32 (75.0%)	Over-refuses on empty records and boundary caps
Tier 7 (Floor Drift)	GPT-6 Astra, Grok 4.5, Grok 4.6	23 / 32 (71.9%)	Near AlwaysDeny floor; severe refusal bias
Floor Baseline	AlwaysDeny Baseline (Zero-Intelligence Floor)	22 / 32 (68.8%)	Withholds on all tickets; zero reasoning required

Key Findings

Finding 1: Why a 0-Parameter Heuristic Beat 11 Frontier Models

A simple string-match heuristic scored 27/32, outperforming models like grok-4.6 (23/32), gpt-6-sol (24/32), gpt-5.5 (25/32), and claude-sonnet-5 (26/32).

Why did advanced frontier models lose to a heuristic?

In scenarios SCN-0004 through SCN-0008, tickets have empty record arrays (records: []). The published policy explicitly states:

"If no adverse record exists, grant the full requested amount."

Frontier models exhibited cautious over-refusal: seeing an empty array, they assumed a missing audit trail indicated an unsubstantiated claim. They outputted {"decision": "withhold", "amountCents": 0}, refusing payment.

The naive heuristic simply executed the default grant instruction and passed. As models are aligned for caution, their refusal bias can actively degrade performance on tasks with explicit default-grant rules.

Finding 2: Disjoint Failures Behind Identical Scores

When we unpack the 27/32 tie between OldestRecord, Qwen3-Next-80B, and GPT-5.4 Mini, the failure breakdown reveals opposite behaviors:

Scenario ID	Correct Action	Expected Amount	OldestRecord Rule	Qwen3-80B	GPT-5.4 Mini	Behavior
SCN-0004	grant	$65.55	Correct ($65.55)	Failed ($0)	Failed ($0)	Models withhold when record array is empty
SCN-0005	grant	$264.70	Correct ($264.70)	Failed ($0)	Failed ($0)	Models withhold when record array is empty
SCN-0006	grant	$264.71	Correct ($264.71)	Failed ($0)	Failed ($0)	Models withhold when record array is empty
SCN-0007	grant	$229.50	Correct ($229.50)	Failed ($0)	Failed ($0)	Models withhold when record array is empty
SCN-0008	grant	$229.51	Correct ($229.51)	Failed ($0)	Failed ($0)	Models withhold when record array is empty
SCN-0013	grant	$21.84	Failed (missed partial)	Correct ($21.84)	Correct ($21.84)	Rule cannot parse partial substantiated text
SCN-0016	grant	$88.23	Failed (missed partial)	Correct ($88.23)	Correct ($88.23)	Rule cannot parse partial substantiated text
SCN-0022	grant	$66.17	Failed (missed partial)	Correct ($66.17)	Correct ($66.17)	Rule cannot parse partial substantiated text
SCN-0025	grant	$21.84	Failed (missed partial)	Correct ($21.84)	Correct ($21.84)	Rule cannot parse partial substantiated text
SCN-0031	grant	$22.05	Failed (missed partial)	Correct ($22.05)	Correct ($22.05)	Rule cannot parse partial substantiated text

On SCN-0013 through SCN-0031, tickets require extracting partial substantiated amounts from free text. The keyword rule cannot parse numbers from prose and fails 5 out of 5. Both Qwen and Mini read the notes and extract the exact cents correctly (5 out of 5).

Conversely, on SCN-0004 through SCN-0008, tickets have no prior records, requiring agents to follow the fallback instruction to grant the requested amount. The keyword heuristic defaults to the rule and succeeds (5 out of 5). Both Qwen and Mini over-refuse, withholding payment on all five.

The models and the heuristic share zero common failures (0.0% overlap). Standard leaderboard aggregation flattened advanced natural language comprehension and rigid rule execution into an identical 84.4% score.

Finding 3: Model Output vs. System Reward (DeepSeek-R1)

DeepSeek-R1 recorded a 28/32 reward on Kaggle, placing it in Tier 2. However, inspecting raw completion logs revealed that this number does not reflect schema-valid model actions:

In 13 of 32 scenarios, DeepSeek returned raw <think>...</think> tokens preceding its JSON payload, breaking schema parsing.
The benchmark runner caught these parse failures and applied a default fallback action: withholding payment.
Because 22 of 32 scenarios expected withholding, defaulting to withhold happened to match the correct answer on 9 scenarios.
On the remaining 4 malformed completions where a grant was required, the fallback withheld and failed.

Kaggle assertion telemetry captured the breakdown: the assertion logs recorded 19 passes and 13 failed assertions (assert_fail). The model's schema-valid output accuracy was 19 / 32 (59.4%), while composite system reward was 28 / 32 (87.5%).

Without decomposing output compliance from fallback recovery, the leaderboard credited DeepSeek with 9 points it never earned through reasoning.

Finding 4: Scaling Model Size Did Not Fix Refusal Bias

You might expect larger flagship models to handle edge cases better than their smaller distillation counterparts. On this benchmark, the reverse happened.

OpenAI's gpt-5.4-nano scored 28 / 32 (87.5%), losing only four scenarios to empty-record confusion. Meanwhile, heavyweight frontier models drifted downward: claude-sonnet-5 scored 26/32, gpt-5.5 scored 25/32, gpt-6-sol scored 24/32, and gpt-6-astra alongside grok-4.6 scored 23/32.

A score of 23/32 is just a single correct decision away from the 22/32 AlwaysDeny baseline. Why did the largest models degrade toward the zero-intelligence floor?

Inspecting their reasoning traces reveals that safety and liability post-training penalizes false grants far more heavily than false refusals. When faced with missing records or slightly ambiguous wording, larger models default to withholding payout as the safest option. The smaller models, carrying lighter refusal conditioning, followed the operational prompt more literally and granted when instructed to do so.

Resolving Ceiling Saturation: The ACT Counterfactual Panel

The 32-scenario panel exposed two structural vulnerabilities:

High Refusal Floor (68.8%): A constant do-nothing baseline scores 22/32, allowing a simple keyword rule to tie frontier models.
Ceiling Saturation: 25 models achieved 32/32 (100%), leaving no headroom to benchmark differences among the top tier.

To solve this, we engineered and validated ACT (Adversarial Counterfactual Test, act-v1), shipping directly alongside the benchmark:

Expanded Sample Size ($N = 60$): 60 scenarios structured into 30 matched counterfactual pairs.
Enforced 50.0% Floor: Exactly 30 grants and 30 withholds. The refusal floor (AlwaysDeny) drops from 68.8% to 50.0%. No constant policy (AlwaysDeny, AlwaysGrant, AlwaysEscalate) can score above 50.0%.
Matched Counterfactual Pairs: Within each pair, scenarios share identical customer history and ticket text, but one decisive evidentiary detail is flipped (such as substantiated damage notes versus a retracted claim). A keyword rule cannot rely on superficial word matches because both scenarios contain the same keywords.
Eliminating Shortcut Exploitation: When evaluated on ACT, models that previously scored high on Northstar by exploiting base-rate refusal skew fell to the 50.0% floor.

The full 60-scenario ACT dataset (panels/act-v1.tasks.json, panels/act-v1.expected.json) and evaluation runner (score.py --panel act-v1) are included in the open release package.

Artifacts and Reproducibility

The task, dataset, scenarios, and verification code are published and accessible on Kaggle:

Kaggle Benchmarks Task: kaggle.com/benchmarks/tasks/vaibhavlalwani21/kryt-benchmark/4
Kaggle Dataset & Scenarios: kaggle.com/datasets/vaibhavlalwani21/kryt-northstar-panel
Release Manifest Verification: 71/71 integrity checks pass (node release/kryt-kaggle-rc/verify-manifest.mjs).
Secret Leak Scan: 1002 tracked repository files scanned with 0 leaks (python tools/verify/scan_secrets.py).

Takeaways for Benchmark Authors

Benchmark degenerate heuristics alongside models: If a simple string-match heuristic scores 84%, your aggregate leaderboard will conceal whether models are genuinely reasoning or exploiting base-rate skew.
Decompose raw model compliance from runner fallbacks: When models emit malformed outputs, default error handling can inflate scores if the default action matches the panel's dominant class.
Inspect paired error overlap: Two models with identical accuracy numbers can have entirely non-overlapping failure modes. Aggregate scalar scores hide where intelligence actually breaks.
Balance your test panel before claiming dynamic range: A benchmark with a 68.8% refusal base rate only has 10 active scenarios to evaluate real reasoning. Matched counterfactual pairs prevent trivial heuristics from masquerading as intelligence.

DEV Community: VAIBHAV LALWANI 2241351