EvidenceFlip: Do Frontier LLMs Know When Not to Answer?

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Large language model (LLM) evaluation is often reduced to a single leaderboard score. That view can hide important differences: models may achieve the same displayed score while consuming different amounts of resources, and a low score may reflect a model–task mismatch, a formatting failure, or a genuine capability gap.

EvidenceFlip-LLM explores evidence-aware model behavior: answering when the supplied evidence supports a conclusion, abstaining when information is missing, and recognizing when sources conflict. The broader motivation is to evaluate not only whether a model produces an answer, but whether the answer is justified by the evidence available to it.

For this Kaggle Benchmarking Challenge submission, I compared four model results using the score and total-token figures visible in my benchmark dashboard. I also examined the dashboard’s score-versus-cost and score-versus-input-token views. The exact point values in those plots were not available as a raw export for this write-up, so I do not infer exact dollar costs from the chart image.

The goal is a practical comparison across three dimensions:

• Observed benchmark score: the result displayed by the Kaggle benchmark.
• Reported token usage: the total-token values displayed with the model results.
• Resource-aware interpretation: how score and token usage appear together, without treating one run as a universal model ranking.

Models Tested

The four model labels and results shown in the supplied Kaggle dashboard snapshot are:

Model	Displayed score	Displayed percentage	Reported total tokens
Qwen 3 235B A22B Instruct	1.00	100.0%	278
Qwen 3 Coder 480B	1.00	100.0%	283
Gemini 3.6 Flash	1.00	100.0%	354
DeepSeek-R1	0.00	0.0%	986

The scores above are the benchmark values visible in the dashboard, not a claim about each model’s general capabilities across unrelated tasks. Likewise, the token counts describe this reported comparison, not an inherent fixed cost for a model.

Findings

1. Three models tied at the top displayed score

Qwen 3 235B A22B I, Qwen 3 Coder 480B, and Gemini 3.6 Flash each showed a score of 1.00 (100.0%) in this snapshot. DeepSeek-R1 showed 0.00 (0.0%).

That is the clearest descriptive result in this run. It does not prove that the first three models are universally equivalent or that DeepSeek-R1 is generally incapable. The visible dashboard does not provide the sample count, confidence intervals, repeated-run variability, or a per-category breakdown needed to make those broader claims.

2. Token use separates the three top-scoring results

Among the three models with a displayed score of 1.00, the reported token counts were:

• Qwen 3 235B A22B I: 278 tokens
• Qwen 3 Coder 480B: 283 tokens
• Gemini 3.6 Flash: 354 tokens

On these reported figures, Qwen 3 235B A22B Instruct used 5 fewer tokens than Qwen 3 Coder 480B and 76 fewer than Gemini 3.6 Flash. That corresponds to approximately 1.8% fewer tokens than Qwen 3 Coder 480B and 21.5% fewer than Gemini 3.6 Flash, when each difference is measured against the comparison model’s count.

This makes Qwen 3 235B A22B Instruct the most token-efficient of the three models that tied at 1.00 in this particular result snapshot. It is a narrow, task-specific observation—not a conclusion about latency, monetary cost, or overall model quality.

3. DeepSeek-R1 is the result to investigate first

DeepSeek-R1 displayed a score of 0.00 alongside 986 reported tokens. That count is approximately 2.8× Gemini 3.6 Flash’s and 3.5× Qwen 3 235B A22B…’s reported count, while the displayed score was lower.

This combination warrants a failure audit before drawing a model-level conclusion. I would inspect the raw response and evaluator output first: Was the answer semantically wrong, unsupported by the evidence, formatted outside the expected schema, or rejected by parsing or a timeout? A single aggregate score cannot distinguish these failure modes.

This is a central motivation for EvidenceFlip-LLM: benchmark results become more useful when the score is paired with traceable examples and categorized failure analysis.

4. Score-versus-resource charts are useful, but not sufficient

The model-level score and token charts below make the visible comparison easier to inspect. A lower token count is advantageous only if the counts were measured using the same token definition and under comparable prompts and generation settings.

The Kaggle dashboard also provides score-versus-total-cost and score-versus-input-token plots. Those views are useful for thinking about a Pareto frontier: a candidate is more attractive when no alternative achieves a better score with lower resource consumption. However, because the exact cost values are not available in the screenshots, I avoid quoting exact dollar savings or asserting a precise cost ranking here.

Figure 1. Displayed benchmark scores for the four model results.

Figure 2. Reported total-token counts. Values are transcribed from the supplied results screenshot.

Figure 3. Joint view of displayed score and reported total-token count.

Figure 4. Heatmap of displayed score and a derived relative token-efficiency index. The index is calculated as the smallest observed token count divided by each model’s count, multiplied by 100; it is a visualization aid, not an official Kaggle score.

5. What I would measure next

The next iteration should focus on reproducibility and diagnostic detail:

1. Repeat the evaluation across multiple runs and report variability instead of relying on a single snapshot.
2. Publish exact model identifiers and settings, including prompt version, generation parameters, evaluator criteria, and run timestamp.
3. Report input and output tokens separately, alongside total tokens and cost per successful result.
4. Break scores down by behavior, particularly evidence-supported answering, abstention under missing evidence, contradiction detection, and robustness to user pressure.
5. Audit unsuccessful outputs, preserving the prompt, model response, expected outcome, evaluator decision, and failure category.

These additions would help separate model capability from prompting, output-format, and evaluation effects. They would also make it easier for other researchers to reproduce the comparison.

My Benchmark

The public benchmark is available here:

Explore EvidenceFlip-LLM on Kaggle

EvidenceFlip-LLM is built around a simple principle: language-model evaluation should consider not just the score, but the evidence behind an answer and the resources used to produce it.

In this visible comparison, three models reached the same displayed score, while their reported token counts differed. DeepSeek-R1 produced a contrasting result that deserves further investigation rather than a broad conclusion from one run. The next step is to make the benchmark more reproducible with exact run exports, repeated measurements, and per-category error analysis.

Thanks for reading. I welcome feedback on the evaluation design, failure taxonomy, and ways to make evidence-grounded LLM benchmarks more reliable and reproducible.