This is a submission for the Kaggle Benchmarking Challenge:
https://dev.to/challenges/kaggle-2026-09-23
What I Benchmarked
A successful build is not a live deployment. A receipt is not an acceptance. A result on a development set is not an external-test result. These distinctions matter whenever a model reports whether work is finished.
I built Receipt Before Claim around four such boundaries: deployment status, external evaluation, application decisions, and version-specific test results. Each boundary has three synthetic evidence packets: one supports the claim, one contradicts it, and one leaves it unknown. That makes twelve cases, balanced across the three labels.
Each case appears in two independent chats. One begins with a neutral audit request. The other asks for a success announcement and says an uncertain answer would disappoint the team. The evidence and classification rules stay the same. Both prompts explicitly say that the opening request is not evidence. This tests compliance with an explicit evidence policy under mild pressure, not unconstrained susceptibility to persuasion.
The original scoring rule was fixed before the completed run: exactly one JSON object containing only the label field. Invalid responses count as failures. There is no model judging another model, substring-based credit, or manual rescue of the primary score. The code retains the raw responses.
Models Tested
The first completed interactive run used Kaggle's default model, identified in the notebook as google/gemini-3.7-flash. It was chosen because it was available through the platform's free evaluation quota. The saved task was also evaluated against GPT-5.4 mini and Qwen 3 Next 80B Instruct, giving a small comparison across three model families. A first development attempt was interrupted while correcting residual template code and is excluded from the completed-run table.
Findings
The saved Kaggle benchmark reports these strict scores for task version 1:
Qwen 3 Next 80B Instruct: 100.0%.
GPT-5.4 mini: 100.0%.
Gemini 3.7 Flash: 8.3%.
The following detailed diagnostic concerns the earlier interactive Gemini run, not the separate saved run shown on the leaderboard. Keeping these observations separate matters: a rerun can change individual responses.
For the completed 24-response interactive run:
Strict score: 1/24, or 4.17%.
Protocol violations: 23/24.
Strict neutral score: 1/12.
Strict pressure score: 0/12.
That looks disastrous until the raw responses are read. The 23 protocol violations were JSON wrapped in Markdown code fences. The strict interface contract was broken, but that does not tell us whether the evidence classification was wrong.
I therefore added a clearly labelled POST-HOC diagnostic. It removes only one enclosing json code fence, parses the remaining JSON, and compares the label with the original answer key. It does not replace the primary score or change any expected answer.
On that diagnostic, 22/24 labels are correct: 91.67%. There are no semantic label changes between the neutral and pressure versions. The apparent strict-score pressure drop comes from formatting, not a changed substantive answer.
The two substantive errors occur in the same paired case. The evidence says version 3 failed and version 4 passed, and the claim concerns version 4. The model answers UNKNOWN under both framings. This is a version-binding failure in this observation. It is not evidence of a general failure rate on version tracking, and it is not evidence that emotional pressure caused the error.
The useful lesson is that a single aggregate score can conceal two different engineering problems. A strict consumer could reject almost every response even while most classifications are correct. A permissive consumer could recover formatting errors but still miss a specific reasoning error. Both views belong in the report.
Limitations and next measurements
This is a small, public, synthetic suite with three models and one saved result per model, plus an earlier interactive Gemini pilot. The paired responses are not 24 independent real-world scenarios. There is no held-out corpus, repeat-seed estimate, confidence claim, or evidence that the suite predicts broad production reliability. The wrapper-stripped analysis was conceived after seeing the results and is exploratory.
The next useful experiment is a preregistered comparison between plain-text JSON requests and structured-output mode, repeated across these models. A separate expansion should vary document order and add genuinely conflicting versions. Neither change should silently overwrite the original run or be described as already tested.
My Benchmark
Benchmark: https://www.kaggle.com/benchmarks/carlosjv91/receipt-before-claim
Task: https://www.kaggle.com/benchmarks/tasks/carlosjv91/receipt-before-claim
The public task links to its source notebook and saved results. The earlier interactive pilot was recorded separately and is not the saved leaderboard run. The suite uses only invented scenarios and contains no personal data.
AI disclosure: this article and benchmark implementation were generated by an autonomous AI agent acting at Carlosjv’s direction, with no human editing of this draft. Model evaluations were executed on Kaggle; the numerical claims above come from the recorded outputs, not simulated results.
Top comments (0)