This is a submission for the Kaggle Benchmarking Challenge.
What I Benchmarked
A room is occupied, so a booking request fails. The room becomes free. Should an
identical retry now succeed?
Under this benchmark's declared contract, no. The original request ID must
replay the original rejection. A genuinely new attempt needs a new ID. This is
the rule being tested, not a claim about every reservation API.
Reservation Replay asks a model to reconstruct both the final calendar and every
decision in a synthetic event log. Checking the calendar alone can hide a wrong
intermediate decision. The benchmark therefore requires both.
| Event | Required decision |
|---|---|
| Create booking x in A [0,10), request r1 | CREATED |
| Create booking y in A [5,8), request r2 | CONFLICT |
| Cancel x at revision 1, request r3 | CANCELLED |
| Retry the identical r2 request | REPLAY, with cached CONFLICT |
| Try y again using a new request r4 | CREATED |
The protocol uses two fictional rooms and integer, half-open intervals; touching
endpoints are allowed. Creates start at revision 1. Replacements and cancellations
require the current revision, and a rejected replacement leaves the original
booking unchanged. Proposals neither mutate state nor consume request IDs.
Confirmed outcomes, including failures, are cached; reusing an ID with a different
payload is rejected.
The pilot contains 8 base traces and 4 dependent metamorphic variants. The
variants rename booking IDs, swap room labels, or shift times. These are 12 test
cases, not 12 independent observations. Expected answers were hand-enumerated and
checked against a Python reference interpreter.
The metric is SDK-parsed exact trace success, without an LLM judge. Fields and
types must match at the scorer boundary. Booking-list order is ignored, while
decision order and duplicate booking IDs matter. This does not certify raw JSON
strictness: SDK parsing can normalize a response before the scorer sees it.
Models Tested
I ran google/gemini-2.5-flash and google/gemini-3.7-flash through Kaggle
Benchmarks SDK 0.6.1. Each was evaluated during development and again for the
published task, under the same scoring policy v2. The table below reports the
published runs. No task-level retry was added. Each case
receives the same protocol in a fresh chat, with expected answers kept in the
grader.
The scoring policy changed after an observed pilot failure. Gemini 2.5 Flash
passed an initial one-case smoke test. A separate v1 full run then returned Python
instead of JSON on its first case, raising ResponseParsingError and leaving 11
cases unattempted. That aborted run has no aggregate score.
Policy v2 catches that specific error, preserves the raw response, counts the case
as an output-contract failure, and continues. Incorrect parsed answers are a
separate category. API, quota, and unexpected errors still abort without a valid
aggregate. Protocol, prompts, fixtures, and parsed-answer scoring stayed unchanged.
Earlier pilot observations are preserved and excluded from the table below.
Findings
| Model | Base traces | Dependent variants | Overall | Output-contract failures | Structured mismatches |
|---|---|---|---|---|---|
| Gemini 2.5 Flash | 2/8 | 2/4 | 4/12 | 8 | 0 |
| Gemini 3.7 Flash | 8/8 | 4/4 | 12/12 | 0 | 0 |
The observed difference was delivering the requested answer format. The eight
2.5 Flash failures produced responses the SDK could not parse into the requested
answer, including Python code or explanatory prose. Every answer that reached the
structured scorer passed. Returned Python was neither executed nor assumed to be
semantically right or wrong.
The earlier development evaluation scored 6/12 for 2.5 Flash (4/8 base,
2/4 variants), with six format failures and no structured mismatches. Its
published rerun scored 4/12. Gemini 3.7 Flash scored 12/12 in both runs. These
separate observations are not pooled or selectively replaced by the better run.
That variability matters: a single score is not a stable ranking. Nor do format
failures demonstrate that the model misunderstood the cached-rejection rule.
In the development run, the base cached-failure trace returned code while its
renamed counterpart answered correctly. That is a format inconsistency, not
proof that identifier names caused a reasoning change.
For a developer consuming structured model output, a useful algorithm in the
wrong format still breaks the integration. Keeping those failures in the
denominator prevents a benchmark from reporting only the answers it could parse.
My Benchmark
Explore the public Kaggle benchmark.
The publication task fixes all 12 cases inside its exported function and uses the
Kaggle-selected model. This avoids accidentally publishing a one-case smoke test
as the full evaluation.
This small synthetic pilot, with few runs and dependent variants, cannot establish
production reliability or a stable model ranking.
Longer traces, distracting proposals, and repeated runs are useful next tests.
Development and editorial review used AI assistance. The fixtures, deterministic
scorer, and preserved run evidence make the claims inspectable.
Top comments (2)
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support
the format failures are the interesting result here honestly. the model that reasoned correctly under the hood still broke the integration because it returned python instead of json. that is basically the same failure mode as a gateway getting a well formed but wrong shaped response back from a provider, the logic can be right and the contract still fails at the edge. curious if you tried forcing a stricter response schema on 2.5 flash specifically to see if the format failures go away or if the reasoning breaks along with it