DEV Community: yongchan kwon (nuckdrip)

When a Failed Request Must Stay Failed: Reservation Replay

yongchan kwon (nuckdrip) — Sat, 26 Sep 2026 15:54:32 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

A room is occupied, so a booking request fails. The room becomes free. Should an
identical retry now succeed?

Under this benchmark's declared contract, no. The original request ID must
replay the original rejection. A genuinely new attempt needs a new ID. This is
the rule being tested, not a claim about every reservation API.

Reservation Replay asks a model to reconstruct both the final calendar and every
decision in a synthetic event log. Checking the calendar alone can hide a wrong
intermediate decision. The benchmark therefore requires both.

Event	Required decision
Create booking x in A [0,10), request r1	CREATED
Create booking y in A [5,8), request r2	CONFLICT
Cancel x at revision 1, request r3	CANCELLED
Retry the identical r2 request	REPLAY, with cached CONFLICT
Try y again using a new request r4	CREATED

The protocol uses two fictional rooms and integer, half-open intervals; touching
endpoints are allowed. Creates start at revision 1. Replacements and cancellations
require the current revision, and a rejected replacement leaves the original
booking unchanged. Proposals neither mutate state nor consume request IDs.
Confirmed outcomes, including failures, are cached; reusing an ID with a different
payload is rejected.

The pilot contains 8 base traces and 4 dependent metamorphic variants. The
variants rename booking IDs, swap room labels, or shift times. These are 12 test
cases, not 12 independent observations. Expected answers were hand-enumerated and
checked against a Python reference interpreter.

The metric is SDK-parsed exact trace success, without an LLM judge. Fields and
types must match at the scorer boundary. Booking-list order is ignored, while
decision order and duplicate booking IDs matter. This does not certify raw JSON
strictness: SDK parsing can normalize a response before the scorer sees it.

Models Tested

I ran google/gemini-2.5-flash and google/gemini-3.7-flash through Kaggle
Benchmarks SDK 0.6.1. Each was evaluated during development and again for the
published task, under the same scoring policy v2. The table below reports the
published runs. No task-level retry was added. Each case
receives the same protocol in a fresh chat, with expected answers kept in the
grader.

The scoring policy changed after an observed pilot failure. Gemini 2.5 Flash
passed an initial one-case smoke test. A separate v1 full run then returned Python
instead of JSON on its first case, raising ResponseParsingError and leaving 11
cases unattempted. That aborted run has no aggregate score.

Policy v2 catches that specific error, preserves the raw response, counts the case
as an output-contract failure, and continues. Incorrect parsed answers are a
separate category. API, quota, and unexpected errors still abort without a valid
aggregate. Protocol, prompts, fixtures, and parsed-answer scoring stayed unchanged.
Earlier pilot observations are preserved and excluded from the table below.

Findings

Model	Base traces	Dependent variants	Overall	Output-contract failures	Structured mismatches
Gemini 2.5 Flash	2/8	2/4	4/12	8	0
Gemini 3.7 Flash	8/8	4/4	12/12	0	0

The observed difference was delivering the requested answer format. The eight
2.5 Flash failures produced responses the SDK could not parse into the requested
answer, including Python code or explanatory prose. Every answer that reached the
structured scorer passed. Returned Python was neither executed nor assumed to be
semantically right or wrong.

The earlier development evaluation scored 6/12 for 2.5 Flash (4/8 base,
2/4 variants), with six format failures and no structured mismatches. Its
published rerun scored 4/12. Gemini 3.7 Flash scored 12/12 in both runs. These
separate observations are not pooled or selectively replaced by the better run.

That variability matters: a single score is not a stable ranking. Nor do format
failures demonstrate that the model misunderstood the cached-rejection rule.
In the development run, the base cached-failure trace returned code while its
renamed counterpart answered correctly. That is a format inconsistency, not
proof that identifier names caused a reasoning change.

For a developer consuming structured model output, a useful algorithm in the
wrong format still breaks the integration. Keeping those failures in the
denominator prevents a benchmark from reporting only the answers it could parse.

My Benchmark

Explore the public Kaggle benchmark.

The publication task fixes all 12 cases inside its exported function and uses the
Kaggle-selected model. This avoids accidentally publishing a one-case smoke test
as the full evaluation.

This small synthetic pilot, with few runs and dependent variants, cannot establish
production reliability or a stable model ranking.
Longer traces, distracting proposals, and repeated runs are useful next tests.

Development and editorial review used AI assistance. The fixtures, deterministic
scorer, and preserved run evidence make the claims inspectable.

The Museum of Almost: a museum you explore by making choices

yongchan kwon (nuckdrip) — Sat, 26 Sep 2026 14:24:57 +0000

Built for the Sanity Challenge, Path Two: Vibe-Code Something Strange.

What I Built

Who should be allowed to borrow your rain?

The Museum of Almost is a small interactive exhibition for people who enjoy speculative design and branching stories. Its six fictional inventions begin with everyday objects: an umbrella that replays the sound of a past rainy walk, a clock that leaves a minute uncounted, a bench that makes room for shared silence.

Each asks you to make a choice. Give your rain to the neighborhood and follow the story to The Listening Bench. Keep an afternoon for yourself and discover The Borrowed Minute. Change your choice to explore another possibility, or open the map to see how the collection connects.

The inventions and their imagined prototype anecdotes are labeled fiction. The six illustrations are CSS object studies. Choices follow authored branches; the app does not generate new stories at runtime.

Demo

Explore The Museum of Almost.

Choose an object, make a choice, and see what follows. Local Sanity-connected build, September 26.

Public visitor walkthrough: no login is needed. Open The Rain Library, choose Give it to the neighborhood, follow the consequence to The Listening Bench, then explore the possibility map. Exhibit links and initial map choices can be shared and reopened; subsequent choices within a visit remain temporary.

The Rain Library offers a choice about sharing an afternoon. Local Sanity-connected build, September 21.

The possibility map shows how choices connect the collection. Local Sanity-connected build, September 21.

Local curator sandbox: use Node.js 24 (tested with 24.19.0) and pnpm 11.19.0. Start a fresh clone without an environment file and with MUSEUM_STORAGE unset:

git clone https://github.com/asdv3189/museum-of-almost.git
cd museum-of-almost
git checkout be19b9b6507b15a0f727fd10bd8ecab2aeff0591
pnpm install --frozen-lockfile
pnpm dev

Open http://127.0.0.1:8792/?view=curator; no account or key is needed on loopback in local mode. Edit an exhibit's description and save, then open Review changes, send for review, approve the revision, and put it on the wall. Check the exhibition before and after publication to see the draft boundary.

This sandbox stores changes in local .data/museum.json. It demonstrates the curator workflow without connecting to or testing Sanity. The hosted museum uses Sanity; its separate live verification is described below, and its writable curator room remains private.

Code

The public source repository contains the Next.js app, Sanity schemas and adapter, domain rules, and tests. The published source was verified against the prepared package at commit be19b9b.

That source snapshot's README still describes deployment as pending. The same source is now deployed at the demo URL above and was verified there on September 26.

My Build Process

Codex desktop produced the implementation, exhibition copy, CSS illustrations, and tests. Other agents reviewed the code and tests, and browser inspection checked the running app. These are agent observations, not human audience testing.

The build brief paired six unrealized inventions with a precise constraint for the curator room: approve an exact revision, then publish only that unchanged revision. This made the schema part of the experience. A visitor's route needs a choice, a consequence, and a destination; an editor's change needs a draft, a review, and a publication boundary.

The first implementation got important details wrong. A map connection opened an object but lost the selected branch, so browser review led to preserving the choice and its consequence together. A generated mapper overwrote Sanity's document type with an embedded content type; independent tests caught the collision. Namespaced document IDs also required authenticated server reads despite the public dataset, which changed the adapter's access configuration.

The September 26 revision made the invitation explicit: choose an object, make a choice, see what follows. It also added URL restoration and repaired reload, Back/Forward, and keyboard-focus behavior. 127 automated tests, typechecking, and the local production build passed. Desktop and mobile browser checks used the locally running app connected to Sanity.

Separately, on September 21, eight workflow checks through the local server exercised real Sanity reads and writes, including stale-request rejection, approval invalidation, and publication visibility. Fourteen successful transitions restored the original fictional content while retaining the audit; a read-only check then matched all six published exhibits to their workspace snapshots.

On September 26, browser checks of the Netlify deployment confirmed all six Sanity-backed exhibits and the Rain Library's neighborhood-sharing consequence leading to The Listening Bench. An empty curator key was rejected; the configured key opened the workbench. A temporary draft edit stayed invisible to a separate unauthenticated visitor. Restoring the original text, requesting review, approving, and publishing then succeeded, leaving five audit events and the original public description intact. These hosted checks used the app's simulated roles; they did not rerun the 127 local tests.

Sanity Project Details

Project ID: xmyaojxc · Dataset: production

Sanity holds the collection and its editing process. Branch destinations use native references. The workspace stores drafts, reviews, approvals, snapshots, and audit events; separate exhibit documents provide published content for GROQ queries.

Editing invalidates approval. Publication rejects stale revisions, missing destinations, and cycles, then updates the workspace and published exhibit together in a revision-guarded Sanity transaction. Visitors keep seeing the previous published version while an edit is under review.

This is a custom application workflow. Curator and reviewer are simulated roles, not separately authenticated people. The app uses neither App SDK nor Sanity's managed Workflows product.

Agent Session

The build log records the brief, implementation decisions, failures, and verification. It is a curated account; no agent-session transcript is attached.