DEV Community: Abhishek Nandy

I Wanted to Be Lazy About Calculus. So I Built a Gemma Lab That Does the Experiments for Me

Abhishek Nandy — Sun, 25 Oct 2026 22:16:11 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

I asked Gemma to explain a vector field. It was confidently wrong. So I built FieldTrip.

Less setup. More aha. A local Gemma-powered vector calculus lab for the Slow and Steady challenge.

Gemma told me a smooth vector field was singular.

The animation looked convincing. The explanation was wrong.

I had asked it about a rotating field with components (-2y, 2x, 0). Its explanation mixed up the curl, the integrals, and an excluded axis that this field does not have.

That became the most useful failure in this project. I needed a clearer division of work: let the model interpret the question, let math tools calculate, and let me inspect both.

I also wanted to stop doing the repetitive parts of exploring calculus: writing plotting code, rebuilding a scene, changing one parameter, recording the result, and doing it again.

So I built FieldTrip, a small vector calculus lab where one question can become a reviewable experiment, an interactive 3D scene, and a table of comparisons you can replay.

The sloth gets to watch the arrows. The lab does the setup.

What I Built

FieldTrip turns supported natural-language questions into visual investigations of vector fields.

Ask:

Show a rotating field and compare loop sizes.

Local Gemma selects a field and a comparison. FieldTrip shows that selection before running it. After I accept, code supplies the formula, calculates derivatives and integrals, and builds the scene.

The lab includes five starting points:

Field	Question to explore
Vortex	Can something rotate without expanding?
Source	Can something expand without rotating?
Saddle	Can a shape stretch while preserving local volume?
Gradient	What does following an uphill direction look like?
Excluded-axis vortex	Why can zero curl coexist with nonzero circulation?

There are particles to follow, arrows to inspect, a boundary curve, a spanning surface, and controls for radius, strength, surface lift, and orientation. The math panel exposes symbolic divergence and curl. A point probe lets you inspect a location.

Explore for me runs parameter comparisons. Each table row has a replay button that restores its configuration. Investigations can be exported as JSON and imported again; calculations are recomputed rather than accepted from the imported numbers.

The public lab runs curated experiments immediately. Local Gemma inference uses the downloaded app and Ollama.

Demo

The public deployment and the local AI version have different responsibilities. Render hosts the interactive front end. Gemma runs on my laptop through Ollama. Opening the public URL does not provision a model for the visitor.

How I Built It

Give Gemma a small, useful job

The first version gave the model too much freedom. It could produce expressions and explanations that sounded plausible while disagreeing with the calculations.

The current planner returns a small structured object:

{
  "field": "vortex",
  "comparison": "radius"
}

The allowed field IDs correspond to the five examples, plus an outside-scope response. The comparison is radius, surface, orientation, or none. Unknown IDs and extra properties are rejected.

Gemma does not generate the preset formulas. It chooses which supported experiment fits the request.

Before anything changes, a review dialog shows the selected field, comparison, formula, and defaults. I can run it or keep the current experiment.

![Reviewing Gemma's selected vortex and radius comparison before execution.

Schema validation checks the shape of the answer. The review step checks whether that answer matches what I meant.

Explicit expressions such as F=(x,y,z) take a separate path through a restricted parser. That route does not need an AI call. The interface distinguishes it from a Gemma-selected experiment.

Make the calculations inspectable

The browser uses math.js for symbolic differentiation and numerical evaluation, Three.js for the scene, and GSAP for interface animation. Particle paths use a fourth-order Runge–Kutta step.

The expression evaluator permits a limited mathematical syntax and known symbols. It does not execute arbitrary generated JavaScript.

For Stokes comparisons, the application independently samples the boundary line integral and the curl flux through a parameterized surface. The boundary is a circle in z=0; the surface lift bends its interior while keeping that boundary fixed.

At the default resolution, the line integral uses 128 angular samples and the surface integral uses 32 radial by 128 angular samples.

The scene is an aid to inspection. Particle density is not presented as a measurement of divergence. Numerical agreement is not presented as a proof.

Automate the repetition

For the vortex F=(-y,x,0), the curl is (0,0,2). Around a counterclockwise circle of radius R, circulation is 2πR².

FieldTrip produces this comparison:

Radius	Boundary circulation	Sampled curl flux
0.5	1.5708	1.5708
1.0	6.2832	6.2832
1.5	14.1372	14.1372
2.0	25.1327	25.1327

Doubling the radius quadruples the result. Reversing the boundary orientation and surface normal changes both signs. Lifting the interior of the spanning surface preserves the boundary and, for this smooth example, preserves the matching flux.

One request becomes four computed configurations. Clicking a row brings that configuration back into the scene.

That is the laziness I wanted: less parameter bookkeeping, more time noticing what changes.

The Example That Makes Me Stop Scrolling

The excluded-axis field is

F = (-y/(x²+y²), x/(x²+y²), 0).

Away from the z-axis, its curl is zero. Yet a positively oriented circle around that axis has circulation 2π.

This does not contradict Stokes' theorem. The spanning disk crosses the axis where the field is undefined, so the theorem's smoothness requirements are not satisfied.

The numerical sampler can miss that axis and return a curl-flux value of zero. A finite set of successful evaluations cannot establish that the entire surface is admissible.

FieldTrip therefore marks the known example ASSUMPTIONS FAIL. The exported invalid flag reports sampled evaluation failures; it is not a certificate that the domain is valid.

This is a classic counterexample to an unchecked inference, not a new mathematical discovery.

That distinction became a design rule: show the values and the conditions under which they mean something.

I Tested the Planner, Including the Uncomfortable Results

I created a frozen set of 24 questions: 15 supported requests and 9 outside the lab's scope. I ran the same bounded-planner prompt and dataset against two local models.

A fully correct plan requires both the expected field ID and comparison ID.

Measure	Gemma 3 1B	Gemma 4 E2B
Correct field, all 24 questions	9/24	16/24
Correct field and comparison, all 24	5/24	14/24
Correct field on supported requests	9/15	15/15
Correct field and comparison on supported requests	5/15	13/15
Correct rejection of unsupported requests	0/9	1/9

Gemma 4 selected every supported field correctly in this small test. It was poor at rejecting unsupported requests.

It also made two comparison mistakes: adding a radius comparison where none was requested, and missing an orientation comparison.

Those failures explain two features in the app: the review step and a conservative code-based scope gate. Known requests such as “solve the heat equation” are stopped before inference. Ambiguous requests such as “change the surface” ask for a field or behavior instead of silently replacing the current experiment.

The scope gate was added after these measurements. Its behavior is not an improvement in the model's rejection score. Its phrase rules are incomplete and can overblock wording; they are a boundary for this small lab, not a general intent detector.

Earlier formula-translation tests also informed the redesign. On a separate 30-question development set, the small model-only approach passed 2/30. A hybrid route passed 27/30, but all 27 successes came from the direct parser. Calling that “90% AI accuracy” would misrepresent what happened.

These are author-created development tests, not an independent benchmark or an end-to-end accuracy claim. The repository includes questions, runners, raw responses, timings, and results. The bounded runs record matching dataset and prompt hashes so the comparison can be inspected.

Why Open Innovation Matters

I built and tested the local version on a Windows laptop with an Intel i7-8565U, 16 GB RAM, and integrated graphics.

With the model and app dependencies downloaded, the local workflow can run without an internet connection. I tested an offline investigation and exported its results. Prompts sent through the local Ollama configuration stay on the laptop; a cloud model key is not required.

Local inference still consumes hardware resources and electricity. The benefit is no per-request cloud inference fee, rather than a claim that computation is free.

Gemma's open weights let me compare models on my own task and keep the inference under my control. The source lets someone change the planner, inspect the formulas, replace a model adapter, or challenge a calculation without waiting for a service provider to expose those decisions.

The original application code is MIT-licensed. Gemma's weights and the third-party libraries retain their own licenses and terms.

Open innovation mattered most when the first approach failed: I could inspect the raw outputs, narrow the model's responsibility, and keep the useful parts of the workflow.

The Lazy Payoff—and the Limits

FieldTrip saves the repetitive setup of a supported investigation: selecting an example, creating its scene, calculating operators, running comparisons, and collecting configurations for replay.

I have not measured learning gains or a manual-versus-automated time saving, so I am not claiming either.

It does not solve arbitrary vector calculus problems, PDEs, quantum equations, or arbitrary surface geometries. It does not prove theorems. Custom non-polynomial domains still need review. Local model requests can take several seconds, and a valid plan can still be the wrong interpretation.

The useful outcome is smaller and concrete: ask for an experiment, review the setup, and spend your attention exploring it.

Run It Yourself

For the public curated lab, open FieldTrip on Render.

For local AI, install Ollama and Python, download the repository, then pull the model:

ollama pull gemma4:e2b

Run start-windows.cmd, open http://localhost:8765, and select gemma4:e2b in the Gemma settings. Use the local /api/generate proxy described in the README. A successful settings save alone does not demonstrate inference; submit a supported question and inspect the plan review.

Try this first:

Show a rotating field and compare loop sizes.

Then try the loophole. The missing axis is the part worth slowing down for.

Prize Categories

Best Use of Gemma: local, structured experiment planning with measured model comparisons and a review step.
Best Use of Render: hosting the public interactive lab so visitors can explore curated experiments without installing the local AI stack.

Built for Slow and Steady: let the lab handle the repetition, and keep the curiosity for yourself.

Overturn: the denials small clinics stopped fighting

Abhishek Nandy — Sun, 18 Oct 2026 14:38:46 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Every small clinic has a drawer of denied claims it has quietly given up on. Each one is real money, often a few thousand dollars. Each one also takes a billing person about twenty minutes to read the denial, decode the code, check the chart, and write an appeal. At a few hundred denials a month, most of them get written off, because the work costs more than the expected return.

Overturn changes that arithmetic. It is an appeal agent that reads a denial notice, decodes it, estimates the chance of winning, compares three possible moves, drafts the letter, and checks the draft against the notice. The billing person sees a ranked queue, so the valuable denials come first.

Who would pay for it: a clinic, or a billing service that works denials for several clinics. The question the app answers is "which of these denials is worth a staff member's time," and that question has a dollar answer.

Demo

Two recordings, one from the laptop and one from the live Render deployment:

The laptop run uses a 7B model and the local retrieval index. The Render run uses a 3B model on a CPU instance. The differences matter, and I explain them below.

Code

The code is on the master branch: https://github.com/abhitorch81/week2hoct/tree/master

The repo contains the Python app, the playbook of the twelve most common denial codes, the test suite, the Mastra agent, the Tinker training pipeline, and the Render blueprint.

How I Built It

Live on Render

The web app runs on Render as a web service, with a separate worker service that runs the Temporal workflows. The screenshots below come from the live deployment.

The deploy pipeline. The service is connected to the GitHub repo on the master branch and deploys automatically on every push. The deploy history shows six deploys, the latest of which was an automatic deploy that took about 47 seconds. Two earlier deploys switched the default model to qwen2.5:3b-instruct, one through the blueprint for the 4 GB instance and one by hand. Rollback is one click on any earlier deploy.

The app in production. The header tells you the state of the system at a glance. The model is qwen2.5:3b-instruct running on Ollama, marked REMOTE because it runs on a private service inside Render's network. The odds model is scikit-learn with an AUC of 0.77. Retrieval is marked memory with 18 wins, not pgvector, because the Render deployment has no database URL configured and falls back to an in-memory index with the same scoring. The runner is Temporal. The external-call counter reads 0.

The page also shows the workflow the app is running. Each card is one step of the pipeline, with its own timing. Step 7, drafting the appeal, took about 103 seconds, and every other step took under 20 milliseconds. That single number explains most of the design: the model is the only slow part, so the model is the only part that needs careful budgeting.

The Render deploy history for the web service, with auto-deploy from master

The Trace and cost panel gives the same story in numbers. End-to-end time was 103,671 ms, with 1,587 LLM tokens. The cost line shows $0.00 for local compute and an illustrative $0.0079 at a $5 per million token closed API price. The "0 / 0" for external and total model calls needs explaining, and I come back to it in the honest-limits section.

The Temporal worker. Each run is a Temporal workflow, and each pipeline step is an activity. The worker process registers on the overturn task queue, and the screenshot shows the worker's health: a recent heartbeat, the Python SDK version, 474 workflow tasks processed, and a workflow cache hit rate of 100 percent. A cache hit means the worker did not have to replay the workflow history to continue it, which is what keeps long runs cheap.

The event history. This is the screenshot I find most useful for explaining durability. The history lists every scheduled and completed activity. You can see the run_step activity scheduled, started, and completed, followed by a workflow task that records the result. If the worker died between two of these events, Temporal would replay the history on a new worker and continue from the last completed step. The completed step's result is stored in the history, so the model does not run again for finished work.

How the Decision Works

The most useful thing Overturn does is refuse to write letters for claims that are not worth fighting. The math is simple enough to check by hand.

For each candidate move, the app computes an expected value:

expected value = win probability × denied amount − staff cost

For CLM-26009, a BCBS claim denied with CO-50 ("not medically necessary"), the denied amount is $6,286.64 and the win probability for an appeal is about 61.25 percent. The expected value is 0.6125 × $6,286.64 − $85 ≈ $3,765, which matches the $3,765 shown in the queue and the agent's answer. The app compares that with a resubmission, which has its own recovery odds and a $22 staff cost, and with a write-off, which has zero recovery and zero cost. The highest value wins.

This is a deliberately simple model. It treats the win probability as the main unknown, and it uses staff time as the only cost. A real practice would add appeal deadlines, the payer's history, and the value of the staff's time elsewhere. Those are inputs the clinic can change, which is why they are environment variables rather than constants.

What the Model Does, and What It Does Not

The model writes one thing: the argument section of the appeal. Everything else in the letter, including the letterhead, the claim block, the specific request, and the list of enclosures, is built from the notice's facts by code. That split is the core safety idea. A model can invent a sentence, but it cannot invent a claim number if the claim number never passes through the model.

The fact-checker enforces this. It compares every claim ID, CPT code, denial code, and amount in the draft against the source notice. A failed draft gets one corrective retry. If the second attempt also fails, the app falls back to a template letter and shows you the rejected draft, so you can see what went wrong.

I tested the checker with a fake model that invents codes on purpose. It rejected every bad draft. That test is test_hallucinated_llm_draft_is_rejected in the repo.

Retrieval: Learning From Past Wins

Overturn searches past wins for similar denials. The search has two parts. Keyword search (BM25) finds exact matches on codes and payer names, which vector search often misses. Vector search finds paraphrases of similar arguments, which keyword search misses. The two result lists are merged with reciprocal rank fusion, and matches that share the denial code and payer get a boost.

On the laptop, the index lives in Postgres with pgvector on Tiger Cloud, and it currently holds 19 documents. On Render, the same scoring runs over an in-memory index with 18 wins. The retrieval code path is the same, but the storage differs. I want to be clear about that, because a reader who sees "pgvector" in the laptop header and "memory" on Render should not assume they are the same deployment.

Durability: Why the Workflow Is Built This Way

A denial pipeline runs for minutes, calls a model that is occasionally slow, and writes to a database. Any of those can fail partway through. A plain function would lose its progress. Overturn runs the same eight steps in two ways: a local durable runner that checkpoints each step, and Temporal, which records each step in an event history on the server.

The benefit is concrete. In the Render run, the drafting step took 103 seconds. If the worker had restarted during that step, Temporal would have retried the activity with backoff, and finished steps would not have run again. The local runner does the same thing with a checkpoint file. I have tested the local runner and the Temporal activity logic. The kill-and-resume demo on Temporal is the one piece I have not yet recorded from start to finish.

Fine-Tuning on Tinker

The task is narrow, which makes it a good candidate for a small fine-tuned model. The pipeline builds 300 training pairs. Each pair is the exact prompt the app sends the model, plus a target argument: the playbook argument for that denial code, followed by two sentences from a past winning appeal for the same code. A pair is kept only if its assembled letter passes the fact-checker and the argument stays under 170 words.

Training uses a LoRA adapter on Qwen3.5-4B, rank 16, with 3 epochs in batches of 8. That is 114 optimizer steps at a learning rate of 1e-4. The loss is computed only on the completion tokens, so the model learns to write the argument and does not learn to reproduce the prompt. The training run finished and printed a tinker:// sampler path, which I set as TINKER_MODEL_PATH. The Model arena then lists the base model and the tuned model next to the local model.

What the tuning can teach is limited. The targets are synthetic, so the tune mostly teaches format and house style: short paragraphs, no letterhead, and placeholders where facts are missing. It cannot discover new legal arguments. A real clinic would replace the targets with the arguments of appeals that actually won, and the app's "Mark as won" button saves exactly that text.

Model Comparison in the Arena

The Model arena runs the same synthetic notices through a local baseline and through hosted open-weight models on Backboard. It ranks each model by first-draft pass rate: how often a draft passes the fact-checker with no retry. That is a narrow measure, but it is the one that matters most for this app, because a draft that needs a retry costs time, and a draft that fails costs the clinic a letter.

Small samples on synthetic data are noisy, and the arena says so on screen. Treat it as a smoke test, not a benchmark.

An Agent on Top of the API

Overturn also has a conversational front end, built with Mastra. The Denial Desk agent uses an open model with memory and two tools: one lists the queue, and one analyzes a claim. It reaches Overturn over HTTP and never sees the raw notice.

In one run, I asked the agent what to work on first. It called the queue tool and returned the top claim, CLM-26009. Then I asked it to analyze that claim. It called the analysis tool, which returned an overturn probability of 0.612, an expected value of $3,765.26, and appeal as the best move. The agent's answer matched the app's dashboard.

tching the app's numbers](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/rp6pg1wgstp37yq7d39c.png)

The agent runs against the local app on port 8000. Pointing it at the Render URL is a one-line change, and I have not yet recorded that version.

Lessons From Putting This on Render

Three things took longer than I expected, and they might save you time.

First, the default branch matters. My GitHub repository's default branch was an initial commit with only a license, and the code lived on master. Render deployed from master correctly, but the repository page showed no README, which made the project look empty to anyone who visited. Setting the default branch fixed that in one step.

Second, deploy the model choice, not just the code. The Render instance is too small for the 7B model, and the switch to 3B was made for the 4 GB instance. The header's model name is the first thing to check after a deploy.

Third, the observability tools tell you different things. The Render event log shows whether the service is up. The Temporal event history shows whether each step ran. The app's trace shows where the time went. None of them alone tells you whether the answer is right, which is why the fact-checker exists.

Why Does Open Innovation Matter?

Why Open Models Matter Here

Privacy is the first reason. A denial notice carries a patient name, a member ID, and a diagnosis. A local model needs no business associate agreement with an API vendor, and there is nothing to leak. That claim holds only if the data-location map above holds, which is why I keep returning to it.

Cost is the second. Per-token pricing on every notice, retry, and redraft erodes the margin on a claim worth a few hundred dollars. A local model is a fixed cost, and that changes which denials are worth working. [PLACEHOLDER: tokens and latency per draft, measured from the trace]

Control is the third. Open weights let a clinic swap the model with one environment variable, fine-tune on its own winning appeals, and compare models on its own data. A closed API would turn each of those into a vendor conversation.

Where the Data Goes

This section is the one I would most like readers to check, so here is the full map.

On the laptop, the real notice goes to the local model on Ollama and the local Temporal dev server. The retrieval index is on Tiger Cloud, and I still need to confirm exactly what it stores. If it holds notice text, patient data leaves the machine, and the privacy claim in this article would be wrong for that path. I am checking that before I publish.

On Render, the web service and the worker talk to a private Ollama service inside Render's network. Render's private networking counts as local in my counter, by design, which is why the header reads 0 external calls. That is a definition I chose, and a reader may disagree with it. For a deployment that is exposed to the public internet, the honest label is "private cloud," not "local."

The Model arena and the Tinker sampler are cloud paths. The arena accepts only synthetic notices, and the app warns before it runs. The header counter does not yet include these calls, so it does not prove the claim for every path. I am fixing that before I say it.

Honest Limits

The data is synthetic. The odds model is only as good as a clinic's history, and a new clinic has little. Drafts are drafts, not legal advice. Appeal windows and criteria vary by payer and plan, so the notice has the final word.

The Render deployment is not the same system as the laptop. It uses a smaller model, a different retrieval backend, and a 103-second drafting step on CPU. A demo that shows both should say so.

The external-call counter is the weakest part of the privacy story. It counts model calls, it treats Render's private network as local, and it does not yet cover the Tinker and Backboard paths. Fixing that is the first item on my list before I make the claim without qualification.

The Temporal kill-and-resume demo has not been recorded end to end, and the Render deployment has not been load-tested. I have marked these rather than let the demo suggest more was tested than was.

What Next

Real denials from a real clinic, under a proper data agreement, is the test that matters. I would also like the data-location check to be automatic, so the privacy claim is enforced in code instead of described in prose.

If you work in revenue cycle, billing, or health tech, the code runs on a laptop with Ollama, and the Render deployment is live. I would like to hear where the playbook is wrong, which denial codes show up most for you, and where the drafts fall short.

Repository: https://github.com/abhitorch81/week2hoct/tree/master

My Agent Session

My First User

The idea came from my friend Manoj Pareek, who works in this business. I gave him the app, and he used it. He was happy with it, and as a thank-you he bought me a KFC treat. That was my first earning from Overturn. It was a gift, not a license fee, so I am not counting it as revenue. It is the first real user signal I have, and I am treating it that way.

Prize Categories

I am entering these categories, because the project uses each technology in a working code path:

Render (Featured): the web service, the worker, and the private Ollama service run on Render from the master branch.
Tinker (Featured): the LoRA fine-tuning pipeline, trained and wired into the arena.
Backboard (Partner): the hosted-model arena.
Temporal (Partner): the durable workflow runner, on Temporal Cloud and locally.
Tiger Data (Partner): pgvector retrieval on the laptop. The Render deployment does not use it yet.
Mastra (Partner): the Denial Desk agent, which calls the Overturn API through two tools.

I did not enter TabPFN, because the odds model falls back to scikit-learn in my current runs. I did not enter Entire, because I have not enabled it on a real session. Sentry is in the code but off.

"The Gene Went Up. Could AI Tell What We Actually Know?"

Abhishek Nandy — Sat, 10 Oct 2026 20:18:19 +0000

This is a submission for the Kaggle Benchmarking Challenge.

A gene can increase in one cell type and decrease in another. Ask whether its average expression increased, and the answer depends on which mixture of cells you mean.

There is a subtler question: can we know that expression increased without knowing its exact percentage increase?

Yes. And a model should be able to say both things at once.

I built CellCaution to test that boundary. In a small, controlled study, Gemma answered every primary case correctly. Verification instructions gave GPT-5.4 mini more correct verdicts but fewer correct complete answers. Claude's strict score looked disastrous until I separated response-format compliance from the mathematical content.

Those findings changed how I would evaluate an AI assistant for scientific analysis: the verdict, the number, and the output contract each need their own evidence.

What I Benchmarked

My interest comes from working on AI for single-cell RNA sequencing. Before asking a model to interpret a complicated biological study, I wanted to know whether it could handle a small descriptive comparison whose answer I could calculate exactly.

Consider these synthetic expression measurements:

Cell type	Control mean	Treated mean
A	17	22
B	53	50

Type A increased by 5 units. Type B decreased by 3. We standardize both groups to the same reference composition, so the comparison uses the same weights on both sides.

At 80% A and 20% B, the control mean is 24.2 and the treated mean is 27.6. The increase is approximately 14.05%.

Now change only what we know about the reference population:

Information supplied	Is a strict increase supported?	Unique percentage	Possible percentage range
Exactly 80% A	Yes	14.05%	14.05% to 14.05%
A is between 65% and 85%	Yes	`null`	7.43% to 16.96%
Any composition is allowed	Insufficient information	`null`	−5.66% to 29.41%

In the middle row, every allowed reference gives an increase. But the constraints do not identify one percentage. Returning null is the mathematically correct answer to the magnitude question, even though the direction is known.

Figure 1. These are final v2 measurements, not an example drawn from the earlier pilot.

That is the specific behavior I wanted to measure: preserving what is known while refusing to manufacture what is not.

A benchmark with exact answer keys

The primary dataset contains 24 cases: eight templates, each presented under fixed, constrained, and unrestricted reference compositions. The templates cover two or three cell types and four expression patterns: mixed changes, additive increases, proportional changes, and no change.

Each response must provide four JSON fields:

{
  "verdict": "supported",
  "standardized_change_percent": null,
  "percentage_range": [7.4324324324, 16.9642857143],
  "explanation": "Every allowed reference gives an increase, but the percentage varies."
}

The verdict is supported when the treated mean is strictly higher at every allowed reference, unsupported when it is higher at none, and insufficient_information when the answer depends on the reference.

The primary score requires both the correct verdict and the correct percentage or null. Range accuracy is measured separately. Numerical answers use a tolerance of 0.05 percentage points. The explanation is retained for inspection, but an LLM judge does not score its prose.

The answer keys use rational arithmetic. Allowed populations are every convex mixture of the supplied reference vertices. Differences are linear in the weights. With positive control means, the percentage ratio at any mixture is a weighted average of the vertex ratios, with weights proportional to each vertex's control mean. Its extrema therefore occur at vertices. This gives exact bounds without relying on a sampled grid or another model's judgment.

Models Tested

I used three models available through the Kaggle Benchmarks registry:

Model	Requested registry ID
Gemma 4 31B	`google/gemma-4-31b`
GPT-5.4 mini	`openai/gpt-5.4-mini-2026-03-17`
Claude Sonnet 4.6	`anthropic/claude-sonnet-4-6@default`

This was a manageable lineup spanning three model families, including Gemma and two hosted assistants. It is a comparison of these requested models on this task, rather than a claim about their providers as a whole.

For each model I recorded 24 direct responses, 24 responses with explicit verification instructions, and eight reordered controls: 168 included case evaluations. The verification prompt asks the model to calculate weighted means at each vertex and check signs, extrema, and uniqueness before returning its answer.

I used SDK defaults rather than a shared, explicitly controlled temperature or reasoning budget. The exported IDs identify the requested models; they do not independently verify the exact backend revision served.

Findings

1. A score must say what it counts

Here are the direct-prompt results under the frozen output contract. Invalid outputs count as incorrect in these totals.

Model	Valid output / 24	Verdict correct / 24	Percentage or null correct / 24	Range correct / 24	Primary correct / 24
Gemma 4 31B	24	24	24	24	24
GPT-5.4 mini	24	18	16	14	12
Claude Sonnet 4.6	1	1	1	1	1

Gemma was consistently correct on this small dataset. That is a useful result, but 24 related synthetic cases do not establish broad scientific reliability.

Claude's result needs a different explanation. In 23 of its 24 direct responses, the frozen parser rejected the response format. Extra text around a JSON block violates the requested contract. A whole JSON object or a single enclosing fence is accepted; surrounding commentary is not.

I kept that strict score and added a post-hoc diagnostic, applied uniformly across models: for an invalid response containing exactly one fenced JSON block, extract that block and run the unchanged grader. No values are repaired, and no additional model calls are made.

Under this diagnostic, Claude's direct primary result becomes 23/24. Its remaining error is mathematical: in the fixed-composition example, it returned about 22.40% instead of 14.05%, consistent with averaging cell-type percentage changes rather than comparing the weighted means.

Figure 2. The extraction diagnostic does not replace the frozen strict score.

The practical lesson is to report both layers. A downstream program needs a usable response. A scientific evaluation also needs to distinguish an unusable response from incorrect mathematical content.

2. Verification helped one component and hurt another

The verification instructions produced this primary-score comparison:

Model	Direct / 24	Verification / 24
Gemma 4 31B	24	24
GPT-5.4 mini	12	9
Claude Sonnet 4.6	1	0

For GPT, verdict accuracy increased from 18 to 21, while percentage-or-null accuracy fell from 16 to 10. The combined primary score fell from 12 to 9. Range accuracy remained 14/24.

Figure 3. One recorded generation per case and prompt arm; this comparison needs replication.

The instruction to verify an answer is not itself evidence of improvement. Here it helped the qualitative decision while the numerical contract became less reliable. A verdict-only benchmark would have missed that tradeoff.

Claude's verification responses were all invalid under the strict contract; all 24 passed the separately reported extraction diagnostic. Again, the strict score alone cannot explain the failure mechanism.

3. Reordering the evidence exposed another instability

For eight constrained cases, I reversed the cell-type rows and every corresponding weight. The mathematical problem and gold answer stay the same.

Gemma preserved its verdict, percentage/null, and range across all 8/8 pairs. GPT preserved all three across 3/8 pairs. Claude also matched across 8/8 under the extraction diagnostic; its reordered strict outputs were invalid, so strict agreement is not assessable.

This is evidence of observed answer instability under equivalent presentations. With one generation per presentation, I cannot separate an ordering effect from ordinary generation variability. The next experiment should repeat both orders with controlled settings.

What I had to fix in the benchmark itself

The first control exports failed the audit: they contained primary-case IDs and verification prompts rather than the intended reordered controls. The execution path had reused results across task contexts.

I excluded those exports and reran the controls under a separate task identity. The included recovery files contain the intended eight control IDs and exact prompts for each model. I then checked all 168 included records against the fixtures and recomputed their strict grades from the raw responses.

That repair matters. An evaluator that checks only whether a run finished can quietly score the wrong experiment.

What changed my thinking

I started by testing whether models could get a biological comparison right. I ended up caring just as much about whether they could represent the boundary of the available evidence.

For an analysis assistant, I would now check three things independently: whether its conclusion follows from the constraints, whether its number is actually identifiable, and whether software can consume its response. A confident explanation cannot substitute for any of those checks.

This study uses synthetic descriptive measurements. It does not test raw sequencing analysis, biological replication, causal treatment effects, significance testing, or clinical decisions. The 24 cases share eight templates, and the verification comparison is exploratory. I would next add repeated runs, more unseen template families, and a calculator-assisted arm to distinguish arithmetic errors from failures to reason about missing information.

My Benchmark

Explore CellCaution on Kaggle

Published CellCaution task

The public leaderboard currently contains Gemma's publication-validation run. The tables in this article describe the separately audited three-model study; they should not be read as three imported leaderboard runs.

The implementation builds on the Kaggle Benchmarks Python library and cookbook. The synthetic fixtures, reference-constraint design, rational answer keys, and component-level evaluation were developed for CellCaution. AI assistance was used for implementation, auditing, visualizations, and editing; I reviewed the measurements and claims against the exported responses.

The question I want a scientific assistant to preserve is simple: what can these measurements establish—and what would it be inventing?

Inspect the audited study evidence

The dataset contains exact prompts, raw model responses, answer keys, recorded scores, and the frozen protocol for all 168 included evaluations. It includes the recovered reordered controls and excludes the invalid original control exports.

One failure, two different problems

In direct case F01, Claude Sonnet 4.6 saw control means of 17 and 53, treated means of 22 and 50, and a fixed reference of 80% type A and 20% type B.

Its returned JSON reported:

{
  "verdict": "supported",
  "standardized_change_percent": 22.397,
  "percentage_range": [22.397, 22.397]
}

Excerpt: the explanation field and surrounding prose are omitted here; the evidence dataset retains the full response.

The verdict was correct. The number was not. The control weighted mean is 24.2 and the treated weighted mean is 27.6, so the requested increase is:

100 × (27.6 − 24.2) / 24.2 = 14.0496%.

The response instead averaged the cell-type percentage changes:

0.8 × [100 × (22 − 17) / 17] + 0.2 × [100 × (50 − 53) / 53] ≈ 22.3973%.

These formulas answer different questions because the cell types have different control baselines. Averaging their percentage changes is not the percentage change of their weighted mean.

This response also included prose outside its JSON block, so it failed the frozen output contract. Extracting the block resolves that formatting issue for the separate diagnostic, but leaves the mathematical error intact. A correct verdict and a detailed explanation did not guarantee a correct magnitude.

Reproduce the audited study

This notebook verifies the frozen protocol and exact prompts, recomputes all 168 included evaluations from raw responses, and reproduces the score tables and charts without additional model calls.

Did the verification result survive repetition?

After the original study, I added two complete GPT-5.4 mini repeats using the same 24 cases, frozen prompts, scorer and requested model ID. Each repeat included direct and verification arms, adding 96 case evaluations. This was an exploratory extension planned after seeing the original findings.

Run	Direct primary / 24	Verification primary / 24	Verdict: direct → verification	Percentage/null: direct → verification
Original	12	9	18 → 21	16 → 10
Repeat 1	12	11	19 → 20	14 → 12
Repeat 2	14	13	20 → 19	15 → 16

All 96 new outputs passed the format contract, and their scores were recomputed from the raw responses against the frozen grader.

Verification produced a lower combined primary score in all three recorded runs. However, the component-level pattern varied. In the second repeat, percentage/null accuracy improved while verdict accuracy declined—the reverse of the original tradeoff.

That changed my interpretation. The evidence supports a narrower observation: this verification checklist did not improve the complete answer score in these three runs. It does not establish that verification generally harms numerical reasoning.

The original study contained 168 evaluations; this extension brings the total to 264. The repeats reuse the same 24 cases from eight templates, so they measure observed response stability rather than generalization to new problems. SDK defaults were retained; temperature and reasoning budgets were not explicitly controlled.

The frozen replication plan, all four response exports and the summary are included in Version 2 of the audited evidence dataset.