This is a submission for the Kaggle Benchmarking Challenge.
I built a Kaggle benchmark to test whether coding agents can make a real CPU inference program faster while preserving its tested behavior. Gemini, Claude and GPT read, edit and test the C source of Karpathy’s llama2.c under the same tool-call and editing limits. Their resulting programs run TinyStories checkpoints of 15M, 42M and 110M parameters across several request workloads.
The benchmark evaluates both the edited program and the effort needed to produce it. It checks numerical and text correctness, request completion time and when generated output first becomes visible, while recording the optimizer’s token usage and elapsed time. The central idea is to measure useful end-to-end optimization: a faster final answer can still hide a worse streaming experience. One early result made that gap concrete.
The original program started showing generated output after 103 ms and finished after 1,321 ms. An AI-optimized version finished in 483 ms, but showed no generated output until then.
The final answer was identical. The complete response was about 2.7 times faster. Yet the user waited nearly five times as long for it to start.
I restored the flush after each token without touching the optimized math. Output began after 37 ms, and the request finished after 477 ms. The useful computational optimization survived; the streaming regression did not have to.
That controlled experiment on Kaggle Linux exposed a missing requirement in my first task. I added a streaming-latency check, then ran three fresh executions per coding agent. All nine passed the automated checks, but their completion-time gains were much less uniform.
What I Benchmarked
I used Andrej Karpathy's llama2.c, pinned to commit 350e04fe, with the author's TinyStories FP32 checkpoints containing 15M, 42M and 110M parameters. These are small models running through llama2.c, not full Llama 2 checkpoints. I chose this implementation because its compact C source and small checkpoints make full-program measurement practical on a CPU. A candidate has to compile and demonstrate a measured gain; a plausible optimization explanation is not enough.
Through Kaggle Benchmarks, the optimizer can read run.c, replace an exact, uniquely matching block, and benchmark its current candidate. Each attempt allows eight tool calls, at most six successful edits and two benchmark calls for feedback. A 600-second guard is checked at tool entry; it does not forcibly cancel an in-flight model request. The limited interaction budget is part of the task, not a matched reasoning or token budget across providers.
The prompt requires FP32 (32-bit floating-point) precision and compatible existing interfaces, and prohibits changes to weights, tokenizers, stopping rules, threads, external libraries, hardcoded answers and evaluator manipulation. Both programs use GCC -O3 with serial CPU execution.
Correctness and latency are separate checks
For numerical checks, both programs receive the same input tokens at each position, including positions beyond the prompt. I compare full-vocabulary logits—the scores used to choose the next token—over traces of 16, 32 and 128 positions. Values must be finite and satisfy:
abs(candidate - reference) <= 1e-4 + 1e-4 * abs(reference)
Generated text must also match exactly on the evaluated requests, including seeded nucleus sampling. The revised task adds four deterministic, untrained checkpoints for grouped-query attention, vector-loop tails and an unshared classifier. Those fixtures check implementation compatibility, not language quality.
The request timer starts after loading weights and the tokenizer. It covers tokenization, prompt processing, model computation, sampling, detokenization and output. A parent process observes stdout, excluding the exact ASCII prompt echo before identifying the first generated byte. This is time to a generated output byte, including pipe transport and scheduling, rather than an internal first-token timestamp or a full network-serving measurement.
The agent can test two short workloads: greedy generation and seeded nucleus sampling, which samples from a high-probability subset of next tokens. Final evaluation adds a longer prompt and a different story prompt, then tests all four at all three trained checkpoint sizes. The 64- and 128-step caps include prompt processing, rather than counting only newly generated tokens. One warmup precedes three measured requests per workload; I use their median.
The first task scored completion time and checked final output. The revised task also requires each candidate first-byte median to stay within:
max(1.10 * reference median, reference median + 0.002 seconds)
The relative or absolute margin allows some timing variation. This gate is a task rule, not a statistical significance test. Completion speedup is scored only when the tested numerical, text, shape and streaming checks pass. The score is the geometric mean of twelve size/workload speedup ratios. A compliant but ineffective optimization can score around 1×. Behavioral failures score zero; infrastructure failures are recorded separately.
Models Tested
The revised experiment used Gemini 3.7 Flash, Claude Sonnet 5.5 and GPT-5.4, with three separately executed optimization attempts per model. Every attempt starts from the pinned original, opens a fresh chat and saves its own candidate, usage and timing records. Historical completion-only attempts are retained separately and are not pooled into these nine runs. I chose this lineup to cover three providers with working tool-enabled integrations in the Kaggle SDK. It is a practical comparison under the task’s interaction limits; it does not match providers’ internal reasoning budgets.
The model calls leave settings at SDK defaults. The documented prompt signature uses temperature 0 and seed 0, with reasoning unspecified, where supported. These are repeated fresh executions under fixed settings, not randomized seed trials or matched reasoning budgets. Provider determinism is not assumed, and measured tool feedback can differ between executions.
GPT-6.1 Sol was attempted in the earlier pilot but failed at the integration interface before producing an optimization. I excluded that run from model-performance conclusions rather than treating an API failure as a bad code change.
Findings
Identical final output hid a streaming regression
Claude's original candidate removed the per-token fflush(stdout) and flushed after the final newline instead. Flushing sends buffered text to the output pipe immediately; removing it can make generated text wait in memory. The candidate improved matrix multiplication too. Captured text still matched, and the first version of my grader rewarded completion time, so I kept its original score. The task had not required preserving streaming latency.
I replayed the saved candidates in a separate Kaggle Linux job with GCC 14.2 and the 15M checkpoint. No model calls were involved. Each of four workloads had one warmup and five measured requests. The replay included an unchanged-code control, a deliberately buffered version of the original, and Claude's candidate with only its per-token flush restored.
| Frozen candidate | Completion speedup | First generated byte speedup |
|---|---|---|
| Gemini | 3.44× | 3.54× |
| Claude | 2.73× | 0.44× |
| GPT-5.4 | 2.16× | 2.19× |
| Claude with flush restored | 2.75× | 2.80× |
All final outputs matched. The unchanged-code control was approximately 1× for both metrics. Below 1× means that metric became slower. The table reports geometric means across the four workloads. These are frozen-code replay results, separate from the fresh revised-task results below.
For the short greedy request, the intervention looked like this:
| Implementation | First generated byte | Request completion |
|---|---|---|
| Original | 103 ms | 1,321 ms |
| Claude with buffered output | 483 ms | 483 ms |
| Same candidate with per-token flush restored | 37 ms | 477 ms |
Restoring the flush moved output forward by about 446 ms while completion remained around 0.48 seconds. The small difference in completion medians does not establish that restoring the flush made computation faster; the clear effect is when output becomes visible.
The control that only removed flushing from the original delayed its first generated byte to about 1.33 seconds, near completion. On the long prompt, Claude's faster math outweighed its buffering delay, so first output was already earlier than the original. Claude's original buffered candidate failed the proposed per-case first-byte limit on three of four workloads; its flush-restored version passed all four. A completion aggregate alone would hide those regressions.
Passing the revised checks did not guarantee improvement
The nine fresh revised-task executions produced:
| Optimizer | Attempts passing automated checks | Completion speedups | Median |
|---|---|---|---|
| Gemini 3.7 Flash | 3/3 | 5.38×, 5.35×, 5.27× | 5.35× |
| Claude Sonnet 5.5 | 3/3 | 3.69×, 3.87×, 3.17× | 3.69× |
| GPT-5.4 | 3/3 | 1.00×, 2.32×, 1.00× | 1.00× |
Gemini and Claude consistently improved completion time in these runs while satisfying the streaming gate. GPT's best candidate improved it too, but two candidates were effectively neutral. Reporting only its 2.32× result would have hidden that outcome.
The request-level breakdown shows that Gemini and Claude improved completion time in every tested size/request combination across their three attempts (36 combinations each). GPT had 11 of 36 ratios below 1×, with a minimum of 0.98×. Those small deviations do not establish a statistically significant regression; they reinforce the distinction between its neutral candidates and its useful one. The full breakdown preserves all 108 ratios rather than averaging away the workloads.
Source inspection gave a concrete example. GPT's first candidate used all six edits, precomputed rotary-position math and removed OpenMP pragmas. The scalar matrix-multiplication loop remained largely unchanged, and the build had no OpenMP flag, so those pragmas were inactive already. The final measured gain was 1.0013×. In another attempt, two focused edits produced 2.32×. The edit count and the presence of an optimization technique did not establish an end-to-end benefit.
These observations concern the saved implementations and requests. Source inspection does not isolate the contribution of each edit; I did not run a profiling ablation of GPT's changes. The results are not a general coding-model ranking. Each provider ran in its own Kaggle job; speedup pairs original and candidate within that job, but normalization does not eliminate every hardware interaction. Three executions and a small fixed request set are not enough for broad statistical claims. The observed ranges are descriptive, not confidence intervals. Millions of compared logit values are correctness checks, not millions of independent tasks.
Finding the gain also takes time and tokens
I checked custom usage totals against all nine SDK trajectory artifacts. The following values are medians per optimization attempt, rather than totals for three attempts:
| Optimizer | Input tokens | Output tokens | Optimizer wall time | Reported quota cost |
|---|---|---|---|---|
| Gemini 3.7 Flash | 318,985 | 50,733 | 435.1 s | $0.2864 |
| Claude Sonnet 5.5 | 38,431 | 2,454 | 48.4 s | $0.0853 |
| GPT-5.4 | 87,784 | 1,736 | 71.0 s | $0.0665 |
Gemini had the largest median gain and used more reported tokens and time. Claude found smaller gains with less effort. GPT's lower quota cost accompanied a much less consistent completion benefit. I keep these dimensions separate instead of combining them into an arbitrary efficiency score.
Input tokens accumulate repeated context across requests. Output counts use provider-reported SDK fields; separate cache and reasoning breakdowns are unavailable. Different tokenizers and default reasoning behavior limit comparisons. Quota cost is not cash I paid.
Optimizer wall time includes model/proxy waiting, editing and benchmark calls for feedback. Reported backend latency excludes those tools. Final validation, including downloads and checks, is timed separately and can take longer than optimization. Nested tool and validation timings overlap and must not be added twice. My earlier combined timer included final grading, so I did not relabel it as model optimization time.
The evaluator needed verification too
An early pilot accidentally compiled the original source for both sides. A quoted #include "run.c" selected a file beside the adapter before the candidate include directory. I discarded those timings.
The corrected adapters use #include <run.c> and verify compiler dependencies. An additional control tests the shadowing case. Numerical negative controls reject a zero-logits implementation, including on the four synthetic shapes. These checks matter before interpreting the attractive speedup numbers.
What I would extend next is broader request variation, more executions, stronger isolation from hostile native code, and checks across more hosts and shapes. The current reference is deliberately simple C, not a mature inference engine. Compiled-adapter compatibility does not establish a stable binary ABI or universal correctness.
The lesson I will carry into real optimization work is concrete: preserve the observable behavior that users need, then measure whether the code change helped. Matching final text was useful. The flush intervention showed the missing contract, and the repeated attempts showed why passing that contract still does not establish a performance gain.
My Benchmark
CPU Inference Optimization and Streaming on Kaggle contains the revised optimization task. The task page exposes the evaluation code. Its executable notebook and saved outcomes for all nine executions are included in the reproduction dataset. The live leaderboard selects a representative individual run, rather than the median reported in this article.
The reproduction dataset also includes the Linux intervention notebook, frozen candidates and controls. The results and reproduction notebook verifies all 88 hashed members of the attached source/result archive and exports the figures. It makes no model calls. Candidate source, patches, raw result/usage records and a checksum manifest are included in the reproduction dataset. Trained weights are downloaded from the author and verified by hash. The original completion-only pilot records remain separate in that dataset.
Credit to Andrej Karpathy for the MIT-licensed llama2.c implementation and TinyStories checkpoint distribution, and to Kaggle for the benchmark SDK and model integration. Codex helped build and check the harness and draft this article. Optimizer edits came from the recorded model runs; the flush restoration and synthetic fixtures are evaluator-authored controls.
Top comments (3)
the first-byte metric is the part that will stick with me. in practice the user notices the stream, not the completion time - when the tokens stop appearing, people assume the box is broken and start killing processes. i run llama.cpp on a 4x3090 rig and watch first-token latency in a little dashboard i built (github.com/SikamikanikoBG/homelab-monitor) mainly to spot when a model server starts returning tokens slower than usual, and the difference between "still computing" and "actually stuck" is exactly the kind of thing that shows up in the logs but not in a completion-time-only benchmark. also nice that the flush fix kept the matmul win - that is the cleanest possible outcome.
does the llama2.c tweak trade off precision for speed, and have you measured its impact on KV‑cache bandwidth when you hit multi‑GPU batch sizes? also, without a perf profile it's unclear whether the gain comes from real compute reduction or just kernel launch overhead shaving
tr.ee/dev-to