Originally published on hexisteme notes.
On July 31st, the accelerator quota API on one of my Kaggle accounts came back at 108,511.735 seconds used against 108,000 allowed — 100.47%. The TPU line read 0 of 72,000. Four competitions were active on that account, and all four had a submission go out within the same 25-minute window that morning. The obvious read: one shared pool, several competitors, and whichever session submits first eats the week. So I built a priority table — rank 1, rank 2, rank 3, do-not-run — with a rationale and a stated falsifier next to each row.
The table had ranks. It did not have a single number in it. That was the first warning sign, and I didn't catch it until afterward.
The table with no numbers
A priority ranking answers "who goes first." It doesn't answer "what does a turn cost." I had ordered four competitions by importance without ever asking what a session from each of them actually spent against the pool I was rationing. That's a strange thing to skip, in hindsight — you can't allocate a budget you haven't priced. The absence of any unit-price number in a document whose entire job was allocation was itself the tell that nobody had measured anything yet. I just didn't read it that way at the time; I read a 100.47% line and reached straight for a rationing policy.
Three measurements
Before shipping the table, I went back and actually measured what was being charged to the pool. Three things fell apart at once.
Rescoring doesn't touch personal quota. Code competitions on this platform re-run your submitted notebook on the host side to score it. I had been assuming that re-run was billed against my personal quota. One of the four competitions had already checked: personal quota read the identical 108,511.735s before and after a full scoring cycle, bit for bit. A second competition confirmed it independently. So on my account, in late July 2026, the only thing that spends personal quota is a new commit — and I want to state that with exactly the scope I measured it at, not one inch wider. It is a reading from one account in one window, not a claim about how the platform bills everyone. The whole point of this essay is that I hadn't checked before; overgeneralizing what I did check would be the same mistake wearing a lab coat. Either way, the allocation table's entire unit — "how many submissions can we afford" — was pricing something that, on this account, cost nothing.
A "GPU" label bills wall clock whether or not a GPU is used. A kernel-log audit on one competition turned up 1,043 seconds of runtime with zero occurrences of the string device: GPU anywhere in the log. The training path was hardcoded to CPU — torch.device('cpu'), map_location='cpu' — and the gradient-boosted model defaulted to CPU too. But the kernel's metadata still said machine_shape: "Gpu", and the quota meter charged the full 1,043 seconds against the accelerator pool anyway, for a run that never touched an accelerator. Flip that one field to enable_gpu:false and, on this account, CPU-only re-runs are unmetered — no quota concept applies to them at all.
The billing multiplier isn't 1x. A separate, direct measurement: 1.5 wall-clock hours of work consumed 2.92 hours of quota — a 1.94x multiplier, scaled to the accelerator count on that machine shape. So the week's usable budget wasn't "108,000 seconds" in any intuitive sense. Divided by the multiplier, it was closer to 30 quota-hours, which is roughly 15.5 hours of actual wall-clock work. A smaller number than the raw quota figure suggests, and one I'd never converted before.
Once I had those three, I could finally build an actual unit-price table instead of a priority table:
| Operation | Quota cost |
|---|---|
| Smoke commit | ≈0.4h |
| Local probe | ≈2.4h |
| Local eval (120-task) | ≈10.2h |
| Submission rescoring | 0 (competition compute, not personal quota) |
| CPU-only kernel | 0 (unmetered) |
Two of the five rows in the thing I was supposedly rationing cost nothing at all.
Recomputed: real demand was 13% of the pool
With unit prices in hand, I profiled what each competition on the account was actually running. Four were active; a fifth was scaffolded but hadn't started work yet.
| Competition | Stack | Actual demand |
|---|---|---|
| A program-synthesis competition (LLM + test-time training) | PyTorch + Unsloth (Triton/CUDA) + custom CUDA kernels | 11,520s |
| An agent-security competition | Pure-Python search. Zero ML-framework imports | Commit time only (~0) |
| A tabular geoscience competition (sequential estimation) | numpy/pandas + a CPU-only auxiliary model | 0 |
| A game-agent competition | ctypes C tree search + LightGBM (CPU). Zero torch/tf/jax imports | 0 |
| Not yet started | Scaffold only | 0 |
Sum it: roughly 11,520 of 108,000 seconds, about 10.7%. Padding every estimate generously, the ceiling is 13%. One competition, out of five on the account, was doing anything that touched an accelerator at all.
So the 100.47% reading wasn't four competitions fighting over a shared pool. It was one competition's mislabeled kernel burning wall clock against an 87%-empty pool. Had I shipped the priority table as written, it would have spent real effort rationing a pool that was almost entirely free, while the actual cause — a checkbox that said "Gpu" on a run that never used one — sat there unfixed, because the table never asked what anything cost.
Worth being precise about what went wrong, because it isn't the usual false alarm. The 100.47% was not a misreading, a broken dashboard, or a contaminated log. Those seconds were charged, and the meter reported them correctly. The number was true; the denominator of judgment behind it was missing. I compared a real usage figure against a total demand nobody had ever priced, and filled the gap with a story about contention. A metric can be perfectly accurate and still support a conclusion it does not license.
The same error, three times, three different answers
This wasn't the first time in this project that the resource carrying the name of the bottleneck turned out not to be the actual bottleneck. It's the third, and each time the correct answer was different:
| When | Believed scarce | Actually scarce | How it was found |
|---|---|---|---|
| Early on | Submission slots (94 of them) | Quota — a submission is 25x cheaper in quota, and its result 14x more precise in sigma, than a local eval run | Measuring unit price directly |
| July 31 | Quota | Commits — rescoring leaves personal quota bit-identical before and after | Comparing quota before/after a scoring cycle |
| A separate competition | GPU time | Per-decision CPU budget — 0.3–30s used against a 600s ceiling, 1,340x of it sitting unused | Instrumenting actual per-decision usage |
Three different names for "the thing we're rationing." Three different real answers underneath. That third row stings the most: while I was carefully protecting GPU time, that same competition was sitting on 99.93% of an already-allocated CPU decision budget, unused. The scarcity I was managing and the slack I was ignoring were right next to each other.
The pattern across all three rows is identical: treat the name of a resource as the bottleneck, and never price the operations charged against it. It's a related failure to one I've written about before — a quality gate whose own measurement noise turned out to exceed the thing it was guarding — but it's a different mistake: that one measured the wrong way, this one never measured at all before deciding how to ration. And because the answer changed each time, "what's scarce this time" has to be asked fresh at every phase of a project. Carrying forward last cycle's answer as if it were a constant is exactly how the next misdiagnosis gets seeded.
The subagent made the same mistake one level up
I parallelized the profiling work across five subagents, one per competition. One of them reported the quota required as 140,000 seconds — a number larger than the entire weekly pool, on its own. The cause: it had double-counted submission rescoring as personal-quota spend, the same wrong assumption that started this whole investigation. The real figure was 11,520 seconds. Its report was off by 12x.
The part worth sitting with is where the correct number was already written down: in that same competition's own report, which the subagent cited as evidence in its output. It read the file. It quoted from the file. It just never applied what the file said to its own arithmetic. This isn't a story about a model being unreliable in some general sense — it's the same failure I'd just made myself, one level up: citing a source doesn't guarantee the citation gets used, and a measurement you delegate needs to be checked against its own primary source before it goes into a summary, not trusted because it came with a citation attached.
The part that transfers
None of this is really about Kaggle GPU quota. It's about any metered resource you've decided is scarce before checking what's actually charged against it — a cloud bill, CI minutes, an API rate limit, database connections, seat licenses. The rules that came out of this:
- Before you believe "X is scarce," measure the unit price of the operations you've attributed to X. The resource's name doesn't set the bottleneck; its measured price does.
- Don't build the allocation policy before you've measured the price. A rationing table built on an unmeasured cost doesn't fix the underlying bug — it makes the bug permanent, because a policy that exists reads as a problem that's already being managed. Concretely, on a cloud bill: before you write the rule that caps how many staging environments each team may run, price one staging deploy, one CI job, and one flaky-test rerun against the invoice line that scared you. If it turns out the line is dominated by an idle managed cluster nobody scheduled work onto, the cap you were about to write would have taxed the wrong people and left the cluster running.
- Check whether a path that looks free is actually being billed. And check the reverse just as hard: a path that looks expensive might already be free.
- Re-ask "what's scarce this time" at every phase, on every project. The answer moves. Treating last time's answer as a constant is how the next misdiagnosis gets seeded.
- A measurement you delegate goes into a summary only after you've checked it against its own source — a citation is not the same claim as a correct conclusion drawn from it.
Where this breaks
An essay telling people to measure before they ration owes its own falsifiers:
- If any of the concurrent competitions turns on real GPU training — an offline model, say — the "13% of the pool" figure stops holding. The unit-price table stays valid; only the allocation recalculation changes.
- The billing behavior itself — that rescoring left personal quota untouched — is platform policy, measured on one account in one window. Platform policy can change without notice. That bit-identical check needs to be re-run periodically, not treated as a fact fixed at measurement time — otherwise this essay becomes the source of the next misdiagnosis instead of the fix for the last one.
- The 1.94x wall-clock multiplier is a function of the accelerator count on that specific machine shape. It doesn't transfer to a different shape without re-measuring.
More notes at hexisteme.github.io/notes.
Top comments (2)
the priority table had a stated falsifier on every row and the unit-price table that replaced it has two columns, operation and quota cost. every number in that second table is a provider-side fact that can move without telling you: the 1.94x multiplier is scaled to the machine shape, the rescoring finding is one account in one window by your own scoping, and "cpu-only kernel, unmetered" is exactly the row a platform quietly rewrites. worth giving it the same two columns the priority table had, a falsifier and a measured-on date, or in three months it's the same confident unchecked artifact, just with numbers in it. the cheap part is that the falsifier for most of those rows is a measurement you already ran once, so it's a re-run rather than new work.
You're right that the unit-price table inherited the priority table's confidence without its falsifiers — the 1.94x multiplier, the single-account rescoring window, and the "cpu-only kernel, unmetered" row are exactly the cells a platform rewrites silently. Adding a falsifier column and a measured-on date would cost almost nothing since most of those checks are re-runs of measurements I already did for the priority table. That two-column discipline is what keeps the artifact honest when the provider shifts pricing or metering three months from now. Thanks for naming the specific rows that need that guardrail.