A stale price never throws, so we made freshness a build failure

A hardcoded unit price never throws. It won't turn a test red. It just quietly becomes false on some Tuesday, and you hear about it from a customer.

I keep a price comparison table for LLM APIs: 57 models across 16 vendors, in one JSON file. The change log next to it is the part worth looking at. It holds 28 records so far this year, and once you drop the launches and the deprecations, 14 of them actually moved a unit price.

The magnitude is what got my attention. For the entries where I could compute the input-price delta, the median change was 60%, and two came in at 200%. At that size, calling the table "a bit out of date" is just a politer way of saying it's wrong. A monthly estimate built on last quarter's numbers isn't off by a rounding error.

The field that made it visible

Every vendor entry carries two extra fields: lastVerified, the date someone last read that vendor's official pricing page, and verifyMethod, how they read it.

{ "name": "Baidu", "lastVerified": "2026-06-05", "verifyMethod": "monitored" }

Sort the 16 by date and a correlation falls out. The seven on auto_official, meaning a scraper reads the vendor's own page, all sit within 24 days. The nine on monitored, meaning a human is supposed to remember, run back to 2026-06-05. That is 101 days. It isn't a forgotten week, it's a standing condition, and no amount of good intentions has ever fixed it.

Report the floor, not the max

The tempting way to show freshness is max(lastVerified). Today that renders as this morning's date and implies the whole table was checked today, while two vendors sit three months back.

So the public number is the oldest date instead, with coverage reported separately against a two-week window:

const freshnessWindowDays = 14;
const freshVendorCount = dates.filter((d) => d >= cutoff).length; // 8 of 16

Fifty percent. It is an unflattering number to publish, and it's the only version that makes anyone go move those two vendors onto the scraper. If a freshness metric takes its reading from whichever record you happened to touch most recently, it will flatter you every time.

Make the build fail

Timestamps don't help prose. Once the price sync started running daily, sentences in the comparison pages reading "Model X (¥a in / ¥b out)" kept their old numbers while the JSON moved underneath them. tsc cannot see that. Neither can review-by-reading.

Two scripts now run in prebuild. One diffs every "model ¥in/¥out" phrase in the copy against the JSON and reports drift. The other rejects any hardcoded ¥1.00-style literal on the pricing page outright, so the figure has to be derived from the data.

I earned that second gate the embarrassing way. Our trust page twice shipped "verbatim" quotes of vendor privacy policies that were not in the linked source. The word verbatim is now a banned regex in that file, and the page may link to a policy but may not characterize what it says.

The gate then caught the article I wrote about the gate. I had typed "13 models" into a sentence with no date anywhere in it, and the check refused to build until I anchored it, on the grounds that a sentence about history should keep its historical number while a sentence about today has to match the table. Fair.

Four steps, no dependencies

• Give every record a lastVerified and a source URL. A number missing either one doesn't get committed.
• Aggregate to the floor. Publish the oldest date, plus coverage inside a window you pick on purpose.
• Derive prose numbers from the data. Where you can't, regex-scan the copy for literals in prebuild and exit non-zero on a hit.
• Timestamp the derived inputs separately. 40 of the 57 models carry their own cached-input price, and the USD vendors get converted at an FX rate (6.7156 today) that moves even when the vendor's dollar price doesn't.

The table still goes stale. It just says so now.

I build llmabacus.com, where this table is the product; the Chinese version of this writeup has the full per-vendor breakdown.